Vasileios Makris successfully defended his master thesis titled: „Applying Semantic Segmentation to Explain Face Recognition“
Abstract
Deep learning face verification systems have achieved strong benchmark performance, but their decisions remain largely opaque. It is difficult for nontechnical users to determine why a system classifies two faces as that of one person or different people given the complexity of modern deep learning models. Lack of transparency creates trust issues and makes it difficult for human operators to assess, verify, override automated decision making processes where errors may directly impact individuals rights or access to services. This thesis presents an operator oriented explainability framework for face verification that combines semantic facial part reasoning, fidelity analysis, and human centred evaluation.
The proposed framework integrates face alignment, AdaFace based verification, semantic face parsing, GradCAM sensitivity mapping, and per region occlusion analysis into a single pipeline. Facial regions such as the eyes, nose, eyebrows, mouth, skin, and hair are used as semantically meaningful explanation units. To evaluate faithfulness of explanations, a cross perturbation deletion evaluation was developed by testing whether region importance rankings generalize across independent families of perturbations. The framework was evaluated on three data sets: (1) the labelled Faces in the Wild dataset for threshold calibration and fidelity analysis (2) the HDA Doppelgänger dataset as a hard non mated pair benchmark (3) a curated 41 pair dataset used for both detailed technical analysis and a two part user study.
The results show that semantic region based explanations are technically feasible and that occlusion based explanations demonstrate greater fidelity than explanations derived solely from sensitivity mappings. Results also indicate that GradCAM and semantic occlusion capture different aspects of model behavior. Much of the apparent agreement between them can be attributed to region size effects rather than true convergence in what they explain. Regarding the user study, the explanation assisted condition did not produce a statistically significant overall accuracy improvement relative to the unaided baseline. However, it did improve confidence calibration and encouraged participants to engage selectively with AI support. Users generally preferred explanations as a secondary decision support aid rather than as a replacement for their own judgment.
Overall, the thesis suggests that the primary value of operator facing explanations for deep learning based face verification lies not in large gains in raw decision accuracy, but in increased transparency and in providing a structured basis for human review of difficult cases.