On September 15th, Lydia Alapatt successfully defended her Master’s Thesis titled: „Face Recognition with Vision-Language Models: Prompt-Based Multimodal Fusion Guided by Scores and Generated Justifications“.
Abstract:
Face recognition is part of many aspects of daily life. While domain-specific models exist to perform the biometric task of face recognition, with the increasing presence of Foundation Models (FMs), the question arises whether these general models can also be employed for this task. Although several studies on this topic have been published in recent years, it can still be considered relatively under-explored. In this thesis, four Vision-Language Models (VLMs) are prompted in a zero-shot setting to perform face verification on 200,000 constructed pairs from Labeled Faces in the Wild (LFW). Out of fifteen candidate models, the four best-performing models from different families are selected. Each of the four models takes on three roles: standalone, helper, and decider. Each model performs face verification in a standalone run, which serves as the baseline. Justifications that support the similarity scores are generated and used to assist the decider models. In a subsequent fusion process consisting of twelve helper-decider combinations, each model statically reuses its respective standalone results in a helper role.
The results of two helper models are passed on to each decider model. Across four scenarios, each with different inputs, a decider model performs score fusion with or without an assessment of the justifications. For two scenarios, the fusion is additionally based on image pairs, and the decider models generate justifications that reflect their decision-making process. Although the content is not verified in this thesis, the 4.8 million justifications can serve as a basis for future research.
This thesis aims to examine whether the fusion process improves the performance of the four models and whether the generated justifications have an impact on the models’ decisions. For three models, InternVL3-8B, Ovis2.6-30B-A3B, and Qwen3-VL-8B-Instruct, the performance with the respective best helper combination improves across all relevant metrics and scenarios compared to the standalone run. Models with a weak baseline performance gain the most from the fusion process. For Gemma4-31B-it, the model with a strong baseline performance, the impact is not meaningful. However, as a helper model, Gemma4-31B-it appears in the best helper combinations of all the other models, and Ovis2.6-30B-A3B appears in the worst helper combinations of all the other models. The justifications are considered by the decider models and have a small but measurable impact.
