ARXIV:2603.25613 · MULTIMODAL LLM BIAS · SUBMITTED 02 APR · 02:30 UTC · FRESHNESS STALE

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

Ünsal Öztürk · Hatef Otroshi Shahreza · Sébastien Marcel · arXiv

This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

Ship in 2-4 weeks›Score4.0Evidence unverified

Opportunity summary

Pain This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

Evidence 0 refs | 0 sources | 17% coverage

Blocker Evidence unverified

Open Build Read PDF Signal Canvas Track

PROBLEM

This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups. Unlike dedicated face recognition systems, MLLMs approach this task through visual…

METHOD

Full abstract

Multimodal Large Language Models (MLLMs) have recently been explored as face verification systems that determine whether two face images are of the same person. Unlike dedicated face recognition systems, MLLMs approach this task through visual prompting and rely on general visual and reasoning abilities. However, the demographic fairness of these models remains largely unexplored. In this paper, we present a benchmarking study that evaluates nine open-source MLLMs from six model families, ranging from 2B to 8B parameters, on the IJB-C and RFW face verification protocols across four ethnicity groups and two gender groups. We measure verification accuracy with the Equal Error Rate and True Match Rate at multiple operating points per demographic group, and we quantify demographic disparity with four FMR-based fairness metrics. Our results show that FaceLLM-8B, the only face-specialised model in our study, substantially outperforms general-purpose MLLMs on both benchmarks. The bias patterns we observe differ from those commonly reported for traditional face recognition, with different groups being most affected depending on the benchmark and the model. We also note that the most accurate models are not necessarily the fairest and that models with poor overall accuracy can appear fair simply because they produce uniformly high error rates across all demographic groups.

RESULT

ScienceToStartup currently rates this 4.0/10 on the public viability pass. Our results show that FaceLLM-8B, the only face-specialised model in our study, substantially outperforms general-purpose MLLMs on both benchmarks. Code availability is flagged in…

WHY NOW

Multimodal LLM Bias moved forward this cycle; last verified April 2026. Public score 4.0/10. Production flags indicate code availability.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score4.0

PainThis research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

Evidence0 refs | 0 sources | 17% coverage

Blockerno shell-level blocker reported

Analysis summary

This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Competitive landscape

This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

Segment

Multimodal LLM Bias

Adoption evidence

No public code link in the paper record yet

Commercial read

4.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "f3bcc4a4-3519-4995-bab1-34edaf7b0973", "arxiv_id": "2603.25613", "canonical_route": "/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "endpoints": { "paper_pack": "/api/v1/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification/paper-pack", "build_passport": "/api/v1/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification", "normalized_query": "2603.25613", "route": "/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "paper_ref": "demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification#webpage", "url": "https://sciencetostartup.com/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "name": "Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification", "description": "This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification#scholarlyArticle", "headline": "Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification", "description": "This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.", "url": "https://sciencetostartup.com/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification", "sameAs": "https://arxiv.org/abs/2603.25613", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2603.25613" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-03-26T16:30:00.000Z", "author": [ { "@type": "Person", "name": "Ünsal Öztürk" }, { "@type": "Person", "name": "Hatef Otroshi Shahreza" }, { "@type": "Person", "name": "Sébastien Marcel" } ], "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 4 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "Multimodal LLM Bias" }, { "@type": "PropertyValue", "propertyID": "commercialReadiness", "value": "code" } ] }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "Multimodal LLM Bias", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "Demographic Fairness in Multimodal LLMs: A Benchmark of Gend", "item": "https://sciencetostartup.com/paper/demographic-fairness-in-multimodal-llms-a-benchmark-of-gender-and-ethnicity-bias-in-face-verification" } ] } ] }

Competitive landscape

This research benchmarks demographic fairness in multimodal LLMs for face verification, revealing significant biases and performance disparities across different models and demographic groups.

Segment

Multimodal LLM Bias

Adoption evidence

No public code link in the paper record yet

Commercial read

4.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline