MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios

Dimensions overall score 7.0

• MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions.method 90%
  Evidencepartial

  MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions...

  Implicationpartial

  Specific numeric details are provided in the abstract and a data table.

  Verificationpartial

  partial

• MDPBench is the first benchmark for multilingual digital and photographed document parsing.method 95%
  Evidencepartial

  We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing.

  Implicationpartial

  Explicitly stated in the abstract and title as a primary contribution.

  Verificationpartial

  partial

• Existing document parsing benchmarks predominantly focus on digital-born and scanned documents in limited languages, leading to model bias.limitation 85%
  Evidencepartial

  However, these benchmarks predominantly focus on digital-born and scanned documents in limited languages. This limitation leads existing document parsing models to exhibit a certain bias toward inputs from standardized, high-resource languages...

  Implicationpartial

  Directly stated in the analysis section, supported by a comparison table of benchmarks.

  Verificationpartial

  partial

• Open-source document parsing models suffer a dramatic performance collapse on non-Latin scripts and real-world photographed documents, with an average drop of 17.8% on photographed documents and 14.0% on non-Latin scripts.result 90%
  Evidencepartial

  ...open-source alternatives suffer dramatic performance collapse, particularly on non-Latin scripts and real-world photographed documents, with an average drop of 17.8% on photographed documents and 14.0% on non-Latin scripts.

  Implicationpartial

  Specific performance drop percentages are stated in the abstract, and the result is highlighted as a key finding.

  Verificationpartial

  partial

• Closed-source models (notably Gemini3-Pro) prove relatively robust compared to open-source alternatives on the MDPBench evaluation.result 85%
  Evidencepartial

  Our comprehensive evaluation of both open-source and closed-source models uncovers a striking finding: while closed-source models (notably Gemini3-Pro) prove relatively robust, open-source alternatives suffer dramatic performance collapse...

  Implicationpartial

  Directly stated in the abstract and supported by a results table showing performance metrics.

  Verificationpartial

  partial

• The annotation pipeline for MDPBench combines automatic labeling by expert models with manual correction and human verification to ensure high quality.method 90%
  Evidencepartial

  ...with high-quality annotations produced through a rigorous pipeline of expert model labeling, manual correction, and human verification.

  Implicationpartial

  Explicitly described in the abstract and detailed in a pipeline diagram.

  Verificationpartial

  partial

• Photographed documents in MDPBench were captured under diverse real-world conditions including indoor/outdoor scenes, physical deformation, and varying camera orientations.method 85%
  Evidencepartial

  ...captured them under diverse environments and conditions, including indoor and outdoor scenes, physical deformation, image degradation, varying camera orientations, and background variation.

  Implicationpartial

  Specific conditions are listed in the methodology description.

  Verificationpartial

  partial

• Current document parsing models have key limitations including challenges with photographed documents, limited recognition of non-Latin scripts, and language-specific reading order issues.limitation 80%
  Evidencepartial

  We further analyze the key limitations of current document parsing models, including challenges with photographed documents, limited recognition of non-Latin scripts, language-specific reading order issues...

  Implicationpartial

  Summarized as findings from the evaluation, though specific evidence for each sub-point is spread throughout the analysis.

  Verificationpartial

  partial

Competitive landscape

A benchmark and evaluation of multilingual document parsing models reveals significant performance gaps, highlighting opportunities for more inclusive and deployment-ready parsing systems.