JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding

Dimensions overall score 7.0

• JaWildText is a new benchmark containing 3,241 instances from 2,961 images with 1.12 million annotated characters spanning 3,643 unique character types for evaluating Japanese scene text understanding.method 95%
  Evidencepartial

  JaWildText contains 3,241 instances from 2,961 images newly captured in Japan, with 1.12 million annotated characters spanning 3,643 unique character types.

  Implicationpartial

  Explicitly stated in the abstract with specific numeric details.

  Verificationpartial

  partial

• The best evaluated vision-language model achieves an average score of only 0.64 across the three tasks in JaWildText, indicating robust Japanese text understanding in the wild remains unsolved.result 90%
  Evidencepartial

  We evaluate 14 open-weight VLMs and find that the best model achieves an average score of 0.64 across the three tasks.

  Implicationpartial

  Directly stated in the abstract and conclusion with clear numeric result.

  Verificationpartial

  partial

• Error analysis shows recognition remains the dominant bottleneck for VLMs on Japanese scene text, with kanji posing a particular challenge.result 90%
  Evidencepartial

  Error analyses show recognition remains the dominant bottleneck, especially for kanji.

  Implicationpartial

  Explicitly stated in both abstract and conclusion.

  Verificationpartial

  partial

• In the Receipt KIE task, format-constrained fields like 'time' are relatively well handled (accuracy above 0.90 for top models), while 'store_name' and 'store_address' remain difficult with best accuracies of only 0.16 and 0.55 respectively.result 85%
  Evidencepartial

  Format-constrained fields such as time are relatively well handled, with several top models achieving accuracy above 0.90. In contrast, store_name and store_address remain difficult, with best accuracies of only 0.16 and 0.55, respectively

  Implicationpartial

  Directly stated with specific numeric evidence from results table.

  Verificationpartial

  partial

• Qwen3-VL-8B outperforms InternVL3.5-38B by 0.09 Overall score, indicating architecture and training data composition can matter more than parameter count alone.result 80%
  Evidencepartial

  Qwen3-VL-8B surpasses InternVL3.5-38B by 0.09 Overall, indicating that architecture and training data composition can matter more than parameter count alone.

  Implicationpartial

  Direct comparison with specific numeric difference and interpretation provided.

  Verificationpartial

  partial

• The best OCR-specialized model (olmOCR-2-7B, score 0.74) falls below the best general-purpose VLM (Qwen3-VL-8B, score 0.79) on Handwriting OCR at comparable parameter scales.result 85%
  Evidencepartial

  the best OCR-specialized model (olmOCR-2-7B, 0.74) falls below the best general-purpose VLM (Qwen3-VL-8B, 0.79) at a comparable parameter scale.

  Implicationpartial

  Direct comparison with specific numeric results from evaluation table.

  Verificationpartial

  partial

• Existing Japanese visual text datasets have primarily focused on scanned documents, leaving in-the-wild scene text underexplored.limitation 85%
  Evidencepartial

  existing Japanese visual text datasets have primarily focused on scanned documents, leaving in-the-wild scene text underexplored.

  Implicationpartial

  Explicitly stated in the abstract as motivation for the benchmark.

  Verificationpartial

  partial

• Multilingual benchmarks that include Japanese often fail to capture language-specific complexities like mixed scripts, frequent vertical writing, and large character inventory.limitation 80%
  Evidencepartial

  Japanese scene text poses challenges that multilingual benchmarks often fail to capture, including mixed scripts, frequent vertical writing, and a character inventory far larger than the Latin alphabet.

  Implicationpartial

  Explicitly stated in abstract as a gap in existing resources.

  Verificationpartial

  partial

Competitive landscape

A new benchmark and evaluation code for vision-language models specifically designed to understand complex Japanese scene text, addressing a significant gap in current multilingual datasets.