Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

Dimensions overall score 7.0

• Large Vision-Language Models (LVLMs) struggle with complex, multi-step reasoning when visual input serves as a clue rather than a direct answer.limitation 90%
  Evidencepartial

  However, a critical cognitive gap emerges when the visual input serves only as a clue rather than the answer. We identify that current models struggle with the complex, multi-step reasoning required to solve problems where information is not explicitly depicted.

  Implicationpartial

  Directly stated in abstract with clear description of the cognitive gap

  Verificationpartial

  partial

• Successfully solving a rebus puzzle requires extracting visual/textual attributes, retrieving linguistic prior knowledge, and performing abstract mapping to synthesize elements into meaning outside pixel space.technical 85%
  Evidencepartial

  Successfully solving a rebus puzzle requires a distinct cognitive workflow: the model must extract visual and textual attributes, retrieve linguistic prior knowledge (such as idioms), and perform abstract mapping to synthesize these elements into a meaning that exists outside the pixel space.

  Implicationpartial

  Directly stated in abstract as the required cognitive workflow

  Verificationpartial

  partial

• RebusBench contains 1,164 puzzles designed to evaluate neurosymbolic capability integrating perception and knowledge.method 95%
  Evidencepartial

  To evaluate this neurosymbolic capability, we introduce RebusBench, a benchmark of 1,164 puzzles designed to test this specific integration of perception and knowledge.

  Implicationpartial

  Explicit numeric count provided in abstract

  Verificationpartial

  partial

• State-of-the-art models (including Qwen, InternVL, and LLaVA) achieve below 10% Exact Match and 20% semantic accuracy on RebusBench.result 90%
  Evidencepartial

  Our evaluation of state-of-the-art models (including Qwen, InternVL, and LLaVA) shows a severe deficiency: performance saturates below 10% Exact Match and 20% semantic accuracy

  Implicationpartial

  Clear numeric performance metrics stated in abstract

  Verificationpartial

  partial

• No significant improvement is observed from model scaling or In-Context Learning (ICL) on rebus puzzle performance.result 85%
  Evidencepartial

  with no significant improvement observed from model scaling or In-Context Learning (ICL).

  Implicationpartial

  Directly stated in abstract as a finding

  Verificationpartial

  partial

• Models possess necessary visual and linguistic components but lack the cognitive reasoning glue to connect them.limitation 80%
  Evidencepartial

  These findings suggest that while models possess the necessary visual and linguistic components, they lack the cognitive reasoning glue to connect them.

  Implicationpartial

  Directly stated conclusion in abstract, though slightly interpretive

  Verificationpartial

  partial

• Current LVLMs have achieved remarkable proficiency in explicit visual recognition and describing what is directly visible.technical 85%
  Evidencepartial

  Large Vision-Language Models (LVLMs) have achieved remarkable proficiency in explicit visual recognition, effectively describing what is directly visible in an image.

  Implicationpartial

  Directly stated in abstract as background context

  Verificationpartial

  partial

Competitive landscape

A new benchmark and evaluation framework for visual reasoning tasks that current state-of-the-art models fail at, highlighting a critical gap in cognitive integration.