CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Dimensions overall score 7.0

• The CARV benchmark is the first diagnostic benchmark for compositional analogical reasoning in multimodal LLMs.method 90%
  Evidencepartial

  To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark.

  Implicationpartial

  Explicitly stated in the abstract as a novel task and 'the first diagnostic benchmark'.

  Verificationpartial

  partial

• State-of-the-art MLLMs perform far below human-level performance on the CARV benchmark, with Gemini-2.5 Pro achieving only 40.4% accuracy.result 95%
  Evidencepartial

  Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%.

  Implicationpartial

  Directly stated in the abstract with a specific numeric result from Table 2.

  Verificationpartial

  partial

• A major bottleneck for closed-source MLLMs on compositional analogical reasoning is the decomposition stage, where they struggle to abstract visual changes into symbolic rules.limitation 90%
  Evidencepartial

  For closed-source models, they mostly fail at the decomposition stage, suggesting that while these models can percept the changes, they struggle to abstract the changes into symbolic rules.

  Implicationpartial

  Directly stated in the analysis section (Q1) with reference to Figure 3a.

  Verificationpartial

  partial

• The performance of MLLMs on the CARV task drops as the number of atomic transformations (N) in the image pairs increases.result 90%
  Evidencepartial

  The performance drops as the number of atomic transformations (N) increases.

  Implicationpartial

  Strongly supported by the analysis in Figure 6 and its caption.

  Verificationpartial

  partial

• MLLMs struggle more with compositional analogical reasoning in the 'Different Source' context compared to the 'Shared Source' context.limitation 85%
  Evidencepartial

  Our results indicate MLLMs generally struggle with the generalization ability, and get worse when switch from Shared Source to Different Source setting.

  Implicationpartial

  Directly stated in the analysis (Q3) and supported by performance drops in Table 2.

  Verificationpartial

  partial

• For open-source MLLMs, the major bottleneck on the CARV task is the perception stage, not decomposition.limitation 85%
  Evidencepartial

  For open-source models, the challenge shifts into perception.

  Implicationpartial

  Directly stated in the analysis (Q1), contrasting with the finding for closed-source models.

  Verificationpartial

  partial

• Combinations of properties like subject, number, and position contribute most to model failure on the CARV task.result 80%
  Evidencepartial

  For most models, combinations among subject, number, and position contribute most to the failure.

  Implicationpartial

  Stated in the caption of Figure 4, though the evidence quote is from the surrounding text.

  Verificationpartial

  partial

• Existing evaluations of analogical reasoning in MLLMs overlook the ability to compose rules from multiple sources.limitation 90%
  Evidencepartial

  Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence.

  Implicationpartial

  Explicitly stated in the abstract as the motivation for the work.

  Verificationpartial

  partial

Competitive landscape

A new benchmark and dataset to diagnose and improve compositional analogical reasoning in multimodal LLMs, revealing significant performance gaps compared to human capabilities.