Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• The CARV benchmark is the first diagnostic benchmark for compositional analogical reasoning in multimodal LLMs.method 90%
• State-of-the-art MLLMs perform far below human-level performance on the CARV benchmark, with Gemini-2.5 Pro achieving only 40.4% accuracy.result 95%
• A major bottleneck for closed-source MLLMs on compositional analogical reasoning is the decomposition stage, where they struggle to abstract visual changes into symbolic rules.limitation 90%
• The performance of MLLMs on the CARV task drops as the number of atomic transformations (N) in the image pairs increases.result 90%
• MLLMs struggle more with compositional analogical reasoning in the 'Different Source' context compared to the 'Shared Source' context.limitation 85%
• For open-source MLLMs, the major bottleneck on the CARV task is the perception stage, not decomposition.limitation 85%
• Combinations of properties like subject, number, and position contribute most to model failure on the CARV task.result 80%
• Existing evaluations of analogical reasoning in MLLMs overlook the ability to compose rules from multiple sources.limitation 90%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card