AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• AMIGO is a long-horizon benchmark for hidden-target identification over galleries of visually similar images.method 95%
  Evidencepartial

  We introduce AMIGO (Agentic Multi-Image Grounding Oracle Benchmark), a long-horizon benchmark for hidden-target identification over galleries of visually similar images.

  Implicationpartial

  Explicitly stated in the abstract and introduction as the core contribution.

  Verificationpartial

  partial

• AMIGO evaluates agentic VLMs on their ability to actively acquire missing information through constrained questioning, unlike benchmarks that assess solving presented problems.method 85%
  Evidencepartial

  In contrast, AMIGO evaluates whether an agentic VLM can acquire the missing information needed to solve the task.

  Implicationpartial

  Directly contrasted with prior benchmarks in the analysis, stating AMIGO's distinct focus.

  Verificationpartial

  partial

• The AMIGO protocol uses a constrained action space where the model asks attribute-focused Yes/No/Unsure questions and is penalized for invalid actions with Skip.method 90%
  Evidencepartial

  In AMIGO, the oracle privately selects a target image, and the model must recover it by asking a sequence of attribute-focused Yes/No/Unsure questions under a strict protocol that penalizes invalid actions with Skip.

  Implicationpartial

  Clearly described in the abstract and detailed in the example figure and its constraints.

  Verificationpartial

  partial

• AMIGO supports controlled oracle imperfections to probe model robustness and verification behavior under inconsistent feedback.method 90%
  Evidencepartial

  AMIGO also supports controlled oracle imperfections to probe robustness and verification behavior under inconsistent feedback.

  Implicationpartial

  Explicitly stated in the abstract and repeated in the analysis excerpt.

  Verificationpartial

  partial

• AMIGO is instantiated with a 'Guess My Preferred Dress' task.method 95%
  Evidencepartial

  We instantiate AMIGO with Guess My Preferred Dress task

  Implicationpartial

  Directly and explicitly stated in the abstract and analysis.

  Verificationpartial

  partial

• AMIGO evaluates interaction quality via metrics like evidence verification, efficiency, protocol compliance, and noise tolerance, not just final success.method 85%
  Evidencepartial

  report metrics covering both outcomes and interaction quality, including identification success, evidence verification, efficiency, protocol compliance, noise tolerance, and trajectory-level diagnostics.

  Implicationpartial

  Explicitly listed in the abstract, indicating a comprehensive evaluation framework.

  Verificationpartial

  partial

• AMIGO differs from related interactive clarification benchmarks by focusing on a closed set of visually similar images and evaluating the interaction policy via trajectory-level signals.technical 80%
  Evidencepartial

  AMIGO differs in three ways. First, it studies hidden-target identification over a closed set of visually similar images... Second, it evaluates the interaction policy itself via trajectory-level signals...

  Implicationpartial

  Directly stated in the analysis section comparing AMIGO to related work, though some inference is needed to connect the full difference.

  Verificationpartial

  partial

• Most current evaluations for agentic vision-language models still focus on single-image, single-turn correctness, which is a limitation AMIGO addresses.limitation 90%
  Evidencepartial

  Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn correctness.

  Implicationpartial

  Directly stated as the motivation in the opening sentence of the abstract.

  Verificationpartial

  partial

Startup potential card