AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation

Dimensions overall score 7.0

• AIBench is the first benchmark to use VQA for evaluating logic correctness and VLMs for assessing aesthetics in academic illustration generation.method 90%
  Evidencepartial

  we propose AIBench, the first benchmark using VQA for evaluating logic correctness of the academic illustrations and VLMs for assessing aesthetics.

  Implicationpartial

  Explicitly stated in the abstract as a primary contribution.

  Verificationpartial

  partial

• The VQA-based evaluation approach in AIBench is more accurate and detailed for visual-logical consistency and relies less on the judger VLM's ability compared to direct VLM-as-Judge methods.technical 85%
  Evidencepartial

  Our VQA-based approach raises more accurate and detailed evaluations on visual-logical consistency while relying less on the ability of the judger VLM.

  Implicationpartial

  Directly stated in the abstract as an advantage of the proposed method.

  Verificationpartial

  partial

• The performance gap between state-of-the-art models on the academic illustration task is significantly larger than on general image generation tasks.result 80%
  Evidencepartial

  we conduct extensive experiments and conclude that the performance gap between models on this task is significantly larger than general ones

  Implicationpartial

  Strongly supported by conclusion in the abstract, though specific numeric evidence is not provided in the excerpt.

  Verificationpartial

  partial

• Logic and aesthetics in academic illustration generation are hard to optimize simultaneously, presenting a trade-off similar to handcrafted illustrations.result 85%
  Evidencepartial

  Moreover, we conclude that aesthetics and logic are somewhat of a trade-off, which also exists in handcrafted illustrations

  Implicationpartial

  Explicitly stated as a conclusion from experiments.

  Verificationpartial

  partial

• Test-time scaling strategies on both reasoning ability and high-density generation ability can significantly boost model performance on academic illustration generation.result 80%
  Evidencepartial

  test-time scaling on both abilities significantly boosts the performance on this task.

  Implicationpartial

  Directly stated in the abstract and analysis as a key finding.

  Verificationpartial

  partial

• AIBench constructs QA pairs across four hierarchical levels (Component, Topology, Phase, Semantic) to enable multi-granularity assessment of visual-logical consistency.method 90%
  Evidencepartial

  we designed four levels of questions... which query whether the generated illustration aligns with the paper on different scales.

  Implicationpartial

  Explicitly described in the framework overview with specific percentage breakdowns provided.

  Verificationpartial

  partial

• The VLM-as-Judge paradigm for evaluating academic illustrations introduces 'metric ambiguity' by conflating objective logical errors with subjective aesthetic flaws.limitation 85%
  Evidencepartial

  This introduces 'metric ambiguity' by conflating objective logical errors with subjective aesthetic flaws

  Implicationpartial

  Directly stated as a limitation of existing approaches that AIBench addresses.

  Verificationpartial

  partial

• Existing academic illustration generation methods like PaperBanana typically condition on limited inputs (e.g., method excerpts and captions), which can encourage style imitation while missing fine-grained technical details.limitation 80%
  Evidencepartial

  it typically conditions on limited inputs (e.g., method excerpts and captions), which can encourage style imitation while missing fine-grained technical details.

  Implicationpartial

  Direct criticism of prior work stated in the analysis section.

  Verificationpartial

  partial

Competitive landscape

AIBench provides the first benchmark for evaluating the visual-logical consistency of academic illustrations generated by AI, revealing significant gaps in current models and guiding future development.