The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

Dimensions overall score 4.0

• Smaller vision-language models (VLMs) exhibit performance gains of up to 58% F1 score when neuroimaging context is introduced, even when the MRI data carries no reliable individual-level diagnostic signal.result 95%
  Evidencepartial

  smaller VLMs exhibit gains of up to 58% F1 upon introduction of neuroimaging context

  Implicationpartial

  Explicitly stated in the abstract with a specific numeric metric (58% F1).

  Verificationpartial

  partial

• Merely mentioning MRI availability in the task prompt accounts for 70-80% of the performance shift observed in VLMs, independent of whether the actual imaging data is present.result 95%
  Evidencepartial

  merely mentioning MRI availability in the task prompt accounts for 70-80% of this shift, independent of whether imaging data is present

  Implicationpartial

  Directly stated in the abstract with a precise percentage range (70-80%).

  Verificationpartial

  partial

• The observed performance shift driven by prompt framing, termed the 'scaffold effect', is a domain-specific instance of modality collapse in clinical VLMs.technical 90%
  Evidencepartial

  a domain-specific instance of modality collapse we term the scaffold effect

  Implicationpartial

  The term 'scaffold effect' is explicitly defined in the abstract as the cause of the performance shift.

  Verificationpartial

  partial

• Expert evaluation reveals that VLMs fabricate neuroimaging-grounded justifications for their classifications across all experimental conditions.limitation 85%
  Evidencepartial

  Expert evaluation reveals fabrication of neuroimaging-grounded justifications across all conditions

  Implicationpartial

  Strongly stated in the abstract, though the specific details of the 'expert evaluation' are elaborated in the analysis excerpt.

  Verificationpartial

  partial

• Applying preference alignment to VLMs, while eliminating MRI-referencing behavior, causes model performance to collapse toward a random baseline.result 85%
  Evidencepartial

  preference alignment, while eliminating MRI-referencing behavior, collapses both conditions toward random baseline

  Implicationpartial

  Directly stated in the abstract as a key finding.

  Verificationpartial

  partial

• The FOR2107 dataset represents a genuinely hard classification problem where apparent VLM gains warrant scrutiny, as prior work found accuracies of only 54–56% with neuroimaging markers.method 90%
  Evidencepartial

  Prior work on this cohort using classical ML pipelines found classification accuracies of only 54–56% with univariate neuroimaging markers

  Implicationpartial

  Explicitly stated in the methodology section with a citation to prior work and specific accuracy figures.

  Verificationpartial

  partial

• Distilled VLMs can become competitive with counterparts that are an order of magnitude larger when evaluated under conditions that induce the scaffold effect.result 80%
  Evidencepartial

  with distilled models becoming competitive with counterparts an order of magnitude larger

  Implicationpartial

  Directly stated in the abstract, though the mechanism (the scaffold effect) is implied as the enabling condition.

  Verificationpartial

  partial

• Surface evaluations (e.g., standard benchmark scores) are inadequate indicators of genuine multimodal reasoning in clinical VLMs.limitation 90%
  Evidencepartial

  Our findings demonstrate that surface evaluations are inadequate indicators of multimodal reasoning

  Implicationpartial

  This is the central conclusion of the paper, explicitly stated in the abstract.

  Verificationpartial

  partial

Competitive landscape

This research reveals a critical 'scaffold effect' in clinical vision-language models, where prompt framing, not actual data integration, drives apparent performance gains, highlighting a significant risk for clinical deployment.