ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

Dimensions overall score 7.0

• ClipTTT adapts Large Vision-Language Models (LVLMs) under degraded conditions on the fly using a single test sample.method 95%
  Evidencepartial

  We propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample.

  Implicationpartial

  This is a core claim stated directly in the abstract and elaborated in the introduction and method sections.

  Verificationpartial

  partial

• ClipTTT leverages the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets.method 95%
  Evidencepartial

  Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs.

  Implicationpartial

  This describes the key mechanism of the proposed method, explicitly stated in the abstract and detailed in the method section.

  Verificationpartial

  partial

• ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.result 95%
  Evidencepartial

  demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.

  Implicationpartial

  This is a primary result claim, stated in the abstract and supported by experimental descriptions.

  Verificationpartial

  partial

• Visual corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications.technical 90%
  Evidencepartial

  We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications.

  Implicationpartial

  This is a foundational observation that motivates the proposed method, stated in the abstract and reinforced in the introduction.

  Verificationpartial

  partial

• ClipTTT was evaluated on standard hallucination benchmarks with 15 common corruptions.method 95%
  Evidencepartial

  Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.

  Implicationpartial

  This describes the experimental setup and scope, explicitly mentioned in the abstract and introduction.

  Verificationpartial

  partial

• The ClipTTT framework uses a student-teacher model for on-the-fly adaptation, where the teacher generates caption candidates and CLIP scores them to select a pseudo-label for the student.method 95%
  Evidencepartial

  Foreach single corrupted test input, we employ a student-teacher framework for on-the-fly adaptation.(1)The Teacher model generatesndiverse caption candidates via sampling.(2)An external CLIP model scores each candidate, and the one with the highest visual-semantic alignment is selected as the pseudo-label.(3)The Student model is trained for one step on this pseudo-label, with gradients updating

  Implicationpartial

  This details the core components and workflow of the ClipTTT method, as illustrated in Figure 3 and described in the text.

  Verificationpartial

  partial

• Sentence-level averaging of CLIP scores is used to avoid truncation issues with CLIP's text encoder and ensure stable similarity estimates for longer captions.technical 90%
  Evidencepartial

  First, CLIP’s text encoder op-erates with a fixed token limit (77 tokens), and long captions may be truncated, leading to unstable similarity estimates. Sentence-level encoding avoids this issue by ensuring each segment remains within the token budget.

  Implicationpartial

  This explains a specific technical choice within the method and its rationale, as detailed in the text.

  Verificationpartial

  partial

• ClipTTT adapts Large Vision-Language Models (LVLMs) under degraded conditions on the fly using a single test sample.method 95%
  Evidencepartial

  We propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample.

  Implicationpartial

  This is a core claim stated directly in the abstract and elaborated in the introduction and method sections.

  Verificationpartial

  partial

• ClipTTT leverages the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets.technical 95%
  Evidencepartial

  Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs.

  Implicationpartial

  This is a key technical detail of the proposed method, explicitly stated in the abstract and detailed in the method section.

  Verificationpartial

  partial

• ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.result 95%
  Evidencepartial

  Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions.

  Implicationpartial

  This is a primary result claimed in the abstract and supported by experimental descriptions.

  Verificationpartial

  partial

• Visual corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications.technical 90%
  Evidencepartial

  We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications.

  Implicationpartial

  This is a foundational observation presented in the abstract that motivates the proposed method.

  Verificationpartial

  partial

• ClipTTT enables rapid adaptation without altering the base LVLMs.method 90%
  Evidencepartial

  enabling rapid adaptation without altering the base LVLMs.

  Implicationpartial

  This is a significant benefit of the proposed method, highlighted in the abstract.

  Verificationpartial

  partial

Competitive landscape

A method to adapt large vision-language models on the fly to reduce hallucinations caused by visual input corruption, using CLIP as a guidance signal.