GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

Dimensions overall score 7.0

• GUIDE is a training-free, plug-and-play framework designed to resolve domain bias in GUI agents by acquiring expertise from web tutorial videos.method 95%
  Evidencepartial

  In this paper, we present GUIDE (GUI Unbiasing via Instructional-Video Driven Expertise), a training-free, plug-and-play framework that resolves GUI agent domain bias by autonomously acquiring domain-specific expertise from web tutorial videos through a retrieval-augmented automated annotation pipeline.

  Implicationpartial

  This is a core statement of the paper's contribution, explicitly mentioned in the abstract and introduction.

  Verificationpartial

  partial

• GUIDE utilizes a subtitle-driven Video-RAG pipeline for progressive three-stage retrieval: domain classification, topic extraction, and relevance matching, to identify task-relevant tutorial videos.method 95%
  Evidencepartial

  First, a subtitle-driven Video-RAG pipeline unlocks video semantics through subtitle analysis, performing progressive three-stage retrieval—domain classification, topic extraction, and relevance matching—to identify task-relevant tutorial videos.

  Implicationpartial

  The abstract and introduction clearly describe the three stages of the Video-RAG pipeline as a key innovation.

  Verificationpartial

  partial

• GUIDE employs a fully automated annotation pipeline based on an inverse dynamics paradigm to infer planning and grounding knowledge from keyframes and UI element detection.method 95%
  Evidencepartial

  Second, a fully automated annotation pipeline built on an inverse dynamics paradigm feeds consecutive keyframes enhanced with UI element detection into VLMs, inferring the required planning and grounding knowledge that are injected into the agent's corresponding modules to address both manifestations of domain bias.

  Implicationpartial

  The abstract and introduction detail the automated annotation pipeline and its reliance on the inverse dynamics paradigm.

  Verificationpartial

  partial

• GUIDE consistently yields over 5% improvements and reduces execution steps on OSWorld without modifying any model parameters or architecture.result 95%
  Evidencepartial

  It consistently yields over 5% improvements and reduces execution steps—without modifying any model parameters or architecture—validating GUIDE as an architecture-agnostic enhancement to bridge GUI agent domain bias.

  Implicationpartial

  The abstract provides specific quantitative results and emphasizes the architecture-agnostic nature of the improvement.

  Verificationpartial

  partial

• GUIDE's retrieval process starts with over 50 candidate YouTube videos and progressively narrows them down through subtitle analysis.method 90%
  Evidencepartial

  From 50+ YouTube candidates, a metadata pre-filter removes outliers, then three subtitle-driven stages progressively narrow results

  Implicationpartial

  The figures and descriptions of the retrieval pipeline indicate an initial large set of candidates that are filtered.

  Verificationpartial

  partial

• The final selection of tutorial videos by GUIDE is limited to a top-K (K<=2) set.method 90%
  Evidencepartial

  Final top-K (K≤2) videos proceed to an

  Implicationpartial

  The figures and descriptions explicitly state that the final output of the retrieval is a small number of top videos.

  Verificationpartial

  partial

• GUIDE can be integrated as a plug-and-play component for both multi-agent systems and single-model GUI agents.method 95%
  Evidencepartial

  Extensive experiments on OSWorld demonstrate GUIDE's generality as a plug-and-play component for both multi-agent systems and single-model agents.

  Implicationpartial

  The abstract and introduction explicitly state the generality of GUIDE across different agent architectures.

  Verificationpartial

  partial

Competitive landscape

A plug-and-play framework that resolves GUI agent domain bias by acquiring expertise from web tutorial videos, improving performance without model retraining.