Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time

Dimensions overall score 7.0

• EcoG-Bench is a new benchmark for evaluating multimodal models on their ability to ground speech with co-speech gestures in egocentric videos, revealing a significant performance gap compared to human accuracy and highlighting the importance of temporal alignment cues. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction.problem 35%
  Evidencepartial

  EcoG-Bench is a new benchmark for evaluating multimodal models on their ability to ground speech with co-speech gestures in egocentric videos, revealing a significant performance gap compared to human accuracy and highlighting the importance of temporal alignment cues. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction.method 35%
  Evidencepartial

  In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• ScienceToStartup currently rates this 7.0/10 on the public viability pass. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}).result 35%
  Evidencepartial

  ScienceToStartup currently rates this 7.0/10 on the public viability pass. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}).

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• Multimodal Understanding moved forward this cycle; last verified April 2026. Public score 7.0/10.why now 35%
  Evidencepartial

  Multimodal Understanding moved forward this cycle; last verified April 2026. Public score 7.0/10.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

Competitive landscape

EcoG-Bench is a new benchmark for evaluating multimodal models on their ability to ground speech with co-speech gestures in egocentric videos, revealing a significant performance gap compared to human accuracy and highlighting the importance of temporal alignment cues.