SHOE: Semantic HOI Open-Vocabulary Evaluation Metric

Dimensions overall score 7.0

• Standard evaluation metrics like mean Average Precision (mAP) fail to credit semantically valid but lexically different predictions in HOI detection.limitation 90%
  Evidencepartial

  fail to credit semantically valid but lexically different predictions (e.g., 'lean on couch' vs. 'sit on couch')

  Implicationpartial

  Directly stated in abstract with clear explanation of limitation

  Verificationpartial

  partial

• SHOE incorporates semantic similarity between predicted and ground-truth HOI labels by decomposing predictions into verb and object components.method 90%
  Evidencepartial

  SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs)

  Implicationpartial

  Directly stated in abstract with clear methodological description

  Verificationpartial

  partial

• SHOE scores align more closely with human judgments than existing metrics, achieving 85.73% agreement with average human ratings.result 95%
  Evidencepartial

  achieving an agreement of 85.73% with the average human ratings

  Implicationpartial

  Directly stated with specific numeric evidence in abstract

  Verificationpartial

  partial

• SHOE outperforms both LLM-based and embedding-based baselines in aligning with human judgments.result 85%
  Evidencepartial

  SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines

  Implicationpartial

  Directly stated in abstract but requires inference that 'existing metrics' includes these baselines

  Verificationpartial

  partial

• SHOE enables flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models.technical 80%
  Evidencepartial

  enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models

  Implicationpartial

  Directly stated but requires inference about scalability from context

  Verificationpartial

  partial

• SHOE can be applied to standard benchmarks such as HICO-DET for evaluating open-vocabulary predictions.technical 85%
  Evidencepartial

  using standard benchmarks such as HICO-DET

  Implicationpartial

  Directly stated with specific benchmark mentioned

  Verificationpartial

  partial

• Open-vocabulary HOI detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios.market 80%
  Evidencepartial

  Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios

  Implicationpartial

  Directly stated but represents a broader research direction rather than a specific claim about SHOE

  Verificationpartial

  partial

• Semantically grounded HOI evaluation better mirrors human understanding of interactions than categorical label-based approaches.technical 85%
  Evidencepartial

  Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions

  Implicationpartial

  Directly stated in abstract conclusion but requires some inference

  Verificationpartial

  partial

Competitive landscape

A new semantic evaluation metric for open-vocabulary human-object interaction detection that aligns better with human judgment, enabling more scalable and flexible assessment of AI models.