HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

Dimensions overall score 7.0

• Current vision-language models (VLMs) struggle with fine-grained spatial reasoning, particularly in interpreting complex and articulated hand poses.limitation 95%
  Evidencepartial

  Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses.

  Implicationpartial

  This is a central theme of the abstract and is directly supported by the introduction of the HandVQA benchmark to address this issue.

  Verificationpartial

  partial

• HandVQA is a large-scale diagnostic benchmark containing over 1.6 million controlled multiple-choice questions designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering.method 95%
  Evidencepartial

  We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions.

  Implicationpartial

  The abstract explicitly states the creation and scale of the HandVQA benchmark and its purpose.

  Verificationpartial

  partial

• The HandVQA benchmark probes spatial relationships between hand joints, including angles, distances, and relative positions.method 95%
  Evidencepartial

  our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions.

  Implicationpartial

  The abstract clearly defines the types of spatial relationships evaluated by the HandVQA benchmark.

  Verificationpartial

  partial

• State-of-the-art VLMs like LLaVA, DeepSeek, and Qwen-VL exhibit systematic limitations in fine-grained spatial reasoning, including hallucinated finger parts and incorrect geometric interpretations.limitation 90%
  Evidencepartial

  Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization.

  Implicationpartial

  The abstract summarizes the findings from evaluating these models on HandVQA, highlighting their failures.

  Verificationpartial

  partial

• Lightweight fine-tuning via LoRA on the HandVQA benchmark can significantly improve VLM accuracy on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).result 95%
  Evidencepartial

  We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).

  Implicationpartial

  The abstract provides specific quantitative improvements achieved through fine-tuning on HandVQA for downstream tasks.

  Verificationpartial

  partial

• Base VLMs, such as LLaVA and Qwen, perform below random chance (33.3%) on distance-based pose descriptors, with high Mean Absolute Error (MAE).result 90%
  Evidencepartial

  As T able 2 shows, base VLMs generally seem to perform poorly on distance pose descriptor with LLaV A and Qwen performing well below the accuracy of 33.3% accuracy that would have been achieved via random choice. Even the MAE remains high for two of these base models with the lowest MAE being 1.208 for Qwen on the FreiHAND dataset.

  Implicationpartial

  Table 2 in the provided text explicitly shows accuracy below 33.3% for LLaVA and Qwen on distance descriptors and mentions high MAE.

  Verificationpartial

  partial

• VLMs struggle with grasping angles, with some base models performing substantially higher than random chance (25%) but still indicating a weakness.limitation 85%
  Evidencepartial

  According to T able 2, the performance of base VLMs across datasets excluding FPHA is generally substantially higher than the accuracy of 25% that would have been achieved via random choice, with the lowest being 34.10% for DeepSeek on the InterHand2.6M

  Implicationpartial

  The text mentions that base VLMs perform better than random chance on angles but still implies a struggle, suggesting room for improvement.

  Verificationpartial

  partial

• HandVQA is a large-scale diagnostic benchmark with over 1.6 million controlled multiple-choice questions designed to evaluate fine-grained spatial reasoning about hands in vision-language models.method 95%
  Evidencepartial

  We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions.

  Implicationpartial

  The abstract explicitly states the creation and scale of the HandVQA benchmark, and the analysis section mentions 'over 1.6 million questions'.

  Verificationpartial

  partial

• Current vision-language models (VLMs) exhibit systematic limitations in fine-grained spatial reasoning about hands, including hallucinated finger parts and incorrect geometric interpretations.limitation 90%
  Evidencepartial

  Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization.

  Implicationpartial

  The abstract and analysis sections clearly state the limitations found in current VLMs when evaluated on the HandVQA benchmark.

  Verificationpartial

  partial

• Base VLMs perform poorly on distance-based spatial reasoning tasks, with some models achieving accuracy below random chance (33.3%).result 90%
  Evidencepartial

  VLMs struggle to grasp distance between joints.As T able 2 shows, base VLMs generally seem to perform poorly on dis- tance pose descriptor with LLaV A and Qwen performing well below the accuracy of 33.3% accuracy that would have been achieved via random choice.

  Implicationpartial

  The analysis section provides specific details about the poor performance of base VLMs on distance descriptors, including accuracy below random choice.

  Verificationpartial

  partial

• Fine-tuning VLMs using the HandVQA benchmark improves their accuracy on novel downstream tasks like hand gesture recognition by +10.33% and hand-object interaction by +2.63% in a zero-shot setting.result 95%
  Evidencepartial

  We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).

  Implicationpartial

  The abstract explicitly states the performance improvement on downstream tasks after fine-tuning with HandVQA, including specific percentage gains.

  Verificationpartial

  partial

• The HandVQA benchmark is constructed using precise 3D annotations from widely-used datasets: FreiHAND, InterHand2.6M, and FPHA.method 95%
  Evidencepartial

  HandVQA is constructed using precise 3D annotations from widely-used datasets—FreiHAND [ 70], InterHand2.6M [46], and FPHA [20].

  Implicationpartial

  The abstract and analysis section clearly state the origin of the 3D annotations used for constructing HandVQA.

  Verificationpartial

  partial

Competitive landscape

A diagnostic benchmark and fine-tuning method to significantly improve vision-language models' spatial reasoning about human hands, enabling applications in robotics and AR/VR.