Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs

Dimensions overall score 7.0

• FSAR-LLaVA is the first end-to-end method to leverage MLLMs as a multimodal knowledge base for directly enhancing Few-shot Action Recognition (FSAR).method 95%
  Evidencepartial

  In this paper, we propose FSAR-LLaVA, the first end-to-end method to leverage MLLMs (such as Video-LLaVA) as a multimodal knowledge base for directly enhancing FSAR.

  Implicationpartial

  This is explicitly stated in the abstract and introduction.

  Verificationpartial

  partial

• The proposed FSAR-LLaVA method extracts spatiotemporally and semantically enriched representations using the MLLM's multimodal decoder.method 95%
  Evidencepartial

  First, at the feature level, we leverage the MLLM's multimodal decoder to extract spatiotemporally and semantically enriched representations, which are then decoupled and enhanced by our Multimodal Feature-Enhanced Module into distinct visual and textual features that fully exploit their semantic knowledge for FSAR.

  Implicationpartial

  This is clearly described as the first step in the method in the abstract and introduction.

  Verificationpartial

  partial

• FSAR-LLaVA decouples multimodal features into distinct visual and textual features using a Multimodal Feature-Enhanced Module.method 95%
  Evidencepartial

  which are then decoupled and enhanced by our Multimodal Feature-Enhanced Module into distinct visual and textual features that fully exploit their semantic knowledge for FSAR.

  Implicationpartial

  This is a core component of the proposed method, as described in the abstract and introduction.

  Verificationpartial

  partial

• The Composite Task-Oriented Prototype Construction module in FSAR-LLaVA is designed to bridge the distribution gap between meta-train and meta-test sets by leveraging MLLM outputs.method 95%
  Evidencepartial

  Next, we leverage the versatility of MLLMs to craft input prompts that flexibly adapt to diverse scenarios, and use their aligned outputs to drive our designed Composite Task-Oriented Prototype Construction, effectively bridging the distribution gap between meta-train and meta-test sets.

  Implicationpartial

  The abstract and introduction clearly state the purpose and inspiration for this module.

  Verificationpartial

  partial

• FSAR-LLaVA introduces a training-free Multimodal Prototype Matching Metric that adaptively selects decisive cues and leverages decoupled feature representations.method 95%
  Evidencepartial

  Finally, to enable multimodal features to guide metric learning jointly, we introduce a training-free Multimodal Prototype Matching Metric that adaptively selects the most decisive cues and efficiently leverages the decoupled feature representations produced by MLLMs.

  Implicationpartial

  This is presented as a key contribution and final component of the method in the abstract.

  Verificationpartial

  partial

• FSAR-LLaVA achieves superior performance across various tasks with minimal trainable parameters.result 95%
  Evidencepartial

  Extensive experiments demonstrate superior performance across various tasks with minimal trainable parameters.

  Implicationpartial

  This is a key result highlighted in the abstract and conclusion.

  Verificationpartial

  partial

• FSAR-LLaVA achieves state-of-the-art performance on the HMDB51 5-way 1-shot task using a fixed input prompt without additional textual label information.result 90%
  Evidencepartial

  Our FSAR-LLaVA Unknown, which uses the fixed input prompt: “What’s the action of the video?” without introducing additional textual label information, fully leverages the multimodal features of MLLM and achieves state-of-the-art performance that requires minimal parameters, as depicted in part (d), which refers to the performance comparison in the HMDB51 5-way 1-shot task.

  Implicationpartial

  This is stated as a specific achievement and supported by a figure reference.

  Verificationpartial

  partial

Competitive landscape

Leverage multimodal large language models to directly enhance few-shot action recognition with enriched representations and adaptive prompting.