Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

Dimensions overall score 4.0

• A multimodal system for blended emotion recognition that leverages late fusion of specialized encoders, including a novel application of Gemini Embedding 2.0 for competitive accuracy with short video inputs. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input.problem 35%
  Evidencepartial

  A multimodal system for blended emotion recognition that leverages late fusion of specialized encoders, including a novel application of Gemini Embedding 2.0 for competitive accuracy with short video inputs. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• We present our system for the BLEMORE Challenge at FG 2026 on blended emotion recognition with relative salience prediction. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input.method 35%
  Evidencepartial

  We present our system for the BLEMORE Challenge at FG 2026 on blended emotion recognition with relative salience prediction. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• ScienceToStartup currently rates this 4.0/10 on the public viability pass. Our 12-encoder system achieves Score = 0.279 (ACCP = 0.391, ACCS = 0.168) on the test set, placing 6th.result 35%
  Evidencepartial

  ScienceToStartup currently rates this 4.0/10 on the public viability pass. Our 12-encoder system achieves Score = 0.279 (ACCP = 0.391, ACCS = 0.168) on the test set, placing 6th.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• Multimodal Emotion Recognition moved forward this cycle; last verified April 2026. Public score 4.0/10.why now 35%
  Evidencepartial

  Multimodal Emotion Recognition moved forward this cycle; last verified April 2026. Public score 4.0/10.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

Competitive landscape

A multimodal system for blended emotion recognition that leverages late fusion of specialized encoders, including a novel application of Gemini Embedding 2.0 for competitive accuracy with short video inputs.