Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation

Dimensions overall score 3.0

• Repeated 3-fold subject-exclusive cross-validation on BP4D+ produces an empirical noise floor of ±0.065 in average F1 score.result 95%
  Evidencepartial

  repeated 3-fold subject-exclusive splits produce an empirical noise floor of $\pm 0.065$ in average F1

  Implicationpartial

  Directly stated in abstract with specific numeric value and dataset reference

  Verificationpartial

  partial

• Cross-validation introduces measurable stochastic variance in facial Action Unit detection evaluation.limitation 90%
  Evidencepartial

  cross-validation itself introduces measurable stochastic variance

  Implicationpartial

  Explicitly stated as a main finding in the abstract

  Verificationpartial

  partial

• Operating-point metrics such as F1 fluctuate more than threshold-independent measures such as AUC in cross-validation.technical 85%
  Evidencepartial

  Operating-point metrics such as F1 fluctuate more than threshold-independent measures such as AUC

  Implicationpartial

  Directly stated in abstract with comparison between metric types

  Verificationpartial

  partial

• Model ranking for AU detection can change under different fold assignments in cross-validation.limitation 85%
  Evidencepartial

  model ranking can change under different fold assignments

  Implicationpartial

  Directly stated in abstract as a consequence of cross-validation variance

  Verificationpartial

  partial

• Leave-One-Dataset-Out (LODO) cross-validation yields more stable and interpretable findings than single-dataset cross-validation.method 90%
  Evidencepartial

  Leave-one-dataset-out cross-validation yields more stable and interpretable findings

  Implicationpartial

  Direct conclusion stated in abstract with rationale

  Verificationpartial

  partial

• LODO evaluation exposes domain-level instability that is not visible under single-dataset cross-validation.result 85%
  Evidencepartial

  LODO removes partition randomness and exposes domain-level instability that is not visible under single-dataset cross-validation

  Implicationpartial

  Directly stated in abstract with specific comparison between protocols

  Verificationpartial

  partial

• Gains often reported in cross-fold validation for AU detection may fall within protocol variance.limitation 80%
  Evidencepartial

  gains often reported in cross-fold validation may fall within protocol variance

  Implicationpartial

  Strongly implied conclusion based on the demonstrated variance, though not explicitly quantified

  Verificationpartial

  partial

• Variation in AU detection performance is substantially larger for low-prevalence Action Units than for average F1.result 80%
  Evidencepartial

  with substantially larger variation for low-prevalence AUs

  Implicationpartial

  Directly stated in abstract with comparative language

  Verificationpartial

  partial

Competitive landscape

A novel evaluation protocol for facial action unit detection that quantifies noise and improves robustness, revealing that many reported gains may be artifacts of the testing method.