ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Dimensions overall score 7.0

• Implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models.result 90%
  Evidencepartial

  implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models

  Implicationpartial

  Directly stated in abstract with clear numeric comparison

  Verificationpartial

  partial

• Existing benchmarks using name-based proxies carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status.limitation 85%
  Evidencepartial

  Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status

  Implicationpartial

  Directly stated in abstract as motivation for new benchmark

  Verificationpartial

  partial

• Safety prompting and chain-of-thought reasoning fail to substantially close the gap between implicit and explicit bias.result 85%
  Evidencepartial

  Safety prompting and chain-of-thought reasoning fail to substantially close this gap

  Implicationpartial

  Directly stated in abstract with clear finding

  Verificationpartial

  partial

• Few-shot prompting reduces implicit bias by 84% but leaves caste bias at four times the level of any other dimension.result 90%
  Evidencepartial

  even few-shot prompting, which reduces implicit bias by 84%, leaves caste bias at four times the level of any other dimension

  Implicationpartial

  Directly stated in abstract with specific numeric results

  Verificationpartial

  partial

• Current alignment and prompting strategies address only the surface of bias evaluation while leaving culturally grounded stereotypic associations largely unresolved.limitation 80%
  Evidencepartial

  These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving culturally grounded stereotypic associations largely unresolved

  Implicationpartial

  Directly stated conclusion in abstract, though somewhat interpretive

  Verificationpartial

  partial

• ImplicitBBQ evaluates implicit bias through characteristic-based cues across six demographic dimensions: age, gender, region, religion, caste, and socioeconomic status.method 90%
  Evidencepartial

  We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, culturally associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status

  Implicationpartial

  Directly stated in abstract with specific details

  Verificationpartial

  partial

• The paper evaluated 11 different models using the ImplicitBBQ benchmark.method 90%
  Evidencepartial

  Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models

  Implicationpartial

  Directly stated in abstract with specific count

  Verificationpartial

  partial

Competitive landscape

A new benchmark and dataset to uncover and measure implicit biases in LLMs, enabling targeted mitigation strategies.