No Single Best Model for Diversity: Learning a Router for Sample Diversity

Dimensions overall score 7.0

• No single LLM dominates at generating diverse responses to a wide range of open-ended prompts.result 90%
  Evidencepartial

  finding no single model dominates at generating diverse responses to a wide range of open-ended prompts

  Implicationpartial

  Directly stated in abstract with evaluation of 18 LLMs supporting the finding

  Verificationpartial

  partial

• For each prompt, there exists a model that significantly outperforms all other models at generating a diverse answer set.result 85%
  Evidencepartial

  per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set

  Implicationpartial

  Directly stated in abstract as a key finding motivating the router approach

  Verificationpartial

  partial

• The introduced router that predicts the best model for each query outperforms the single best model baseline on NB-Wildchat (26.3% vs 23.8%).result 95%
  Evidencepartial

  On NB-Wildchat, our trained router outperforms the single best model baseline (26.3% vs $23.8%)

  Implicationpartial

  Specific numeric results provided in abstract with clear comparison

  Verificationpartial

  partial

• The router generalizes to an out-of-domain dataset (NB-Curated).result 80%
  Evidencepartial

  We further show generalization to an out-of-domain dataset (NB-Curated)

  Implicationpartial

  Directly stated in abstract but without specific numeric results provided

  Verificationpartial

  partial

• The router generalizes to different answer-generation prompting strategies.result 80%
  Evidencepartial

  as well as different answer-generation prompting strategies

  Implicationpartial

  Directly stated in abstract but without specific details about which strategies

  Verificationpartial

  partial

• The paper introduces diversity coverage as a metric to measure comprehensive answer generation quality.method 90%
  Evidencepartial

  we introduce \textbf{diversity coverage}, a metric that measures the total quality scores assigned to each \textbf{unique} answer in the predicted answer set relative to the best possible answer set with the same number of answers

  Implicationpartial

  Explicitly stated as a new metric introduction with clear definition

  Verificationpartial

  partial

• The work lays foundation for studying comprehensive answer generation when access to multiple models is available.technical 85%
  Evidencepartial

  Our work lays foundation for studying generating comprehensive answers when we have access to a suite of models

  Implicationpartial

  Directly stated as conclusion in abstract

  Verificationpartial

  partial

• The evaluation involved 18 different LLMs to assess diversity generation capabilities.method 90%
  Evidencepartial

  Using this metric, we evaluate 18 LLMs

  Implicationpartial

  Specific number of models evaluated is explicitly stated

  Verificationpartial

  partial

Competitive landscape

A router that selects the best LLM for each query to maximize response diversity, outperforming single models.