CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems

Dimensions overall score 4.0

• Collaborative Entropy (CoE) is a unified information-theoretic metric for semantic uncertainty in multi-LLM collaboration that combines intra-model semantic entropy and inter-model divergence.method 95%
  Evidencepartial

  To address this gap, we propose Collaborative Entropy (CoE), a unified information-theoretic metric for semantic uncertainty in multi-LLM collaboration. CoE is defined on a shared semantic cluster space and combines two components: intra-model semantic entropy and inter-model divergence to the ensemble mean.

  Implicationpartial

  Explicitly defined in the abstract and detailed in the method section with a formal equation.

  Verificationpartial

  partial

• CoE provides stronger uncertainty estimation than standard entropy- and divergence-based baselines on TriviaQA and SQuAD datasets.result 90%
  Evidencepartial

  Experiments on \textit{TriviaQA} and \textit{SQuAD} with LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Mistral-7B-Instruct show that CoE provides stronger uncertainty estimation than standard entropy- and divergence-based baselines.

  Implicationpartial

  Directly stated in the abstract as an experimental result with specific datasets named.

  Verificationpartial

  partial

• The performance gains of CoE become larger as additional heterogeneous models are introduced into the multi-LLM system.result 85%
  Evidencepartial

  with gains becoming larger as additional heterogeneous models are introduced.

  Implicationpartial

  Explicitly stated in the abstract as a finding from experiments.

  Verificationpartial

  partial

• CoE is not a weighted ensemble predictor or output-scoring rule, but a system-level uncertainty measure that characterizes collaborative confidence and cross-model semantic disagreement.technical 95%
  Evidencepartial

  Crucially, CoE isnota weighted ensemble predictor or an output-scoring rule: it is a system-level uncertainty measure that characterizes collaborative confidence and cross-model semantic disagreement, independently of any downstream task objective.

  Implicationpartial

  Explicitly and repeatedly stated in the abstract and method sections.

  Verificationpartial

  partial

• Existing single-model uncertainty quantification methods do not adequately capture semantic disagreement across multiple models in a multi-LLM system.limitation 90%
  Evidencepartial

  existing methods quantify uncertainty within each model but do not adequately capture semantic disagreement across models.

  Implicationpartial

  Directly stated as the core problem and motivation in the abstract.

  Verificationpartial

  partial

• Semantic Entropy (SE), as a single-model metric, cannot capture disagreement when multiple heterogeneous models each have low internal entropy yet commit to different semantic clusters.limitation 90%
  Evidencepartial

  However, as a single-model metric, SE cannot capture the disagreement that arises whenmultipleheterogeneous models each have low internal entropy yet commit todifferentsemantic clusters.

  Implicationpartial

  Explicitly stated as a limitation of the baseline method, forming the central motivation for CoE.

  Verificationpartial

  partial

• CoE has a theoretical property of non-negativity and achieves zero-value certainty under perfect semantic consensus among models.technical 85%
  Evidencepartial

  We analyze several core properties of CoE, including non-negativity, zero-value certainty under perfect semantic consensus, and the behavior of CoE when individual models collapse to delta distributions.

  Implicationpartial

  Stated in the abstract as an analyzed core property, with theoretical analysis referenced.

  Verificationpartial

  partial

• The inter-model divergence component (UE) in CoE uses KL divergence, whose reference-directed asymmetry is critical for capturing directional epistemic disagreement.technical 80%
  Evidencepartial

  where D(·∥·) is a distributional divergence. In our primary formulation we use the asymmetric KL divergence, DKL(pi∥¯p) =Pl j=1 pi(cj |x) log pi(cj |x) ¯p(cj |x) , whose reference-directed asymmetry is critical for capturing directional epistemic disagreement.

  Implicationpartial

  Directly stated in the method section explaining the formulation choice.

  Verificationpartial

  partial

Competitive landscape

A novel information-theoretic metric to quantify semantic disagreement and collaborative confidence across multiple Large Language Models.