Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents

Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• The uncertainty-aware multi-agent system (UA-MULTI) using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate on an underspecified SWE-bench variant.result 95%
  Evidencepartial

  Our results demonstrate that this multi-agent system using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate

  Implicationpartial

  This is a direct result stated in the abstract and supported by Figure 2 and the text discussing UA-MULTI's performance.

  Verificationpartial

  partial

• The UA-MULTI system significantly outperforms a standard single-agent setup (UA-SINGLE) with a 61.20% task resolve rate.result 95%
  Evidencepartial

  significantly outperforming a standard single-agent setup (61.20%)

  Implicationpartial

  This is a direct comparison of results stated in the abstract and supported by Figure 2 and the text discussing UA-MULTI vs UA-SINGLE.

  Verificationpartial

  partial

• The proposed uncertainty-aware multi-agent scaffold explicitly decouples underspecification detection from code execution.method 90%
  Evidencepartial

  we propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution.

  Implicationpartial

  This is a core methodological contribution described in the abstract and illustrated in Figure 1.

  Verificationpartial

  partial

• The multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues.result 90%
  Evidencepartial

  we find that the multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues.

  Implicationpartial

  This is a key finding about the behavior of the proposed system, stated in the abstract and elaborated upon in the findings.

  Verificationpartial

  partial

• The UA-MULTI system closes the performance gap with agents operating on fully specified instructions (FULL baseline).result 85%
  Evidencepartial

  closing the performance gap with agents operating on fully specified instructions.

  Implicationpartial

  The abstract states this, and the results in Figure 2 show UA-MULTI (69.40%) is close to the FULL baseline (which is implied to be higher, though its exact value isn't explicitly stated in the provided text, the comparison is made).

  Verificationpartial

  partial

• The UA-SINGLE configuration prompts a single coding agent to check for underspecification at each turn.method 90%
  Evidencepartial

  In this configuration, a single coding agent is prompted at each turn to check for underspecification and, if detected, to query the user.

  Implicationpartial

  This describes the method for the UA-SINGLE baseline, as stated in the text.

  Verificationpartial

  partial

• The INTERACTIVEBASELINE configuration explicitly informs the agent that the issue description is incomplete, making it compulsory to query the user before proceeding with execution.method 90%
  Evidencepartial

  Importantly, the task prompt is modified to explicitly inform the agent that the issue description is incomplete, making it compulsory to query the user before proceeding with any execution.

  Implicationpartial

  This accurately describes the setup of the INTERACTIVEBASELINE as presented in the text.

  Verificationpartial

  partial

• The uncertainty-aware multi-agent system (UA-MULTI) achieves a 69.40% task resolve rate on an underspecified SWE-bench variant.result 95%
  Evidencepartial

  Our results demonstrate that this multi-agent system using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate

  Implicationpartial

  This is a direct result stated in the abstract and supported by Figure 2 and the text comparing UA-MULTI to other baselines.

  Verificationpartial

  partial

• The UA-MULTI system significantly outperforms a standard single-agent setup (UA-SINGLE) with a 61.20% resolve rate.result 95%
  Evidencepartial

  significantly outperforming a standard single-agent setup (61.20%)

  Implicationpartial

  This is a direct comparison of results stated in the abstract and explicitly detailed in the text and Figure 2.

  Verificationpartial

  partial

• The proposed uncertainty-aware multi-agent scaffold explicitly decouples underspecification detection from code execution.method 90%
  Evidencepartial

  we propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution.

  Implicationpartial

  This is a core methodological contribution described in the abstract and illustrated in Figure 1.

  Verificationpartial

  partial

• The multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues.result 90%
  Evidencepartial

  we find that the multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues.

  Implicationpartial

  This is a key finding about the behavior of the proposed system, stated in the abstract and elaborated upon in the findings.

  Verificationpartial

  partial

• The UA-MULTI system closes the performance gap with agents operating on fully specified instructions (FULL baseline).result 85%
  Evidencepartial

  closing the performance gap with agents operating on fully specified instructions.

  Implicationpartial

  The abstract states this, and the results in Figure 2 show UA-MULTI (69.40%) is close to INTERACTIVEBASELINE (70.40%) and significantly better than HIDDEN (54.80%). The FULL baseline is not explicitly given a percentage in the provided text, but the comparison implies this.

  Verificationpartial

  partial

Startup potential card