GNNVerifier: Graph-based Verifier for LLM Task Planning

Dimensions overall score 8.0

• GNNVerifier achieves significant gains in improving plan quality across diverse datasets, backbone LLMs, and planners.result 90%
  Evidencepartial

  Extensive experiments across diverse datasets, backbone LLMs, and planners demonstrate that our GNNVerifier achieves significant gains in improving plan quality.

  Implicationpartial

  Directly stated in abstract with mention of extensive experiments

  Verificationpartial

  partial

• LLM-based verifiers struggle to detect failures caused by structural relations across steps, such as type mismatches, missing intermediates, or broken dependencies.limitation 85%
  Evidencepartial

  LLM-based verifiers can be misled by plausible narration and struggle to detect failures caused by structural relations across steps, such as type mismatches, missing intermediates, or broken dependencies.

  Implicationpartial

  Directly stated in abstract as limitation of existing approaches

  Verificationpartial

  partial

• The method represents a plan as a directed graph with enriched attributes, where nodes denote sub-tasks and edges encode execution order and dependency constraints.method 95%
  Evidencepartial

  Firstly, we represent a plan as a directed graph with enriched attributes, where nodes denote sub-tasks and edges encode execution order and dependency constraints.

  Implicationpartial

  Explicitly described as first major component of the method

  Verificationpartial

  partial

• A graph neural network performs structural evaluation and diagnosis, producing graph-level plausibility scores and node/edge-level risk scores.method 90%
  Evidencepartial

  Secondly, a graph neural network (GNN) then performs structural evaluation and diagnosis, producing a graph-level plausibility score for plan acceptance as well as node/edge-level risk scores to localize erroneous regions.

  Implicationpartial

  Explicitly described as second major component of the method

  Verificationpartial

  partial

• The method constructs controllable perturbations from ground truth plan graphs to automatically generate training data with fine-grained annotations.method 90%
  Evidencepartial

  Thirdly, we construct controllable perturbations from ground truth plan graphs, and automatically generate training data with fine-grained annotations.

  Implicationpartial

  Explicitly described as third major component of the method

  Verificationpartial

  partial

• The verifier enables an LLM to conduct local edits (e.g., tool replacement or insertion) to correct plans when graph-level scores are insufficient.method 85%
  Evidencepartial

  Finally, guided by the feedback from our GNN verifier, we enable an LLM to conduct local edits (e.g., tool replacement or insertion) to correct the plan when the graph-level score is insufficient.

  Implicationpartial

  Directly stated as final component of the method

  Verificationpartial

  partial

• The approach requires structured plan representations that may not exist in all applications.limitation 80%
  Evidencepartial

  Requires structured plan representations that may not exist in all applications

  Implicationpartial

  Identified as a caveat in the analysis section

  Verificationpartial

  partial

• The method adds computational overhead that could slow real-time systems.limitation 75%
  Evidencepartial

  Adds computational overhead that could slow real-time systems

  Implicationpartial

  Identified as a caveat in the analysis section

  Verificationpartial

  partial

Startup potential card