Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysis

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• The proposed deterministic static-analysis framework detects Knowledge Conflicting Hallucinations (KCHs) in LLM-generated code with 100% precision and 87.6% recall.result 95%
  Evidencepartial

  On a manually-curated dataset of 200 Python snippets, our framework detected KCHs with 100% precision and 87.6% recall (0.934 F1-score)

  Implicationpartial

  Explicitly stated in abstract with clear numeric results

  Verificationpartial

  partial

• The framework successfully auto-corrected 77.0% of all identified hallucinations in LLM-generated code.result 95%
  Evidencepartial

  successfully auto-corrected 77.0% of all identified hallucinations

  Implicationpartial

  Explicitly stated in abstract with clear numeric results

  Verificationpartial

  partial

• The framework uses deterministic rules to find and fix both API and identifier-level conflicts in LLM-generated code.method 90%
  Evidencepartial

  This non-executing approach uses deterministic rules to find and fix both API and identifier-level conflicts

  Implicationpartial

  Directly stated in abstract describing the method

  Verificationpartial

  partial

• The framework parses generated code into an Abstract Syntax Tree (AST) and validates it against a dynamically-generated Knowledge Base (KB) built via library introspection.technical 90%
  Evidencepartial

  We propose a post-processing framework that parses generated code into an Abstract Syntax Tree (AST) and validates it against a dynamically-generated Knowledge Base (KB) built via library introspection

  Implicationpartial

  Directly stated in abstract describing the technical approach

  Verificationpartial

  partial

• Existing mitigations like constrained decoding or non-deterministic LLM-in-the-loop repair are often unreliable for Knowledge Conflicting Hallucinations (KCHs).limitation 85%
  Evidencepartial

  Existing mitigations like constrained decoding or non-deterministic LLM-in-the-loop repair are often unreliable for these errors

  Implicationpartial

  Directly stated in abstract as motivation for the work

  Verificationpartial

  partial

• Knowledge Conflicting Hallucinations (KCHs) are subtle, semantic errors that evade linters and cause runtime failures in LLM-generated code.technical 90%
  Evidencepartial

  frequently introduce Knowledge Conflicting Hallucinations (KCHs), subtle, semantic errors, such as non-existent API parameters, that evade linters and cause runtime failures

  Implicationpartial

  Directly stated in abstract defining the problem

  Verificationpartial

  partial

• The deterministic post-processing approach is a viable and reliable alternative to probabilistic repair for trustworthy code generation.result 85%
  Evidencepartial

  Our findings demonstrate that this deterministic post-processing approach is a viable and reliable alternative to probabilistic repair, offering a clear path toward trustworthy code generation

  Implicationpartial

  Directly stated conclusion in abstract

  Verificationpartial

  partial

• The framework uses a non-executing approach to detect and correct hallucinations in LLM-generated code.method 90%
  Evidencepartial

  This non-executing approach uses deterministic rules to find and fix both API and identifier-level conflicts

  Implicationpartial

  Directly stated in abstract describing the technical approach

  Verificationpartial

  partial

Startup potential card