RuleForge: Automated Generation and Validation for Web Vulnerability Detection at Scale

Dimensions overall score 7.0

• RuleForge automatically generates JSON-based detection rules from structured Nuclei templates describing CVE details.method 95%
  Evidencepartial

  We present RuleForge, an AWS internal system that automatically generates detection rules--JSON-based patterns that identify malicious HTTP requests exploiting specific vulnerabilities--from structured Nuclei templates describing CVE details.

  Implicationpartial

  Directly stated in abstract with clear description of system functionality

  Verificationpartial

  partial

• RuleForge's LLM-as-a-judge validation system achieves AUROC of 0.75 for evaluating candidate rules across sensitivity and specificity dimensions.result 92%
  Evidencepartial

  This validation approach evaluates candidate rules across two dimensions--sensitivity (avoiding false negatives) and specificity (avoiding false positives)--achieving AUROC of 0.75

  Implicationpartial

  Directly stated in abstract with specific numeric metric

  Verificationpartial

  partial

• RuleForge reduces false positives by 67% compared to synthetic-test-only validation in production.result 93%
  Evidencepartial

  reducing false positives by 67% compared to synthetic-test-only validation in production.

  Implicationpartial

  Directly stated in abstract with clear numeric improvement

  Verificationpartial

  partial

• The volume of newly disclosed CVEs in 2025 exceeded 48,000, creating a need for automated detection mechanisms.market 91%
  Evidencepartial

  In 2025, the National Vulnerability Database published over 48,000 new vulnerabilities, motivating the need for automation.

  Implicationpartial

  Directly stated in abstract with specific statistic

  Verificationpartial

  partial

• RuleForge employs a 5x5 generation strategy with five parallel candidates and up to five refinement attempts each.method 90%
  Evidencepartial

  Our 5x5 generation strategy (five parallel candidates with up to five refinement attempts each) combined with continuous feedback loops enables systematic quality improvement.

  Implicationpartial

  Directly stated in abstract with specific technical details

  Verificationpartial

  partial

• RuleForge includes extensions enabling rule generation from unstructured data sources.technical 88%
  Evidencepartial

  We also present extensions enabling rule generation from unstructured data sources

  Implicationpartial

  Directly stated in abstract but less detailed than other claims

  Verificationpartial

  partial

• The paper demonstrates a proof-of-concept agentic workflow for multi-event-type detection.technical 87%
  Evidencepartial

  and demonstrate a proof-of-concept agentic workflow for multi-event-type detection.

  Implicationpartial

  Directly stated in abstract but described as proof-of-concept

  Verificationpartial

  partial

• Applying LLMs to cybersecurity tasks requires overconfidence mitigation and domain expertise in prompt design and quality review.limitation 89%
  Evidencepartial

  Our lessons learned highlight critical considerations for applying LLMs to cybersecurity tasks, including overconfidence mitigation and the importance of domain expertise in both prompt design and quality review of generated rules through human-in-the-loop validation.

  Implicationpartial

  Directly stated in abstract as lessons learned from the system

  Verificationpartial

  partial

Competitive landscape

RuleForge automates the generation of security detection rules from vulnerability descriptions using LLMs, significantly reducing false positives and improving detection capacity.