ARXIV:2604.19533 · AGENTS · SUBMITTED 22 APR · 03:20 UTC · FRESHNESS STALE

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

Alankrit Chona · Igor Kozlov · Ambuj Kumar · arXiv

An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

Ship in 2-4 weeks›Score7.0Evidence unverified

Opportunity summary

Pain An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

Evidence 0 refs | 3 sources | 50% coverage

Blocker Evidence unverified

Open Build Read PDF Signal Canvas Track

PROBLEM

An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across…

METHOD

Full abstract

We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.

RESULT

ScienceToStartup currently rates this 7.0/10 on the public viability pass. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks. Code availability…

WHY NOW

Agents moved forward this cycle; last verified April 2026. Public score 7.0/10. Production flags indicate code availability.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score7.0

PainAn LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

Evidence0 refs | 3 sources | 50% coverage

Blockerno shell-level blocker reported

Analysis summary

An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Competitive landscape

An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

Segment

Agents

Adoption evidence

No public code link in the paper record yet

Commercial read

7.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "32d3e9e3-7d3c-41dd-bb6f-d960325fcea3", "arxiv_id": "2604.19533", "canonical_route": "/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "endpoints": { "paper_pack": "/api/v1/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops/paper-pack", "build_passport": "/api/v1/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps", "normalized_query": "2604.19533", "route": "/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "paper_ref": "cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops#webpage", "url": "https://sciencetostartup.com/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "name": "Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps", "description": "An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops#scholarlyArticle", "headline": "Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps", "description": "An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.", "url": "https://sciencetostartup.com/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops", "sameAs": "https://arxiv.org/abs/2604.19533", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2604.19533" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-04-21T14:53:23.000Z", "author": [ { "@type": "Person", "name": "Alankrit Chona" }, { "@type": "Person", "name": "Igor Kozlov" }, { "@type": "Person", "name": "Ambuj Kumar" } ], "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 7 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "Agents" }, { "@type": "PropertyValue", "propertyID": "commercialReadiness", "value": "code" } ] }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "Agents", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "Cyber Defense Benchmark: Agentic Threat Hunting Evaluation f", "item": "https://sciencetostartup.com/paper/cyber-defense-benchmark-agentic-threat-hunting-evaluation-for-llms-in-secops" } ] } ] }

Competitive landscape

An LLM agent benchmark for evaluating threat hunting capabilities in cybersecurity, revealing current model limitations.

Segment

Agents

Adoption evidence

No public code link in the paper record yet

Commercial read

7.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline