ARXIV:2603.09435 · NLP COMPLIANCE EVALUATION · SUBMITTED 02 APR · 02:30 UTC · FRESHNESS STALE

VerifiedSource: PDF linkedPartialPaperPack: 3 of 4 citation fields filledMissingMissing fields: authorsPartialProof: unverified proof status

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

arXiv

A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

Blocked on Code›Score6.0Evidence unverified

Opportunity summary

Pain A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

Evidence 0 refs | 0 sources | 17% coverage

Blocker Evidence unverified

Open Build Read PDF Signal Canvas Track

PROBLEM

A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems. The EU AI Act has emerged as a landmark in the regulatory landscape.

METHOD

Full abstract

The rapid rollout of AI in heterogeneous public and societal sectors has subsequently escalated the need for compliance with regulatory standards and frameworks. The EU AI Act has emerged as a landmark in the regulatory landscape. The development of solutions that elicit the level of AI systems' compliance with such standards is often limited by the lack of resources, hindering the semi-automated or automated evaluation of their performance. This generates the need for manual work, which is often error-prone, resource-limited or limited to cases not clearly described by the regulation. This paper presents an open, transparent, and reproducible method of creating a resource that facilitates the evaluation of NLP models with a strong focus on RAG systems. We have developed a dataset that contain the tasks of risk-level classification, article retrieval, obligation generation, and question-answering for the EU AI Act. The dataset files are in a machine-to-machine appropriate format. To generate the files, we utilise domain knowledge as an exegetical basis, combining with the processing and reasoning power of large language models to generate scenarios along with the respective tasks. Our methodology demonstrates a way to harness language models for grounded generation with high document relevancy. Besides, we overcome limitations such as navigating the decision boundaries of risk-levels that are not explicitly defined within the EU AI Act, such as limited and minimal cases. Finally, we demonstrate our dataset's effectiveness by evaluating a RAG-based solution that reaches 0.87 and 0.85 F1-score for prohibited and high-risk scenarios.

RESULT

ScienceToStartup currently rates this 6.0/10 on the public viability pass. Our methodology demonstrates a way to harness language models for grounded generation with high document relevancy.

WHY NOW

NLP Compliance Evaluation moved forward this cycle; last verified April 2026. Public score 6.0/10.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score6.0

PainA dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

Evidence0 refs | 0 sources | 17% coverage

Blockermissing authors

Analysis summary

A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

VerifiedSource: PDF linkedPartialPaperPack: 3 of 4 citation fields filledMissingMissing fields: authorsPartialProof: unverified proof status

Competitive landscape

A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

Segment

NLP Compliance Evaluation

Adoption evidence

No public code link in the paper record yet

Commercial read

6.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "a922c674-ce1a-4de9-96c8-9a20ef373353", "arxiv_id": "2603.09435", "canonical_route": "/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "endpoints": { "paper_pack": "/api/v1/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems/paper-pack", "build_passport": "/api/v1/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems", "normalized_query": "2603.09435", "route": "/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "paper_ref": "ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems#webpage", "url": "https://sciencetostartup.com/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "name": "AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems", "description": "A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems#scholarlyArticle", "headline": "AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems", "description": "A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.", "url": "https://sciencetostartup.com/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems", "sameAs": "https://arxiv.org/abs/2603.09435", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2603.09435" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-03-10T09:47:50.000Z", "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 6 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "NLP Compliance Evaluation" } ] }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "NLP Compliance Evaluation", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "AI Act Evaluation Benchmark: An Open, Transparent, and Repro", "item": "https://sciencetostartup.com/paper/ai-act-evaluation-benchmark-an-open-transparent-and-reproducible-evaluation-dataset-for-nlp-and-rag-systems" } ] } ] }

Competitive landscape

A dataset for evaluating NLP models against EU AI Act compliance, enabling automated assessments of AI systems.

Segment

NLP Compliance Evaluation

Adoption evidence

No public code link in the paper record yet

Commercial read

6.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline