ARXIV:2605.12922 · LLM BEHAVIOR · SUBMITTED 14 MAY · 20:10 UTC · FRESHNESS FRESH

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction

Vardhan Dongre · Joseph Hsieh · Viet Dac Lai · Seunghyun Yoon · Trung Bui · Dilek Hakkani-Tür · arXiv

This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

Blocked on Code›Score3.0Evidence unverified

Opportunity summary

Pain This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

Evidence 0 refs | 0 sources | 0% coverage

Blocker Evidence unverified

Open Build Read PDF Signal Canvas Track

PROBLEM

This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations. This degradation has been measured behaviorally but not mechanistically explained.

METHOD

Full abstract

Large language models can follow complex instructions in a single turn, yet over long multi-turn interactions they often lose the thread of instructions, persona, and rules. This degradation has been measured behaviorally but not mechanistically explained. We propose a channel-transition account: goal-defining tokens become less accessible through attention, while goal-related information may persist in residual representations. We introduce the Goal Accessibility Ratio (GAR), measuring attention from generated tokens to task-defining goal tokens, and combine it with sliding-window ablations and residual-stream probes. When attention to instructions closes, what survives reveals architecture. Across architectures, the transition yields qualitatively distinct failure modes: some models preserve goal-conditioned behavior at vanishing attention, others fail despite decodable residual goal information, and the layer at which this encoding emerges varies from 2 to 27. A within-model causal ablation that force-closes the attention channel in Mistral collapses recall from near-perfect to 11% on a 20-fact retention task and raises persona-constraint violations above an adversarial-pressure baseline without user pressure, with both effects emerging at the predictable crossover turn. Linear probes recover per-episode recall outcomes from residual representations with AUC up to 0.99 across all four primary architectures, while input embeddings remain at chance. Across architectures and model scales, the gap between attention loss and residual decodability predicts whether goal-conditioned behavior survives channel closure. We contribute GAR as a diagnostic, the channel-transition framework as a controlled mechanistic account, and a parametric prediction of failure timing under windowed attention closure.

RESULT

ScienceToStartup currently rates this 3.0/10 on the public viability pass. We contribute GAR as a diagnostic, the channel-transition framework as a controlled mechanistic account, and a parametric prediction of failure timing under windowed attention…

WHY NOW

LLM Behavior moved forward this cycle; last verified May 2026. Public score 3.0/10.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score3.0

PainThis research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

Evidence0 refs | 0 sources | 0% coverage

Blockerno shell-level blocker reported

Analysis summary

This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Competitive landscape

This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

Segment

LLM Behavior

Adoption evidence

No public code link in the paper record yet

Commercial read

3.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "ec2d08db-b044-4e82-a36f-a77c12bc9a58", "arxiv_id": "2605.12922", "canonical_route": "/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "endpoints": { "paper_pack": "/api/v1/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction/paper-pack", "build_passport": "/api/v1/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction", "normalized_query": "2605.12922", "route": "/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "paper_ref": "when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction#webpage", "url": "https://sciencetostartup.com/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "name": "When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction", "description": "This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction#scholarlyArticle", "headline": "When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction", "description": "This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.", "url": "https://sciencetostartup.com/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction", "sameAs": "https://arxiv.org/abs/2605.12922", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2605.12922" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-05-13T02:58:18.000Z", "author": [ { "@type": "Person", "name": "Vardhan Dongre" }, { "@type": "Person", "name": "Joseph Hsieh" }, { "@type": "Person", "name": "Viet Dac Lai" }, { "@type": "Person", "name": "Seunghyun Yoon" }, { "@type": "Person", "name": "Trung Bui" }, { "@type": "Person", "name": "Dilek Hakkani-Tür" } ], "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 3 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "LLM Behavior" } ] }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "LLM Behavior", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "When Attention Closes: How LLMs Lose the Thread in Multi-Tur", "item": "https://sciencetostartup.com/paper/when-attention-closes-how-llms-lose-the-thread-in-multi-turn-interaction" } ] } ] }

Competitive landscape

This research investigates why large language models lose context in multi-turn conversations by analyzing attention mechanisms and residual representations.

Segment

LLM Behavior

Adoption evidence

No public code link in the paper record yet

Commercial read

3.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction

When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline