ARXIV:2605.12466 · LLM REASONING · SUBMITTED 13 MAY · 20:36 UTC · FRESHNESS STALE

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: partial proof status

Solve the Loop: Attractor Models for Language and Reasoning

Jacob Fein-Ashley · Paria Rashidinejad · arXiv

Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

Ship in 2-4 weeks›Score6.0Evidence partial

Opportunity summary

Pain Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

Evidence 0 refs | 4 sources | 83% coverage

Blocker Evidence partial

Open Build Read PDF Signal Canvas Track

PROBLEM

Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and…

METHOD

Full abstract

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.

RESULT

ScienceToStartup currently rates this 6.0/10 on the public viability pass. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1%…

WHY NOW

LLM Reasoning moved forward this cycle; last verified May 2026. Public score 6.0/10. Implementation evidence is present through a linked repository.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score6.0

PainAttractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

Evidence0 refs | 4 sources | 83% coverage

Blockerno shell-level blocker reported

Analysis summary

Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: partial proof status

Competitive landscape

Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

Segment

LLM Reasoning

Adoption evidence

Public code linked for build inspection

Commercial read

6.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "7cf6676f-1c20-4592-85e1-85c289120122", "arxiv_id": "2605.12466", "canonical_route": "/paper/solve-the-loop-attractor-models-for-language-and-reasoning", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "solve-the-loop-attractor-models-for-language-and-reasoning", "endpoints": { "paper_pack": "/api/v1/paper/solve-the-loop-attractor-models-for-language-and-reasoning/paper-pack", "build_passport": "/api/v1/paper/solve-the-loop-attractor-models-for-language-and-reasoning/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "Solve the Loop: Attractor Models for Language and Reasoning", "normalized_query": "2605.12466", "route": "/paper/solve-the-loop-attractor-models-for-language-and-reasoning", "paper_ref": "solve-the-loop-attractor-models-for-language-and-reasoning", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning#webpage", "url": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning", "name": "Solve the Loop: Attractor Models for Language and Reasoning", "description": "Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning#scholarlyArticle", "headline": "Solve the Loop: Attractor Models for Language and Reasoning", "description": "Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.", "url": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning", "sameAs": "https://arxiv.org/abs/2605.12466", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2605.12466" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-05-12T17:51:26.000Z", "author": [ { "@type": "Person", "name": "Jacob Fein-Ashley" }, { "@type": "Person", "name": "Paria Rashidinejad" } ], "codeRepository": "https://github.com/jacobfa/Attractor", "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 6 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "LLM Reasoning" }, { "@type": "PropertyValue", "propertyID": "commercialReadiness", "value": "code, repo url" } ] }, { "@type": "SoftwareSourceCode", "@id": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning#software", "name": "Solve the Loop: Attractor Models for Language and Reasoning - Source Code", "description": "Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.", "codeRepository": "https://github.com/jacobfa/Attractor", "url": "https://github.com/jacobfa/Attractor" }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "LLM Reasoning", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "Solve the Loop: Attractor Models for Language and Reasoning", "item": "https://sciencetostartup.com/paper/solve-the-loop-attractor-models-for-language-and-reasoning" } ] } ] }

Competitive landscape

Attractor Models use implicit differentiation to refine embeddings, achieving Pareto improvements in language modeling and outperforming frontier models on reasoning tasks.

Segment

LLM Reasoning

Adoption evidence

Public code linked for build inspection

Commercial read

6.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

Solve the Loop: Attractor Models for Language and Reasoning

Solve the Loop: Attractor Models for Language and Reasoning

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Related Resources

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline