ARXIV:2604.02201 · RNN THEORY · SUBMITTED 03 APR · 20:50 UTC · FRESHNESS STALE

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

On the Role of Depth in the Expressivity of RNNs

Maude Lizaire · Michael Rizvi-Martel · Éric Dupuis · Guillaume Rabusseau · arXiv

This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

Blocked on Code›Score3.0Evidence unverified

Opportunity summary

Pain This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

Evidence 0 refs | 0 sources | 33% coverage

Blocker Evidence unverified

Open Build Read PDF Signal Canvas Track

PROBLEM

This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth…

METHOD

Full abstract

The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs' memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We broaden our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.

RESULT

ScienceToStartup currently rates this 3.0/10 on the public viability pass. The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations.

WHY NOW

RNN Theory moved forward this cycle; last verified April 2026. Public score 3.0/10.

Continue into Read for claims, analysis, references, and neighboring papers.

Opportunity summary

Score3.0

PainThis paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

Evidence0 refs | 0 sources | 33% coverage

Blockerno shell-level blocker reported

Analysis summary

This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

VerifiedSource: PDF linkedVerifiedPaperPack: citation fields availablePartialProof: unverified proof status

Competitive landscape

This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

Segment

RNN Theory

Adoption evidence

No public code link in the paper record yet

Commercial read

3.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

{ "contract_version": "paper-r2", "paper_id": "93173982-f6ef-4e13-9a5e-7dd2c6753f1e", "arxiv_id": "2604.02201", "canonical_route": "/paper/on-the-role-of-depth-in-the-expressivity-of-rnns", "active_tab": "synced from current hash by the drawer client", "selected_artifact": "on-the-role-of-depth-in-the-expressivity-of-rnns", "endpoints": { "paper_pack": "/api/v1/paper/on-the-role-of-depth-in-the-expressivity-of-rnns/paper-pack", "build_passport": "/api/v1/paper/on-the-role-of-depth-in-the-expressivity-of-rnns/build-passport", "mcp_resource": "sciencetostartup://surfaces/paper-workspace" } }

{ "surface": "paper", "mode": "paper", "query": "On the Role of Depth in the Expressivity of RNNs", "normalized_query": "2604.02201", "route": "/paper/on-the-role-of-depth-in-the-expressivity-of-rnns", "paper_ref": "on-the-role-of-depth-in-the-expressivity-of-rnns", "topic_slug": null, "benchmark_ref": null, "dataset_ref": null }

{ "@context": "https://schema.org", "@graph": [ { "@type": "WebPage", "@id": "https://sciencetostartup.com/paper/on-the-role-of-depth-in-the-expressivity-of-rnns#webpage", "url": "https://sciencetostartup.com/paper/on-the-role-of-depth-in-the-expressivity-of-rnns", "name": "On the Role of Depth in the Expressivity of RNNs", "description": "This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.", "isPartOf": { "@id": "https://sciencetostartup.com/#website" } }, { "@type": "ScholarlyArticle", "@id": "https://sciencetostartup.com/paper/on-the-role-of-depth-in-the-expressivity-of-rnns#scholarlyArticle", "headline": "On the Role of Depth in the Expressivity of RNNs", "description": "This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.", "url": "https://sciencetostartup.com/paper/on-the-role-of-depth-in-the-expressivity-of-rnns", "sameAs": "https://arxiv.org/abs/2604.02201", "identifier": { "@type": "PropertyValue", "propertyID": "arXiv", "value": "2604.02201" }, "isAccessibleForFree": true, "isPartOf": { "@id": "https://sciencetostartup.com/#website" }, "datePublished": "2026-04-02T15:56:09.000Z", "author": [ { "@type": "Person", "name": "Maude Lizaire" }, { "@type": "Person", "name": "Michael Rizvi-Martel" }, { "@type": "Person", "name": "Éric Dupuis" }, { "@type": "Person", "name": "Guillaume Rabusseau" } ], "additionalProperty": [ { "@type": "PropertyValue", "propertyID": "viabilityScore", "value": 3 }, { "@type": "PropertyValue", "propertyID": "researchDomain", "value": "RNN Theory" } ] }, { "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://sciencetostartup.com" }, { "@type": "ListItem", "position": 2, "name": "RNN Theory", "item": "https://sciencetostartup.com/topics" }, { "@type": "ListItem", "position": 3, "name": "On the Role of Depth in the Expressivity of RNNs", "item": "https://sciencetostartup.com/paper/on-the-role-of-depth-in-the-expressivity-of-rnns" } ] } ] }

Competitive landscape

This paper theoretically analyzes the benefits of depth in Recurrent Neural Networks, showing it enhances memory capacity and expressivity.

Segment

RNN Theory

Adoption evidence

No public code link in the paper record yet

Commercial read

3.0/10 public viability

Direct

not classified

Adjacent

not classified

Substitute

not classified

Unknown

not classified

On the Role of Depth in the Expressivity of RNNs

On the Role of Depth in the Expressivity of RNNs

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline

Claim map

Constellation map

Competitive landscape

Buzz

PDF

REFERENCES

Related Papers

Subscribe to the weekly brief

Build artifacts

Brief

Experiment plan

Validation checklist

Scientific founder

Translational engineer

Domain operator

GTM lead

Regulatory/clinical advisor

Timeline