Preprint FZJ-2026-05876

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png
A unified theory of feature learning in RNNs and DNNs

 ;  ;  ;

2026
arXiv

arXiv () [10.48550/ARXIV.2602.15593]

This record in other databases:  

Please use a persistent id in citations: doi:  doi:

Abstract: Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($μ$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.

Keyword(s): Machine Learning (cs.LG) ; Disordered Systems and Neural Networks (cond-mat.dis-nn) ; FOS: Computer and information sciences ; FOS: Physical sciences


Contributing Institute(s):
  1. Computational and Systems Neuroscience (IAS-6)
Research Program(s):
  1. 5231 - Neuroscientific Foundations (POF4-523) (POF4-523)
  2. 5232 - Computational Principles (POF4-523) (POF4-523)
  3. GRK 2416 - GRK 2416 MultiSenses-MultiScales: Neue Ansätze zur Aufklärung neuronaler multisensorischer Integration (368482240) (368482240)
  4. DFG project G:(GEPRIS)533396241 - Evolutionäre Optimierung neuronaler Netzwerkdynamik auf eine empfängerspezifische interareale Kommunikation (533396241) (533396241)
  5. ACA - Advanced Computing Architectures (SO-092) (SO-092)
  6. DFG project G:(GEPRIS)561027837 - Einbettung neuronaler Ensembles und deren Signaturen (964/25) (561027837) (561027837)

Appears in the scientific report 2026
Database coverage:
OpenAccess
Click to display QR Code for this record

The record appears in these collections:
Institute Collections > IAS > IAS-6
Document types > Reports > Preprints
Workflow collections > Public records
Publications database
Open Access

 Record created 2026-11-11, last modified 2026-11-11


OpenAccess:
Download fulltext PDF
Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)