guest ::
| Home > Publications database > Response functions in residual networks as a measure for signal propagation > print |
| Home > Publications database > Response functions in residual networks as a measure for signal propagation > print |
| 001 | 1040930 | ||
| 005 | 20250411203140.0 | ||
| 037 | _ | _ | |a FZJ-2025-02060 |
| 100 | 1 | _ | |a Fischer, Kirsten |0 P:(DE-Juel1)180150 |b 0 |e Corresponding author |u fzj |
| 111 | 2 | _ | |a DPG Spring Meeting of the Condensed Matter Section |c Regensburg |d 2025-03-16 - 2025-03-21 |w Germany |
| 245 | _ | _ | |a Response functions in residual networks as a measure for signal propagation |
| 260 | _ | _ | |c 2025 |
| 336 | 7 | _ | |a Conference Paper |0 33 |2 EndNote |
| 336 | 7 | _ | |a Other |2 DataCite |
| 336 | 7 | _ | |a INPROCEEDINGS |2 BibTeX |
| 336 | 7 | _ | |a conferenceObject |2 DRIVER |
| 336 | 7 | _ | |a LECTURE_SPEECH |2 ORCID |
| 336 | 7 | _ | |a Conference Presentation |b conf |m conf |0 PUB:(DE-HGF)6 |s 1744375014_4023 |2 PUB:(DE-HGF) |x After Call |
| 520 | _ | _ | |a Residual networks (ResNets) demonstrate superior trainability and performance compared to feed-forward networks, particularly at greater depths, due to the introduction of skip connections that enhance signal propagation to deeper layers. Prior studies have shown that incorporating a scaling parameter into the residual branch can further improve generalization performance. However, the underlying mechanisms behind these effects and their robustness across network hyperparameters remain unclear.For feed-forward networks, finite-size theories have proven valuable in understanding signal propagation and optimizing hyperparameters. Extending this approach to ResNets, we develop a finite-size field theory to systematically analyze signal propagation and its dependence on the residual branch's scaling parameter. Through this framework, we derive analytical expressions for the response function, which measures the network's sensitivity to varying inputs. We obtain a formula for the optimal scaling parameter, revealing that it depends minimally on other hyperparameters, such as weight variance, thereby explaining its universality across hyperparameter configurations. |
| 536 | _ | _ | |a 5232 - Computational Principles (POF4-523) |0 G:(DE-HGF)POF4-5232 |c POF4-523 |f POF IV |x 0 |
| 536 | _ | _ | |a 5234 - Emerging NC Architectures (POF4-523) |0 G:(DE-HGF)POF4-5234 |c POF4-523 |f POF IV |x 1 |
| 536 | _ | _ | |a MSNN - Theory of multi-scale neuronal networks (HGF-SMHB-2014-2018) |0 G:(DE-Juel1)HGF-SMHB-2014-2018 |c HGF-SMHB-2014-2018 |f MSNN |x 2 |
| 536 | _ | _ | |a RenormalizedFlows - Transparent Deep Learning with Renormalized Flows (BMBF-01IS19077A) |0 G:(DE-Juel-1)BMBF-01IS19077A |c BMBF-01IS19077A |x 3 |
| 536 | _ | _ | |a ACA - Advanced Computing Architectures (SO-092) |0 G:(DE-HGF)SO-092 |c SO-092 |x 4 |
| 536 | _ | _ | |a neuroIC002 - Recurrence and stochasticity for neuro-inspired computation (EXS-SF-neuroIC002) |0 G:(DE-82)EXS-SF-neuroIC002 |c EXS-SF-neuroIC002 |x 5 |
| 536 | _ | _ | |a GRK 2416 - GRK 2416: MultiSenses-MultiScales: Neue Ansätze zur Aufklärung neuronaler multisensorischer Integration (368482240) |0 G:(GEPRIS)368482240 |c 368482240 |x 6 |
| 700 | 1 | _ | |a Dahmen, David |0 P:(DE-Juel1)156459 |b 1 |u fzj |
| 700 | 1 | _ | |a Helias, Moritz |0 P:(DE-Juel1)144806 |b 2 |u fzj |
| 856 | 4 | _ | |u https://www.dpg-verhandlungen.de/year/2025/conference/regensburg/part/soe/session/7/contribution/4 |
| 909 | C | O | |o oai:juser.fz-juelich.de:1040930 |p VDB |
| 910 | 1 | _ | |a Forschungszentrum Jülich |0 I:(DE-588b)5008462-8 |k FZJ |b 0 |6 P:(DE-Juel1)180150 |
| 910 | 1 | _ | |a Forschungszentrum Jülich |0 I:(DE-588b)5008462-8 |k FZJ |b 1 |6 P:(DE-Juel1)156459 |
| 910 | 1 | _ | |a Forschungszentrum Jülich |0 I:(DE-588b)5008462-8 |k FZJ |b 2 |6 P:(DE-Juel1)144806 |
| 913 | 1 | _ | |a DE-HGF |b Key Technologies |l Natural, Artificial and Cognitive Information Processing |1 G:(DE-HGF)POF4-520 |0 G:(DE-HGF)POF4-523 |3 G:(DE-HGF)POF4 |2 G:(DE-HGF)POF4-500 |4 G:(DE-HGF)POF |v Neuromorphic Computing and Network Dynamics |9 G:(DE-HGF)POF4-5232 |x 0 |
| 913 | 1 | _ | |a DE-HGF |b Key Technologies |l Natural, Artificial and Cognitive Information Processing |1 G:(DE-HGF)POF4-520 |0 G:(DE-HGF)POF4-523 |3 G:(DE-HGF)POF4 |2 G:(DE-HGF)POF4-500 |4 G:(DE-HGF)POF |v Neuromorphic Computing and Network Dynamics |9 G:(DE-HGF)POF4-5234 |x 1 |
| 914 | 1 | _ | |y 2025 |
| 920 | 1 | _ | |0 I:(DE-Juel1)IAS-6-20130828 |k IAS-6 |l Computational and Systems Neuroscience |x 0 |
| 980 | _ | _ | |a conf |
| 980 | _ | _ | |a VDB |
| 980 | _ | _ | |a I:(DE-Juel1)IAS-6-20130828 |
| 980 | _ | _ | |a UNRESTRICTED |
| Library | Collection | CLSMajor | CLSMinor | Language | Author |
|---|