guest ::
| Home > Publications database > Understanding and Improving Optimization in Predictive Coding Networks |
| Home > Publications database > Understanding and Improving Optimization in Predictive Coding Networks |
| Contribution to a conference proceedings | FZJ-2024-07229 |
; ;
2023
arXiv
This record in other databases:
Please use a persistent id in citations: doi:10.48550/ARXIV.2305.13562
Abstract: Backpropagation (BP), the standard learning algorithm for artificial neural networks, is often considered biologically implausible. In contrast, the standard learning algorithm for predictive coding (PC) models in neuroscience, known as the inference learning algorithm (IL), is a promising, bio-plausible alternative. However, several challenges and questions hinder IL's application to real-world problems. For example, IL is computationally demanding, and without memory-intensive optimizers like Adam, IL may converge to poor local minima. Moreover, although IL can reduce loss more quickly than BP, the reasons for these speedups or their robustness remains unclear. In this paper, we tackle these challenges by 1) altering the standard implementation of PC circuits to substantially reduce computation, 2) developing a novel optimizer that improves the convergence of IL without increasing memory usage, and 3) establishing theoretical results that help elucidate the conditions under which IL is sensitive to second and higher-order information.
Keyword(s): Neural and Evolutionary Computing (cs.NE) ; Neurons and Cognition (q-bio.NC) ; FOS: Computer and information sciences ; FOS: Biological sciences
|
The record appears in these collections: |
PDF