Codes and Expansions (CodEx) Seminar

Sarah McCarty (Iowa State University):
Shallow ReLU Neural Networks and their Representations of Piecewise Linear Functions

The ReLU function is a simple function that is a popular choice as an activation function for neural networks. Infinite-width, shallow ReLU networks are integrals over a parameter space and are poorly understood compared to finite-width networks. We prove a conjecture that every piecewise linear infinite-width, shallow ReLU neural network is also expressible as a finite-width network. The conjecture originated with Ongie et al. (A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case, 2019). In the proof, we will show the simple derivatives of a piecewise linear function force the measure over the parameter space to be zero almost everywhere. We then extend results of finite-width newtorks to infinite-width networks as corollaries.