Network richness
Working notes · 2026
A wide neural network, trained by gradient descent, does not have to learn features. Under the usual NTK scaling it can behave like a kernel machine: the parameters barely move, the representation is frozen at initialization, and the interesting part of training is linear. That is the lazy regime. The other pole — rich, feature-learning, μP, mean-field, depending on who is talking — is the one in which the network actually changes what it computes.
I am trying to treat “richness” as a quantity rather than a mood. Following Karkada and the width-first analyses around it, one useful scale is the size of second-order curvature relative to width. With depth held fixed, that curvature behaves like
Here n is width, r is a parameterization that interpolates between lazy (r = 0) and rich (r = 1/2), and Γ(L) is a depth factor I do not yet know how to derive. The first piece is on firmer ground than the second. Empirically, depth seems able to push a network toward richness even when the parameterization is lazy, which existing theory does not really capture. That gap is the thing I keep circling.
This page is not a paper. It is a place to put the question while the question is still a question. Related reading is on the papers list; a dated version lives with the notes.
What I think the problem is
Width, depth, and parameterization are usually studied one at a time. If richness is a tug-of-war among all three, then statements of the form “wide networks are lazy” or “deep networks learn features” are both incomplete. I want a single scale that makes those statements comparable, and I want to know whether depth belongs inside the exponent or outside it.