# Network richness

Author: Zhenzhao Tu

<p class="date">Working notes · 2026</p>

A wide neural network, trained by gradient descent, does not have to learn features. Under the usual NTK scaling it can behave like a kernel machine: the parameters barely move, the representation is frozen at initialization, and the interesting part of training is linear. That is the lazy regime. The other pole — rich, feature-learning, *μ*P, mean-field, depending on who is talking — is the one in which the network actually changes what it computes.

I am trying to treat “richness” as a quantity rather than a mood. Following Karkada and the width-first analyses around it, one useful scale is the size of second-order curvature relative to width. With depth held fixed, that curvature behaves like

<p class="eq">
  <math display="block" alttext="chi is asymptotic to n to the 2 r minus 1, times Gamma of L">
    <mrow>
      <mi>χ</mi>
      <mo>≍</mo>
      <msup>
        <mi>n</mi>
        <mrow><mn>2</mn><mi>r</mi><mo>−</mo><mn>1</mn></mrow>
      </msup>
      <mo>·</mo>
      <mi>Γ</mi><mo>(</mo><mi>L</mi><mo>)</mo>
    </mrow>
  </math>
</p>

Here *n* is width, *r* is a parameterization that interpolates between lazy (*r* = 0) and rich (*r* = 1/2), and *Γ*(*L*) is a depth factor I do not yet know how to derive. The first piece is on firmer ground than the second. Empirically, depth seems able to push a network toward richness even when the parameterization is lazy, which existing theory does not really capture. That gap is the thing I keep circling.

This page is not a paper. It is a place to put the question while the question is still a question. Related reading is on the [papers](/papers/) list; a dated version lives with the [notes](/notes/network-richness/).

## What I think the problem is

Width, depth, and parameterization are usually studied one at a time. If richness is a tug-of-war among all three, then statements of the form “wide networks are lazy” or “deep networks learn features” are both incomplete. I want a single scale that makes those statements comparable, and I want to know whether depth belongs inside the exponent or outside it.