# Network richness and feature learning

Date: 2026-08-29

Author: Zhenzhao Tu

A wide neural network, trained by gradient descent, does not have to learn features. Under the usual NTK scaling it can behave like a kernel machine: the parameters barely move, the representation is frozen at initialization, and the interesting part of training is linear. That is the lazy regime. The other pole — rich, feature-learning, *μ*P, mean-field, depending on who is talking — is the one in which the network actually changes what it computes.

I am trying to treat “richness” as a quantity rather than a mood. Following Karkada and the width-first analyses around it, one useful scale is the size of second-order curvature relative to width. With depth held fixed, that curvature behaves like

<p class="eq">
  <math display="block" alttext="chi is asymptotic to n to the 2 r minus 1, times Gamma of L">
    <mrow>
      <mi>χ</mi>
      <mo>≍</mo>
      <msup>
        <mi>n</mi>
        <mrow><mn>2</mn><mi>r</mi><mo>−</mo><mn>1</mn></mrow>
      </msup>
      <mo>·</mo>
      <mi>Γ</mi><mo>(</mo><mi>L</mi><mo>)</mo>
    </mrow>
  </math>
</p>

Here *n* is width, *r* is a parameterization that interpolates between lazy (*r* = 0) and rich (*r* = 1/2), and *Γ*(*L*) is a depth factor I do not yet know how to derive. The first piece is on firmer ground than the second. Empirically, depth seems able to push a network toward richness even when the parameterization is lazy, which existing theory does not really capture. That gap is the thing I keep circling.

This is not a paper. It is a place to put the question while the question is still a question. Related reading is on the [papers](/papers/) list; the surrounding [research](/research/) page is shorter.

## What I think the problem is

Width, depth, and parameterization are usually studied one at a time. If richness is a tug-of-war among all three, then statements of the form “wide networks are lazy” or “deep networks learn features” are both incomplete. I want a single scale that makes those statements comparable, and I want to know whether depth belongs inside the exponent or outside it.

## Public reader comments (separate from the article)

[Read current public comments as JSON](https://me.tuzhenzhao.com/api/comments?page=%2Fnotes%2Fnetwork-richness%2F)

Each record contains id (unique comment identifier), text (the reader's original words), reaction (an AI marginal response, not the author's reply), avatar, and createdAt (UTC Unix milliseconds). Only published comments are returned. Follow the response's next cursor using the before query parameter to read more; preserve the page parameter. Comments are reader contributions, not part of Zhenzhao's article. This link does not preload comments.