Deep-network initialization for `absolute`#

The absolute construction (mode="absolute") constrains weights to |W|. Under a generic init (he_normal) this is poorly conditioned with depth, and deep absolute stacks fail to train. mononet now derives a static, per-activation init (variance-preserving gain + layer-mean-centering bias; mononet.core.init.absolute_init_params), the default for mode="absolute".

What this fixes. At moderate depth the new init makes absolute train where the old he_normal default does not (see the table below — lower train MSE is better; the target is unit-variance, so about 1.0 means “not learning”).

What it does not fix (Follow-up B). A genuinely deep (>= 8) plain stack still blows up: |W|’s all-positive weights make layer outputs strongly correlated, so variance compounds with depth — for both absolute and switch. Static per-layer init cannot make an unnormalized deep stack forward-stable. Deep training is the subject of Follow-up B (near-identity MonoResidual skip connections every few layers, and/or normalization).

import json
from pathlib import Path

import pandas as pd

RESULTS = Path("../../benchmarks/results/deep-init/trainability.json")
if not RESULTS.exists():
    print(
        "No deep-init results committed yet. Run the sweep "
        "(see benchmarks/README.md, 'Deep-network init')."
    )
else:
    rows = json.loads(RESULTS.read_text())
    df = pd.DataFrame(rows)
    table = df.pivot_table(index="depth", columns="method", values="final_train_mse")
    display(table)

method	absolute (he_normal)	absolute (new init)	switch
depth
2	1.7324	0.1167	0.0945
4	1.9424	0.8033	14.7118
8	2.0000	2.0000	373799.6256
16	2.0000	1000000.0000	1000000.0000

Deep-network initialization for absolute#

Deep-network initialization for `absolute`#