{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
   "metadata": {},
   "source": [
    "# Deep-network initialization for `absolute`\n",
    "\n",
    "The `absolute` construction (`mode=\"absolute\"`) constrains weights to `|W|`. Under a generic\n",
    "init (`he_normal`) this is poorly conditioned with depth, and deep `absolute` stacks fail to\n",
    "train. `mononet` now derives a **static, per-activation init** (variance-preserving gain +\n",
    "layer-mean-centering bias; `mononet.core.init.absolute_init_params`), the default for\n",
    "`mode=\"absolute\"`.\n",
    "\n",
    "**What this fixes.** At **moderate depth** the new init makes `absolute` train where the old\n",
    "`he_normal` default does not (see the table below — lower train MSE is better; the target is\n",
    "unit-variance, so about 1.0 means \"not learning\").\n",
    "\n",
    "**What it does not fix (Follow-up B).** A genuinely deep (>= 8) *plain* stack still blows up:\n",
    "`|W|`'s all-positive weights make layer outputs strongly correlated, so variance compounds\n",
    "with depth — for **both** `absolute` and `switch`. Static per-layer init cannot make an\n",
    "unnormalized deep stack forward-stable. Deep training is the subject of Follow-up B\n",
    "(near-identity `MonoResidual` skip connections every few layers, and/or normalization)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-02T16:12:31.846480Z",
     "iopub.status.busy": "2026-07-02T16:12:31.846395Z",
     "iopub.status.idle": "2026-07-02T16:12:32.373267Z",
     "shell.execute_reply": "2026-07-02T16:12:32.372866Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>method</th>\n",
       "      <th>absolute (he_normal)</th>\n",
       "      <th>absolute (new init)</th>\n",
       "      <th>switch</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>depth</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.7324</td>\n",
       "      <td>0.1167</td>\n",
       "      <td>0.0945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.9424</td>\n",
       "      <td>0.8033</td>\n",
       "      <td>14.7118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.0000</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>373799.6256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2.0000</td>\n",
       "      <td>1000000.0000</td>\n",
       "      <td>1000000.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "method  absolute (he_normal)  absolute (new init)        switch\n",
       "depth                                                          \n",
       "2                     1.7324               0.1167        0.0945\n",
       "4                     1.9424               0.8033       14.7118\n",
       "8                     2.0000               2.0000   373799.6256\n",
       "16                    2.0000         1000000.0000  1000000.0000"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "RESULTS = Path(\"../../benchmarks/results/deep-init/trainability.json\")\n",
    "if not RESULTS.exists():\n",
    "    print(\n",
    "        \"No deep-init results committed yet. Run the sweep \"\n",
    "        \"(see benchmarks/README.md, 'Deep-network init').\"\n",
    "    )\n",
    "else:\n",
    "    rows = json.loads(RESULTS.read_text())\n",
    "    df = pd.DataFrame(rows)\n",
    "    table = df.pivot_table(index=\"depth\", columns=\"method\", values=\"final_train_mse\")\n",
    "    display(table)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}