{ "cells": [ { "cell_type": "markdown", "id": "b1c2d3e4-f5a6-7890-bcde-fa1234567891", "metadata": {}, "source": [ "# Phase 2a Flavor-Comparison Summary\n", "\n", "This notebook summarises the Optuna hyperparameter-search results for the\n", "**Phase 2a flavor study** — comparing the four monotonic-layer flavors across\n", "the five ICML 2023 benchmark datasets. A flavor is a **mode × residual** combination:\n", "\n", "| Axis | Values |\n", "|------|--------|\n", "| mode | `absolute` (weight-magnitude constraint, Runje & Shankaranarayana 2023) · `switch` (activation-switch, Sartor et al. 2025) |\n", "| residual | `plain` (stacked Mono layers) · `residual` (dual-gated MonoResidual blocks) |\n", "\n", "The four flavors are `switch-plain`, `switch-residual`, `absolute-plain`, `absolute-residual`,\n", "each tuned independently per dataset via Optuna.\n", "\n", "For each dataset the table below shows:\n", "\n", "- One row per flavor (tuned best from Phase 2a search), with **test** mean ± std across seeds.\n", "- A `paper (CMNN) [prior protocol]` row with the number reported in Table 1 / Table 2 of\n", " Runje & Shankaranarayana (ICML 2023) — included for like-for-like comparison.\n", " These paper numbers were obtained with a test-selected protocol and are **not directly comparable** to our held-out results — see [Protocol](protocol.md).\n", "- An `XGBoost` row as a non-monotonic gradient-boosting baseline\n", " (skipped gracefully when the raw data are not present in the render environment).\n", "\n", "**Blog dataset note:** the paper and the search both optimise MSE internally, but the\n", "table displays **RMSE** for Blog to match the scale used in the paper's Table 1.\n", "\n", "Numbers come from the maintainer's search run\n", "(`benchmarks/notebooks/search-*.ipynb`), committed as\n", "`benchmarks/results/phase2/*.json`. Until those runs are executed and\n", "committed, this notebook renders a placeholder message." ] }, { "cell_type": "code", "execution_count": 1, "id": "c2d3e4f5-a6b7-8901-cdef-ab2345678902", "metadata": { "execution": { "iopub.execute_input": "2026-07-01T13:13:13.868911Z", "iopub.status.busy": "2026-07-01T13:13:13.868797Z", "iopub.status.idle": "2026-07-01T13:13:16.785120Z", "shell.execute_reply": "2026-07-01T13:13:16.784622Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### auto (metric: mse)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
valuestd
method
absolute-plain9.51910.2601
absolute-residual9.94280.1841
switch-plain9.55810.2512
switch-residual10.43850.5537
paper (CMNN) [prior protocol]8.3700-
XGBoost12.7538-
\n", "
" ], "text/plain": [ " value std\n", "method \n", "absolute-plain 9.5191 0.2601\n", "absolute-residual 9.9428 0.1841\n", "switch-plain 9.5581 0.2512\n", "switch-residual 10.4385 0.5537\n", "paper (CMNN) [prior protocol] 8.3700 -\n", "XGBoost 12.7538 -" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import json\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "\n", "from benchmarks.datasets.download import default_dest\n", "from benchmarks.datasets.registry import load\n", "from benchmarks.baselines.xgboost import run_xgboost\n", "\n", "RESULTS = Path(\"../../benchmarks/results/phase2\")\n", "\n", "# Paper-quoted Table 1/2 numbers (tagged [prior protocol]). Blog is RMSE; others per dataset.\n", "PAPER = {\n", " \"auto\": (\"mse\", 8.37),\n", " \"heart\": (\"accuracy\", 0.885),\n", " \"compas\": (\"accuracy\", 0.692),\n", " \"loan\": (\"accuracy\", 0.653),\n", " \"blog\": (\"rmse\", None), # TODO(maintainer): transcribe paper Table 1 Blog RMSE\n", "}\n", "\n", "results = sorted(RESULTS.glob(\"*.json\")) if RESULTS.exists() else []\n", "if not results:\n", " print(\"No Phase-2a results committed yet. Run tools/mononet-benchmark-search.\")\n", "else:\n", " by_ds: dict[str, list[dict]] = {}\n", " for f in results:\n", " rec = json.loads(f.read_text())\n", " by_ds.setdefault(rec[\"dataset\"], []).append(rec)\n", " for ds, recs in sorted(by_ds.items()):\n", " metric = recs[0][\"test_metric\"]\n", " rows = []\n", " for rec in sorted(recs, key=lambda r: r[\"flavor\"]):\n", " mean = rec[\"test_mean\"]\n", " # Blog: study reports mse; show rmse for like-for-like paper comparison.\n", " if ds == \"blog\" and metric == \"mse\":\n", " mean = mean ** 0.5\n", " rows.append({\"method\": rec[\"flavor\"], \"value\": round(mean, 4),\n", " \"std\": round(rec[\"test_std\"], 4)})\n", " pmetric, pval = PAPER.get(ds, (metric, None))\n", " rows.append({\"method\": \"paper (CMNN) [prior protocol]\", \"value\": pval, \"std\": \"-\"})\n", " try:\n", " xgb = run_xgboost(load(ds, data_dir=default_dest()), seed=0)\n", " key = \"accuracy\" if \"accuracy\" in xgb else (\"rmse\" if ds == \"blog\" else \"mse\")\n", " rows.append({\"method\": \"XGBoost\", \"value\": round(xgb[key], 4), \"std\": \"-\"})\n", " except Exception as exc: # data not downloaded in the render env\n", " rows.append({\"method\": \"XGBoost\", \"value\": f\"(skipped: {type(exc).__name__})\", \"std\": \"-\"})\n", " print(f\"### {ds} (metric: {'rmse' if ds == 'blog' else metric})\")\n", " display(pd.DataFrame(rows).set_index(\"method\"))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" } }, "nbformat": 4, "nbformat_minor": 5 }