Phase 2a Flavor-Comparison Summary

This notebook summarises the Optuna hyperparameter-search results for the Phase 2a flavor study — comparing the four monotonic-layer flavors across the five ICML 2023 benchmark datasets. A flavor is a mode × residual combination:

Axis	Values
mode	absolute (weight-magnitude constraint, Runje & Shankaranarayana 2023) · switch (activation-switch, Sartor et al. 2025)
residual	plain (stacked Mono layers) · residual (dual-gated MonoResidual blocks)

The four flavors are switch-plain, switch-residual, absolute-plain, absolute-residual, each tuned independently per dataset via Optuna.

For each dataset the table below shows:

• One row per flavor (tuned best from Phase 2a search), with test mean ± std across seeds.

• A paper (CMNN) [prior protocol] row with the number reported in Table 1 / Table 2 of Runje & Shankaranarayana (ICML 2023) — included for like-for-like comparison. These paper numbers were obtained with a test-selected protocol and are not directly comparable to our held-out results — see Protocol.

• An XGBoost row as a non-monotonic gradient-boosting baseline (skipped gracefully when the raw data are not present in the render environment).

Blog dataset note: the paper and the search both optimise MSE internally, but the table displays RMSE for Blog to match the scale used in the paper’s Table 1.

Numbers come from the maintainer’s search run (benchmarks/notebooks/search-*.ipynb), committed as benchmarks/results/phase2/*.json. Until those runs are executed and committed, this notebook renders a placeholder message.

import json
from pathlib import Path

import pandas as pd

from benchmarks.datasets.download import default_dest
from benchmarks.datasets.registry import load
from benchmarks.baselines.xgboost import run_xgboost

RESULTS = Path("../../benchmarks/results/phase2")

# Paper-quoted Table 1/2 numbers (tagged [prior protocol]). Blog is RMSE; others per dataset.
PAPER = {
    "auto": ("mse", 8.37),
    "heart": ("accuracy", 0.885),
    "compas": ("accuracy", 0.692),
    "loan": ("accuracy", 0.653),
    "blog": ("rmse", None),   # TODO(maintainer): transcribe paper Table 1 Blog RMSE
}

results = sorted(RESULTS.glob("*.json")) if RESULTS.exists() else []
if not results:
    print("No Phase-2a results committed yet. Run tools/mononet-benchmark-search.")
else:
    by_ds: dict[str, list[dict]] = {}
    for f in results:
        rec = json.loads(f.read_text())
        by_ds.setdefault(rec["dataset"], []).append(rec)
    for ds, recs in sorted(by_ds.items()):
        metric = recs[0]["test_metric"]
        rows = []
        for rec in sorted(recs, key=lambda r: r["flavor"]):
            mean = rec["test_mean"]
            # Blog: study reports mse; show rmse for like-for-like paper comparison.
            if ds == "blog" and metric == "mse":
                mean = mean ** 0.5
            rows.append({"method": rec["flavor"], "value": round(mean, 4),
                         "std": round(rec["test_std"], 4)})
        pmetric, pval = PAPER.get(ds, (metric, None))
        rows.append({"method": "paper (CMNN) [prior protocol]", "value": pval, "std": "-"})
        try:
            xgb = run_xgboost(load(ds, data_dir=default_dest()), seed=0)
            key = "accuracy" if "accuracy" in xgb else ("rmse" if ds == "blog" else "mse")
            rows.append({"method": "XGBoost", "value": round(xgb[key], 4), "std": "-"})
        except Exception as exc:  # data not downloaded in the render env
            rows.append({"method": "XGBoost", "value": f"(skipped: {type(exc).__name__})", "std": "-"})
        print(f"### {ds}  (metric: {'rmse' if ds == 'blog' else metric})")
        display(pd.DataFrame(rows).set_index("method"))

### auto  (metric: mse)

	value	std
method
absolute-plain	9.5191	0.2601
absolute-residual	9.9428	0.1841
switch-plain	9.5581	0.2512
switch-residual	10.4385	0.5537
paper (CMNN) [prior protocol]	8.3700	-
XGBoost	12.7538	-