Deep monotonic residual — real-dataset accuracy

This page reports whether the now-trainable deep monotonic residual stacks (MonoResidual with sub_depth=2 skips — see the residual construction) improve held-out test accuracy over the shallow tuned flavors, across the five benchmark datasets.

Question

Stage 1 showed residual skips make depth-32 monotone stacks trainable on synthetic data. This study measures whether that trainability translates into better test metrics on real tabular data, under the standard benchmark protocol (stability-aware CV model selection; multi-protocol test reporting; the test set is touched once).

A null or negative result — depth not improving, or mildly hurting, accuracy on these small/medium tabular datasets — is an expected and reported outcome. Stage 1 establishes the capability; Stage 2 measures whether it pays off.

Flavors and effective depth

Six flavors per dataset: {switch, absolute} × {plain, residual, deep}. deep is just the residual construction searched at a larger depth band, so we report it collapsed into residual: the collapsed residual cell is whichever of {residual, deep} the stability-aware CV score preferred.

The depth hyperparameter counts residual blocks, and each block wraps a 2-layer monotone sub-network (sub_depth=2) in a skip connection (skip every two layers). We report the effective monotone-layer count L, not the raw HP:

• plain-D → L = D + 1 (D stack layers + linear read-out head)

• residual-D → L = 2·D + 2 (1 projection + 2·D block layers + head)

so the deep band D ∈ {6, 10, 16} is L ∈ {14, 22, 34}. plain searches D ∈ [1,4], residual/deep search D ∈ [1,4]/{6,10,16}. The separate unconstrained branch for non-monotone features is not counted in L.

Results

Metric per dataset: MSE (auto), RMSE (blog), accuracy (heart/compas/loan) — ↓ lower is better for MSE/RMSE, ↑ higher for accuracy. Each cell reports IQM (interquartile mean; robust primary estimator), mean ± std (paper-comparable), effective layers L, and a collapse count ⚠ shown only when seeds degenerated (constant base-rate prediction; · = none). Bold = best per dataset by IQM (ties broken by fewest collapses).

Main results (collapsed plain/residual)

dataset	mode	variant	layers	IQM	mean ± std	⚠
auto (MSE ↓)	switch	plain	2	9.78	9.76 ± 0.18	·
	switch	residual	4	9.89	10.11 ± 0.62	2/20
	absolute	plain	2	10.91	10.90 ± 0.21	·
	absolute	residual	4	9.92	9.94 ± 0.33	·
heart (acc ↑)	switch	plain	4	0.836	0.711 ± 0.249	4/20
	switch	residual	14	0.831	0.829 ± 0.012	2/20
	absolute	plain	3	0.836	0.839 ± 0.012	·
	absolute	residual	4	0.821	0.825 ± 0.008	·
compas (acc ↑)	switch	plain	2	0.679	0.679 ± 0.002	·
	switch	residual	14	0.641	0.632 ± 0.033	4/20
	absolute	plain	4	0.683	0.683 ± 0.002	·
	absolute	residual	10	0.684	0.684 ± 0.002	·
loan (acc ↑)	switch	plain	3	0.647	0.647 ± 0.001	·
	switch	residual	6	0.647	0.646 ± 0.001	·
	absolute	plain	3	0.648	0.648 ± 0.000	·
	absolute	residual	14	0.649	0.650 ± 0.001	·
blog (RMSE ↓)	switch	plain	2	0.185	0.185 ± 0.002	·
	switch	residual	4	0.182	0.182 ± 0.000	1/10
	absolute	plain	2	0.189	0.189 ± 0.000	·
	absolute	residual	4	0.173	0.173 ± 0.001	·

Robustness — all six flavors

The full breakdown behind the collapse above (no residual/deep merge), adding the median and the raw depth HP. d = the tuned depth (residual blocks).

dataset	flavor	layers (blocks)	mean ± std	median	IQM	⚠
auto	switch-plain	2 (d1)	9.76 ± 0.18	9.77	9.78	·
auto	switch-residual	4 (d1)	10.11 ± 0.62	9.87	9.89	2/20
auto	switch-deep	22 (d10)	10.02 ± 0.33	9.98	9.98	·
auto	absolute-plain	2 (d1)	10.90 ± 0.21	10.91	10.91	·
auto	absolute-residual	4 (d1)	9.94 ± 0.33	9.88	9.92	·
auto	absolute-deep	14 (d6)	10.54 ± 1.10	10.06	10.21	·
heart	switch-plain	4 (d3)	0.711 ± 0.249	0.836	0.836	4/20
heart	switch-residual	8 (d3)	0.832 ± 0.017	0.836	0.833	3/20
heart	switch-deep	14 (d6)	0.829 ± 0.012	0.836	0.831	2/20
heart	absolute-plain	3 (d2)	0.839 ± 0.012	0.836	0.836	·
heart	absolute-residual	4 (d1)	0.825 ± 0.008	0.820	0.821	·
heart	absolute-deep	34 (d16)	0.820 ± 0.000	0.820	0.820	·
compas	switch-plain	2 (d1)	0.679 ± 0.002	0.679	0.679	·
compas	switch-residual	6 (d2)	0.679 ± 0.004	0.679	0.679	·
compas	switch-deep	14 (d6)	0.632 ± 0.033	0.640	0.641	4/20
compas	absolute-plain	4 (d3)	0.683 ± 0.002	0.683	0.683	·
compas	absolute-residual	10 (d4)	0.684 ± 0.002	0.684	0.684	·
compas	absolute-deep	34 (d16)	0.666 ± 0.009	0.668	0.669	·
loan	switch-plain	3 (d2)	0.647 ± 0.001	0.647	0.647	·
loan	switch-residual	6 (d2)	0.646 ± 0.001	0.647	0.647	·
loan	switch-deep	22 (d10)	0.646 ± 0.000	0.646	0.646	·
loan	absolute-plain	3 (d2)	0.648 ± 0.000	0.648	0.648	·
loan	absolute-residual	4 (d1)	0.648 ± 0.000	0.648	0.648	·
loan	absolute-deep	14 (d6)	0.650 ± 0.001	0.649	0.649	·
blog	switch-plain	2 (d1)	0.185 ± 0.002	0.185	0.185	·
blog	switch-residual	4 (d1)	0.182 ± 0.000	0.182	0.182	1/10
blog	switch-deep	34 (d16)	0.191 ± 0.002	0.191	0.191	·
blog	absolute-plain	2 (d1)	0.189 ± 0.000	0.189	0.189	·
blog	absolute-residual	4 (d1)	0.173 ± 0.001	0.173	0.173	·
blog	absolute-deep	22 (d10)	0.175 ± 0.001	0.175	0.175	·

compas-deep budget note. The two compas-deep studies were tuned with search_seeds=1 (vs. 3 elsewhere): a 34-layer stack × 5-fold CV × 3 seeds × 50 trials was compute-intractable on the run hardware. The stability-aware selection is only a mild helper (robust reporting is the primary collapse mitigation), so this affects the selected HP marginally, not the reported test estimates.

What the numbers say

• Depth helps only on loan — the largest dataset (419k rows), where the best model is a 14-layer absolute residual stack (IQM 0.649/0.650, deep band d6). On every other dataset the best model is ≤ 4 effective layers, and the deep band (14–34 layers) is neutral-to-worse (e.g. compas absolute-deep 0.666 vs residual 0.684; auto switch-deep 9.98 vs plain 9.78). This is the pre-registered null-ish result: extra monotone depth does not pay off on small/medium tabular data.

• Absolute mode is the strongest construction once the read-out head is linear (see protocol — the ReLU-head fix): it wins on heart, compas, loan, and blog; switch wins only on auto.

• Instability clusters on plain/shallow switch, never on absolute: the ⚠ collapses are all switch (heart plain 4/20, compas-deep 4/20, auto residual 2/20, blog residual 1/10). The median/IQM are unaffected — this is why we report robustly and surface the collapse count rather than hiding it in a mean (compare heart switch-plain: mean 0.711 vs IQM 0.836).

Future work

A within-dataset size ladder on loan (subsample train to {5k, 20k, 50k, 100k, 200k, 419k}, shallow vs deep at each, plot the deep−shallow IQM Δ vs N) would test the hypothesis that deep monotone stacks win once the dataset is large enough — the cross-dataset evidence here (deep wins only on the largest set) is consistent but confounded by how much signal each dataset routes through the monotone path (e.g. blog sends only 9 of 276 features through it). See Loan size-ladder for that experiment.

Reproduce

uv run --extra torch --group bench python -m benchmarks.search \
    --datasets auto,heart,compas,loan,blog
uv run --group bench python -m benchmarks._common.make_tables   # regenerate the tables

This runs all six flavors per dataset and writes benchmarks/results/phase2/<dataset>-<flavor>.json. See benchmarks/RUNBOOK-stage2.md for the full GPU run procedure.