# Deep monotonic residual — real-dataset accuracy

This page reports whether the now-trainable *deep* monotonic residual stacks
(`MonoResidual` with `sub_depth=2` skips — see
[the residual construction](../concepts/monotonic-residual.md)) improve held-out
test accuracy over the shallow tuned flavors, across the five benchmark datasets.

## Question

Stage 1 showed residual skips make depth-32 monotone stacks *trainable* on
synthetic data. This study measures whether that trainability translates into
better test metrics on real tabular data, under the
[standard benchmark protocol](protocol.md) (stability-aware CV model selection;
multi-protocol test reporting; the test set is touched once).

A **null or negative result** — depth not improving, or mildly hurting, accuracy
on these small/medium tabular datasets — is an expected and reported outcome.
Stage 1 establishes the capability; Stage 2 measures whether it pays off.

## Flavors and effective depth

Six flavors per dataset: `{switch, absolute} × {plain, residual, deep}`. `deep`
is just the residual construction searched at a larger depth band, so we report
it **collapsed into `residual`**: the collapsed `residual` cell is whichever of
`{residual, deep}` the stability-aware CV score preferred.

The **`depth` hyperparameter counts residual blocks**, and each block wraps a
2-layer monotone sub-network (`sub_depth=2`) in a skip connection (skip every two
layers). We report the **effective monotone-layer count** `L`, not the raw HP:

- `plain`-`D` → `L = D + 1` (D stack layers + linear read-out head)
- `residual`-`D` → `L = 2·D + 2` (1 projection + `2·D` block layers + head)

so the deep band `D ∈ {6, 10, 16}` is `L ∈ {14, 22, 34}`. `plain` searches
`D ∈ [1,4]`, `residual`/`deep` search `D ∈ [1,4]`/`{6,10,16}`. The separate
unconstrained branch for non-monotone features is not counted in `L`.

## Results

Metric per dataset: MSE (`auto`), RMSE (`blog`), accuracy (`heart`/`compas`/`loan`)
— **↓ lower is better** for MSE/RMSE, **↑ higher** for accuracy. Each cell reports
**IQM** (interquartile mean; robust primary estimator), **mean ± std**
(paper-comparable), effective layers `L`, and a collapse count `⚠` shown only when
seeds degenerated (constant base-rate prediction; `·` = none). **Bold** = best per
dataset by IQM (ties broken by fewest collapses).

<!-- generated by `python -m benchmarks._common.make_tables`; do not edit by hand -->

### Main results (collapsed plain/residual)

| dataset | mode | variant | layers | IQM | mean ± std | ⚠ |
|---|---|---|--:|--:|--:|:-:|
| auto (MSE ↓) | switch | plain | 2 | **9.78** | 9.76 ± 0.18 | · |
|  | switch | residual | 4 | 9.89 | 10.11 ± 0.62 | 2/20 |
|  | absolute | plain | 2 | 10.91 | 10.90 ± 0.21 | · |
|  | absolute | residual | 4 | 9.92 | 9.94 ± 0.33 | · |
| heart (acc ↑) | switch | plain | 4 | 0.836 | 0.711 ± 0.249 | 4/20 |
|  | switch | residual | 14 | 0.831 | 0.829 ± 0.012 | 2/20 |
|  | absolute | plain | 3 | **0.836** | 0.839 ± 0.012 | · |
|  | absolute | residual | 4 | 0.821 | 0.825 ± 0.008 | · |
| compas (acc ↑) | switch | plain | 2 | 0.679 | 0.679 ± 0.002 | · |
|  | switch | residual | 14 | 0.641 | 0.632 ± 0.033 | 4/20 |
|  | absolute | plain | 4 | 0.683 | 0.683 ± 0.002 | · |
|  | absolute | residual | 10 | **0.684** | 0.684 ± 0.002 | · |
| loan (acc ↑) | switch | plain | 3 | 0.647 | 0.647 ± 0.001 | · |
|  | switch | residual | 6 | 0.647 | 0.646 ± 0.001 | · |
|  | absolute | plain | 3 | 0.648 | 0.648 ± 0.000 | · |
|  | absolute | residual | 14 | **0.649** | 0.650 ± 0.001 | · |
| blog (RMSE ↓) | switch | plain | 2 | 0.185 | 0.185 ± 0.002 | · |
|  | switch | residual | 4 | 0.182 | 0.182 ± 0.000 | 1/10 |
|  | absolute | plain | 2 | 0.189 | 0.189 ± 0.000 | · |
|  | absolute | residual | 4 | **0.173** | 0.173 ± 0.001 | · |

### Robustness — all six flavors

The full breakdown behind the collapse above (no `residual`/`deep` merge), adding
the **median** and the raw `depth` HP. `d` = the tuned `depth` (residual blocks).

| dataset | flavor | layers (blocks) | mean ± std | median | IQM | ⚠ |
|---|---|--:|--:|--:|--:|:-:|
| auto | switch-plain | 2 (d1) | 9.76 ± 0.18 | 9.77 | 9.78 | · |
| auto | switch-residual | 4 (d1) | 10.11 ± 0.62 | 9.87 | 9.89 | 2/20 |
| auto | switch-deep | 22 (d10) | 10.02 ± 0.33 | 9.98 | 9.98 | · |
| auto | absolute-plain | 2 (d1) | 10.90 ± 0.21 | 10.91 | 10.91 | · |
| auto | absolute-residual | 4 (d1) | 9.94 ± 0.33 | 9.88 | 9.92 | · |
| auto | absolute-deep | 14 (d6) | 10.54 ± 1.10 | 10.06 | 10.21 | · |
| heart | switch-plain | 4 (d3) | 0.711 ± 0.249 | 0.836 | 0.836 | 4/20 |
| heart | switch-residual | 8 (d3) | 0.832 ± 0.017 | 0.836 | 0.833 | 3/20 |
| heart | switch-deep | 14 (d6) | 0.829 ± 0.012 | 0.836 | 0.831 | 2/20 |
| heart | absolute-plain | 3 (d2) | 0.839 ± 0.012 | 0.836 | 0.836 | · |
| heart | absolute-residual | 4 (d1) | 0.825 ± 0.008 | 0.820 | 0.821 | · |
| heart | absolute-deep | 34 (d16) | 0.820 ± 0.000 | 0.820 | 0.820 | · |
| compas | switch-plain | 2 (d1) | 0.679 ± 0.002 | 0.679 | 0.679 | · |
| compas | switch-residual | 6 (d2) | 0.679 ± 0.004 | 0.679 | 0.679 | · |
| compas | switch-deep | 14 (d6) | 0.632 ± 0.033 | 0.640 | 0.641 | 4/20 |
| compas | absolute-plain | 4 (d3) | 0.683 ± 0.002 | 0.683 | 0.683 | · |
| compas | absolute-residual | 10 (d4) | 0.684 ± 0.002 | 0.684 | 0.684 | · |
| compas | absolute-deep | 34 (d16) | 0.666 ± 0.009 | 0.668 | 0.669 | · |
| loan | switch-plain | 3 (d2) | 0.647 ± 0.001 | 0.647 | 0.647 | · |
| loan | switch-residual | 6 (d2) | 0.646 ± 0.001 | 0.647 | 0.647 | · |
| loan | switch-deep | 22 (d10) | 0.646 ± 0.000 | 0.646 | 0.646 | · |
| loan | absolute-plain | 3 (d2) | 0.648 ± 0.000 | 0.648 | 0.648 | · |
| loan | absolute-residual | 4 (d1) | 0.648 ± 0.000 | 0.648 | 0.648 | · |
| loan | absolute-deep | 14 (d6) | 0.650 ± 0.001 | 0.649 | 0.649 | · |
| blog | switch-plain | 2 (d1) | 0.185 ± 0.002 | 0.185 | 0.185 | · |
| blog | switch-residual | 4 (d1) | 0.182 ± 0.000 | 0.182 | 0.182 | 1/10 |
| blog | switch-deep | 34 (d16) | 0.191 ± 0.002 | 0.191 | 0.191 | · |
| blog | absolute-plain | 2 (d1) | 0.189 ± 0.000 | 0.189 | 0.189 | · |
| blog | absolute-residual | 4 (d1) | 0.173 ± 0.001 | 0.173 | 0.173 | · |
| blog | absolute-deep | 22 (d10) | 0.175 ± 0.001 | 0.175 | 0.175 | · |

> **`compas-deep` budget note.** The two `compas-deep` studies were tuned with
> `search_seeds=1` (vs. 3 elsewhere): a 34-layer stack × 5-fold CV × 3 seeds ×
> 50 trials was compute-intractable on the run hardware. The stability-aware
> selection is only a mild helper (robust *reporting* is the primary collapse
> mitigation), so this affects the selected HP marginally, not the reported
> test estimates.

## What the numbers say

- **Depth helps only on `loan`** — the largest dataset (419k rows), where the
  best model is a 14-layer absolute residual stack (IQM 0.649/0.650, `deep` band
  `d6`). On every other dataset the best model is **≤ 4 effective layers**, and
  the deep band (14–34 layers) is neutral-to-worse (e.g. `compas` absolute-deep
  0.666 vs residual 0.684; `auto` switch-deep 9.98 vs plain 9.78). This is the
  pre-registered null-ish result: extra monotone depth does not pay off on
  small/medium tabular data.
- **Absolute mode is the strongest construction** once the read-out head is
  linear (see [protocol](protocol.md) — the ReLU-head fix): it wins on `heart`,
  `compas`, `loan`, and `blog`; `switch` wins only on `auto`.
- **Instability clusters on plain/shallow `switch`**, never on `absolute`: the
  `⚠` collapses are all `switch` (`heart` plain 4/20, `compas`-deep 4/20, `auto`
  residual 2/20, `blog` residual 1/10). The median/IQM are unaffected — this is
  why we report robustly and surface the collapse count rather than hiding it in
  a mean (compare `heart switch-plain`: mean 0.711 vs IQM 0.836).

## Future work

A within-dataset **size ladder on `loan`** (subsample train to
{5k, 20k, 50k, 100k, 200k, 419k}, shallow vs deep at each, plot the deep−shallow
IQM Δ vs N) would test the hypothesis that **deep monotone stacks win once the
dataset is large enough** — the cross-dataset evidence here (deep wins only on
the largest set) is consistent but confounded by how much signal each dataset
routes through the monotone path (e.g. `blog` sends only 9 of 276 features
through it). See [Loan size-ladder](loan-size-ladder.md) for that experiment.

## Reproduce

```
uv run --extra torch --group bench python -m benchmarks.search \
    --datasets auto,heart,compas,loan,blog
uv run --group bench python -m benchmarks._common.make_tables   # regenerate the tables
```

This runs all six flavors per dataset and writes
`benchmarks/results/phase2/<dataset>-<flavor>.json`. See
[`benchmarks/RUNBOOK-stage2.md`](https://github.com/davorrunje/mononet/blob/main/benchmarks/RUNBOOK-stage2.md)
for the full GPU run procedure.