Auto MPG

Task: regression — predict fuel efficiency (MPG) for 398 cars from the UCI Auto MPG dataset.

Metric: MSE. The paper reports ≈ 8.37 for the CMNN baseline, but under a test-selected protocol (HP search, early stopping, and best-epoch all on the test set, then best-5-of-10). A correctly-wired held-out harness should not reproduce 8.37 — it reports a somewhat higher, honest number. See Protocol.

Monotone features (non-increasing in MPG — heavier/more powerful → lower fuel economy):

• weight — vehicle curb weight (lbs)

• displacement — engine displacement (cubic inches)

• horsepower — engine power output

Non-monotone features: cylinders, acceleration, model_year, origin.

Reference: Table 1, row “Auto MPG” of Runje & Shankaranarayana (2023), arXiv:2205.11775.

Before running: fetch the datasets with python -m benchmarks.datasets.download.

from pathlib import Path

from benchmarks._common.config_io import load_config
from benchmarks._common.results import aggregate
from benchmarks._common.runner import run
from benchmarks.datasets.download import default_dest
from benchmarks.datasets.registry import load

DATA_DIR = default_dest()
CONFIG_DIR = Path("../../../benchmarks/configs")

PAPER_MSE = 8.37  # Table 1 value from Runje & Shankaranarayana (2023)

# Small budget for scaffold execution; maintainer uses full seeds/epochs.
QUICK_SEEDS = (0, 1)
QUICK_EPOCHS = 5

bundle = load("auto", data_dir=DATA_DIR)

flavors = [
    ("torch", "switch", False),
    ("torch", "absolute", False),
    ("jax", "switch", False),
    ("keras", "switch", False),
]

results = {}
for backend, mode, residual in flavors:
    cfg = load_config(
        CONFIG_DIR / "auto.toml",
        backend=backend,
        mode=mode,
        residual=residual,
    )
    # Override for quick scaffold run
    import dataclasses
    cfg = dataclasses.replace(cfg, seeds=QUICK_SEEDS, epochs=QUICK_EPOCHS)
    rows = run(cfg, bundle)
    agg = aggregate(rows, metric="mse", lower_is_better=True, top_k=len(rows))
    flavor_key = f"{backend}/{mode}{'[residual]' if residual else ''}"
    results[flavor_key] = agg
    print(f"{flavor_key}: MSE = {agg.mean:.4f} ± {agg.std:.4f}")

import pandas as pd

rows_table = [{"flavor": "paper (CMNN)", "MSE": PAPER_MSE, "std": "-"}]
for flavor, agg in results.items():
    rows_table.append({"flavor": flavor, "MSE": round(agg.mean, 4), "std": round(agg.std, 4)})

df = pd.DataFrame(rows_table).set_index("flavor")
df