COMPAS Recidivism

Task: binary classification — predict two-year recidivism from the ProPublica COMPAS dataset (≈ 6,000 defendants).

Metric: accuracy (paper reports ≈ 0.672 for the CMNN baseline).

Monotone features (non-decreasing in recidivism probability — more prior offenses → higher risk):

• priors_count — number of prior criminal charges

• juv_fel_count — number of juvenile felony charges

• juv_misd_count — number of juvenile misdemeanor charges

• juv_other_count — number of other juvenile charges

Non-monotone features: sex, age, race, charge_degree.

Reference: Table 1, row “COMPAS” of Runje & Shankaranarayana (2023), arXiv:2205.11775.

Before running: fetch the datasets with python -m benchmarks.datasets.download.

from pathlib import Path

from benchmarks._common.config_io import load_config
from benchmarks._common.results import aggregate
from benchmarks._common.runner import run
from benchmarks.datasets.download import default_dest
from benchmarks.datasets.registry import load

DATA_DIR = default_dest()
CONFIG_DIR = Path("../../../benchmarks/configs")

PAPER_ACCURACY = 0.672  # Table 1 value from Runje & Shankaranarayana (2023)

# Small budget for scaffold execution; maintainer uses full seeds/epochs.
QUICK_SEEDS = (0, 1)
QUICK_EPOCHS = 5

bundle = load("compas", data_dir=DATA_DIR)

flavors = [
    ("torch", "switch", False),
    ("torch", "absolute", False),
    ("jax", "switch", False),
    ("keras", "switch", False),
]

results = {}
for backend, mode, residual in flavors:
    cfg = load_config(
        CONFIG_DIR / "compas.toml",
        backend=backend,
        mode=mode,
        residual=residual,
    )
    import dataclasses
    cfg = dataclasses.replace(cfg, seeds=QUICK_SEEDS, epochs=QUICK_EPOCHS)
    rows = run(cfg, bundle)
    agg = aggregate(rows, metric="accuracy", lower_is_better=False, top_k=len(rows))
    flavor_key = f"{backend}/{mode}{'[residual]' if residual else ''}"
    results[flavor_key] = agg
    print(f"{flavor_key}: accuracy = {agg.mean:.4f} ± {agg.std:.4f}")

import pandas as pd

rows_table = [{"flavor": "paper (CMNN)", "accuracy": PAPER_ACCURACY, "std": "-"}]
for flavor, agg in results.items():
    rows_table.append({"flavor": flavor, "accuracy": round(agg.mean, 4), "std": round(agg.std, 4)})

df = pd.DataFrame(rows_table).set_index("flavor")
df