Blog Feedback#

Task: regression — predict the number of comments a blog post will receive in the next 24 hours (UCI Blog Feedback dataset, ≈ 52,000 instances, 280 features).

Metric: RMSE (paper value to be confirmed by maintainer against the original airtai/monotonic-nn Blog.ipynb notebook).

Monotone features (non-decreasing in comment count — more past engagement → more future comments):

  • features 50–53, 55–59 (indices in the processed feature matrix): total comments before basetime, links, trackbacks, and related statistics over various time windows.

Note: The monotone column assignment for Blog is provisional. The maintainer should verify against Blog.ipynb in airtai/monotonic-nn before the headline run.

Non-monotone features: remaining 271 features (text statistics, source metrics, etc.).

Reference: Table 1, row “Blog Feedback” of Runje & Shankaranarayana (2023), arXiv:2205.11775.

Before running: fetch the datasets with python -m benchmarks.datasets.download.

from pathlib import Path

from benchmarks._common.config_io import load_config
from benchmarks._common.results import aggregate
from benchmarks._common.runner import run
from benchmarks.datasets.download import default_dest
from benchmarks.datasets.registry import load

DATA_DIR = default_dest()
CONFIG_DIR = Path("../../../benchmarks/configs")

# PAPER_RMSE: to be filled in by maintainer after confirming against Blog.ipynb
PAPER_RMSE = None  # placeholder

# Small budget for scaffold execution; maintainer uses full seeds/epochs.
QUICK_SEEDS = (0, 1)
QUICK_EPOCHS = 5

bundle = load("blog", data_dir=DATA_DIR)

flavors = [
    ("torch", "switch", False),
    ("torch", "absolute", False),
    ("jax", "switch", False),
    ("keras", "switch", False),
]

results = {}
for backend, mode, residual in flavors:
    cfg = load_config(
        CONFIG_DIR / "blog.toml",
        backend=backend,
        mode=mode,
        residual=residual,
    )
    import dataclasses
    cfg = dataclasses.replace(cfg, seeds=QUICK_SEEDS, epochs=QUICK_EPOCHS)
    rows = run(cfg, bundle)
    agg = aggregate(rows, metric="rmse", lower_is_better=True, top_k=len(rows))
    flavor_key = f"{backend}/{mode}{'[residual]' if residual else ''}"
    results[flavor_key] = agg
    print(f"{flavor_key}: RMSE = {agg.mean:.4f} ± {agg.std:.4f}")

import pandas as pd

paper_rmse_display = PAPER_RMSE if PAPER_RMSE is not None else "(TBD)"
rows_table = [{"flavor": "paper (CMNN)", "RMSE": paper_rmse_display, "std": "-"}]
for flavor, agg in results.items():
    rows_table.append({"flavor": flavor, "RMSE": round(agg.mean, 4), "std": round(agg.std, 4)})

df = pd.DataFrame(rows_table).set_index("flavor")
df