Hierarchical models: how much to pool

PythonCompare the three on how well they predict the rest of the season.

import pandas as pd

h = pd.read_csv("data/batting_2026.csv")
league = h.hits.sum() / h.at_bats.sum()
noise = (h.hits / h.at_bats * (1 - h.hits / h.at_bats) / h.at_bats).mean()
k = league * (1 - league) / max((h.hits / h.at_bats).var() - noise, 1e-5) - 1

est = h.assign(
    complete=league,
    no_pooling=h.hits / h.at_bats,
    partial=(h.hits + league * k) / (h.at_bats + k),
    rest=h.remaining_hits / h.remaining_at_bats,
)
errors = {c: ((est[c] - est.rest) ** 2).mean() for c in ["complete", "no_pooling", "partial"]}
print(f"prior worth {k:.0f} at-bats")
for name, e in errors.items():
    print(f"{name:<11} mean squared error {e:.5f}")

Ran when this sheet was built. Output:

prior worth 84 at-bats
complete    mean squared error 0.00127
no_pooling  mean squared error 0.00583
partial     mean squared error 0.00152