Skip to content

Methodology

Know what is measured. Know what is assumed.

The simulator separates the historical game state, the coaching recommendation, and the estimated value of the options offered.

01 · Historical evidence

Start with the documented situation.

Scenario checkpoints use official gamebooks, play-by-play records, and team reporting. Score, clock, down, distance, ball location, and timeouts are preserved rather than randomized into new combinations.

Each completed rep provides sources and the actual outcome. Field art and player movement illustrate the situation; they are not film reconstructions.

02 · Decision value

Compare complete paths.

When a model comparison is available, the review considers success, failure, and the remaining game. A conversion percentage and a chance to win are different quantities.

Currently, 19 scenarios have CoachSim fitted comparisons, and 15 more use attributed, published Next Gen Stats comparisons, including a near-tie. Other scenarios use conditional coaching analysis and score or clock arithmetic.

How the estimates are built

Training data
NFL play-by-play from 2016–2022
Calibration
2023 season
Held-out evaluation
2024 season
Model family
Histogram gradient-boosted trees

The win model uses score, remaining time, down, distance, field position, and both teams’ timeouts. The conversion model uses down, distance, and field position. The kick model uses distance. Games played on the dates of replays covered by this fitted model are excluded from training, calibration, and evaluation.

The held-out win model’s Brier score is 0.1564, compared with 0.2500 for a constant-rate baseline. Lower is better. This measures prediction error on observed outcomes; it does not validate what would have happened under a different coaching decision.

Ordinary two-point tries use a historical conversion estimate of 47.8%, fitted only from standard two-yard attempts. A tested machine-learning model did not outperform the baseline for those tries. Long kicks beyond 58 yards include an explicit distance adjustment because the sample is sparse.

Published comparisons and historical baselines

Reviews labeled “Historical NGS model” quote the decision advantage and event probabilities reported for that historical game. A published advantage may be available without every absolute win-probability branch. We show that gap without inventing missing values. These are model-era benchmarks, not win probabilities recalibrated for the 2026 kickoff and overtime rules. The visible verdict identifies the historical model.

Inputs labeled “League baseline” use comparable 2016–2024 play-by-play events, excluding the games identified in each batch’s provenance manifest. They are smoothed historical rates, not matchup-specific forecasts. Sample sizes and sampling intervals appear in the review. The shortened Jacksonville PAT uses a clearly labeled kick-distance proxy.

The overtime tie, penalty-adjusted try, and Raiders clock decision use complete conditional policy analysis. Neither an event success rate nor the eventual result proves a full-game advantage. Unresolved complete-policy comparisons receive close-call feedback. The Chargers–Eagles comparison preserves the published gap below 0.05 percentage points; it is not rounded into an exact tie. The Eagles–Bears conversion-order exercise conditions on another touchdown and no opponent score; its tie probability is not game win probability.

What can change the recommendation?

CoachSim’s fitted estimates and event baselines are retrospective and use league data. They omit team strength, personnel, injuries, tracking data, weather, and the specific play call. Published NGS comparisons use a separate model and may incorporate tracking information; each review identifies its source. None reproduces everything a coaching staff knew at the time.

The fitted policy calculations also make simplifying assumptions: a conversion gains the necessary yards, a failure gains none, and no penalty or return occurs. Punt distance and play duration are assumptions, not certain outcomes. Small gaps between options should not be treated as proven advantages.

Current overtime possession choices, intentional concessions, and personnel-timeout tradeoffs do not yet have fitted policy comparisons. Those reps identify the factors supporting the coaching recommendation instead.

Practice decisions that still exist

Audited against the 2026 NFL rulebook on 2026-10-15. We retired 7 playable situations built around legacy kickoff or overtime choices, plus four obsolete research-only examples. Retired reps cannot start, resume or replay; completed history is preserved.

Legal fourth-down and try checkpoints remain useful. Fitted immediate transitions now assume a standard direct-end-zone touchback at the 35 and a 33-yard PAT. A regulation tie uses a neutral 50% continuation value, not a team-specific overtime forecast. The underlying win model still learns historical outcomes; these transition updates are not a complete 2026 model recalibration.

How feedback and timing work

Progress grades follow the authored rubric; new published-model cases use the reported ranking. When an existing fitted model favors a different option, the review presents that disagreement. Alternatives less than two percentage points below the best estimate receive close-call feedback. Conditional coaching preferences retain their assumption label, and unresolved comparisons can mark both calls close.

In historical strategic reps, the optional decision timer creates training pressure; it does not consume the historical game clock. Final-kick drills separately simulate execution and clock operation.

Data and references

Official gamebooks and scenario-specific reporting are linked in each completed rep.

Model version: league-hgb-2026-10-15-v2. Estimates and rules audit updated October 15, 2026.