How well can AI predict Formula 1?

Each model forecasts the entire Formula 1 field using public information available during its run. It decides how to research, reason, write code, and simulate. The target is the final race classification, including non-classification, non-starts, disqualifications, and retirement risk.

The benchmark evaluates the submitted probabilities. The written order, explanation, and research findings provide context; they do not earn extra points. Research quality and causal claims are examined in the post-race review, not assigned a separate numerical score.

The current cohort contains GPT-5.6 Sol, Muse Spark 1.3, and Grok 4.6. Each entrant keeps a pinned model version and a restricted provider policy. Provider fallback cannot switch the model. Reasoning uses the highest setting supported when the entrant was created.

Agents have web search, page extraction, a shell, and a disposable workspace. They may build their own models, but no simulation method is prescribed. The budget is a ceiling, not a target. Models run independently; they do not see competing forecasts.

The same limits apply to each post-race review. Forecast cost includes model API calls, search, and extraction. Total spend also includes reviews and failed or voided attempts across all season races. Forecast averages use the same shared races as the score. Where research fees are unavailable, comparisons use model API costs for every entrant. Recorded spend includes only saved charges and is not a complete bill. New research costs are estimates from recorded tool requests at published Parallel rates, before credits or discounts. Historical research fees are unavailable. Hosting is excluded.

A forecast starts with the race entry list, the model’s own previous forecasts and research, results from earlier races, scores, review snapshots, and its current notes. The normal workflow runs after qualifying. Agents independently verify current evidence, so research sources and completion times can differ.

New runs must start and finish before the configured race start. A race with a saved result cannot be forecast again through the live workflow. Once completed, a prediction stays frozen.

After the official result is supplied, a separate review compares the prediction with what happened and revisits its assumptions. The model edits its own notes for the next race. The workflow checks that earlier reviews are complete. This is learning through context and notes, not weight training; scratch code does not carry forward.

Archived runs retain their original outputs. Where an older run has no explicit finishing-order report, the race table clearly labels its derived average-rank ordering.

Ranked Probability Score (RPS) measures how far the predicted chances were from what happened. Lower is better; zero is perfect. For each driver, the scorer compares the probability of finishing at or above every position with the official result. Errors farther from the outcome affect more thresholds. This rewards accurate distributions and penalizes misplaced confidence.

RPS = (1 / N²) × Σ drivers Σ positions (F(p) − O(p))²

N is the field size. F(p) is the forecast probability of finishing P1 through Pp; O(p) is 1 if the driver actually finished at or above that position, otherwise 0. The reported race score averages across all drivers and all N position thresholds. Lower is better; a perfect distribution scores zero.

NC, DNS and DSQ sit beyond the last numbered position for RPS, so it does not distinguish those three outcomes. The log-loss metric evaluates their separate probabilities. A late retirement can still have a numbered classification; retirement is scored separately.

All numerical error scores are lower-is-better. Sorting the driver table by podium chance answers a different question from sorting by average rank. Neither changes the original probabilities.

Each shared, completed race contributes equally to the season score: the arithmetic mean of its race RPS values. Only races with valid scores for every model in the comparison enter the aggregate. A missing forecast never counts as zero.

Coverage shows a model’s scored forecasts out of races with at least one score. Partially scored races remain available in the race details, but are excluded from the shared-race average until all models have scores.

The cumulative chart shows the average through each included round. The per-race chart shows the individual race score. Changes reflect both forecasting performance and differences in circuits and race outcomes; a trend alone does not isolate the effect of memory or learning.

Official classifications are supplied manually using the final FIA result. The scorer validates every driver and the classification structure. A changed result after a completed review is flagged before the next forecast.

Completed attempts are reused, never replaced. Recorded failed attempts can be retried explicitly before the race deadline; the failed files and traces are archived. Each retry has a fresh budget. Running or incomplete attempts are not automatically restarted.

Original predictions, reports, explanations, run metadata, full trajectories, and notes snapshots are retained locally. This site is generated from those saved files. Schema validation checks probability consistency and required report fields; it does not independently prove that a cited source supports a claim.