Beginner Guide

Prediction Market Calibration

How market prices map to real-world chances and how researchers check that mapping.

By Top Prediction Markets EditorialReviewed July 20, 20263 min read

Answer first

In simple terms, prediction market calibration asks whether a price that looks like X% actually corresponds to an X% chance in reality. Researchers measure this by comparing market-implied probabilities to observed event frequencies (using tools like reliability diagrams and the Brier score) and by checking consistency across bins or time. Markets tend to be well-calibrated for well-defined, near-term events with active trading and less so for rare events, long horizons, or thin markets.

What it means

In simple terms, a probability is calibrated when it matches reality over many similar cases. If a prediction market price implies a 62% chance for many events, then roughly 62% of those events should happen.

A Yes contract — an event contract that pays $1 if the event happens — is how most markets report probabilities. The market price of that contract is the implied probability.

Calibration is different from accuracy. A market can rank outcomes correctly (higher prices for more likely events) but still be miscalibrated if the numerical probabilities are systematically too high or too low.

Why it matters

The key thing to know is that calibration gives you a reality check on what the numbers mean. Good calibration means you can interpret a price as an honest estimate of chance. Poor calibration means the price still might be useful for ranking or eliciting belief, but not as a direct probability.

  • For researchers and modelers, calibration tells whether markets are trustworthy probability forecasts.
  • For policy and decision-makers, calibrated probabilities allow straightforward expected-value reasoning.

How it works

Here's the basic idea researchers use to check calibration:

  1. Convert prices to implied probabilities. For a Yes contract priced at $0.62, the implied probability is 62%.

  2. Group (bin) predictions. Researchers often collect many market predictions and sort them into probability bins (for example, 0–10%, 10–20%, …, 90–100%).

  3. Compare observed frequencies to predicted probabilities. For each bin, calculate the fraction of events that actually happened. A well-calibrated bin centered at 62% should have about 62% of events occur.

  4. Visualize with a reliability diagram (calibration curve). Plot observed frequency (y-axis) against predicted probability (x-axis). Perfect calibration lies on the 45° line.

  5. Use numeric summaries. The Brier score measures squared error between prediction and outcome (0 or 1). Lower Brier scores indicate better probabilistic forecasts. Researchers also decompose the Brier score into reliability (calibration), resolution, and uncertainty to isolate calibration problems.

  6. Check stability and sample size. Calibration can look different over short windows or with few events. Good practice is to test over many events and, if possible, over multiple markets or time periods.

Researchers also apply statistical tests (for example, Hosmer–Lemeshow–style tests) and machine-learning calibration methods (like isotonic regression or logistic calibration) when they want to correct raw market probabilities.

A simple example

A simple example shows what calibration means for a single price and a set of events. Suppose a prediction market offers a Yes contract for a certain type of company completing a product launch within three months at $0.62.

If you buy one Yes contract at $0.62, you pay $0.62. If the company completes the launch, the contract pays $1, so your gain before fees is $0.38. If the company does not complete the launch, the contract expires at $0, and your loss is $0.62.

Now imagine you observe 100 similar product-launch markets where the price was near $0.62 at the same time horizon. If 62 of those 100 launches succeed, the market price is well-calibrated for that class of events. If 80 succeed, the market has been underestimating the true chance; if 40 succeed, it has been overestimating.

Common mistakes

Treating a single price as reliably calibrated

One market price by itself tells you nothing about calibration. Calibration is about averages across many similar predictions. Small samples can look wildly miscalibrated even when the underlying process is fine.

Confusing calibration with sharpness or accuracy

A market can produce sharply different prices across events (good resolution) yet still be miscalibrated. Sharpness means the market avoids 50/50 answers; calibration means those numbers match outcomes.

Ignoring time and market conditions

Prices change. A probability that looks calibrated at one moment may become miscalibrated after new information arrives, when trading thins, or when incentives shift. Calibration checks should respect the same horizon and market state used when the price was quoted.

Related concepts

Frequently asked questions

What is a reliability diagram and how do I read one?

A reliability diagram plots observed event frequencies (y-axis) against predicted probabilities (x-axis). Points or a curve on the 45° line indicate good calibration; deviations show under- or overconfidence.

What is the Brier score and why use it?

The Brier score is the mean squared error between predicted probabilities and actual outcomes (0 or 1). It summarizes both calibration and resolution; lower values mean better probabilistic forecasts.

Are prediction market prices always calibrated?

No. Markets are often well-calibrated for many short-term, well-defined events with active trading, but they can be miscalibrated for rare events, long horizons, or thinly traded markets.

How do researchers correct miscalibrated market probabilities?

They may use statistical post-processing like isotonic regression or logistic calibration to adjust raw probabilities, or recalibrate using historical performance grouped by similar event types and horizons.

Are prediction markets legal everywhere?

Rules vary by location and platform. See our dedicated guide on whether prediction markets are legal in the US.

Related guides