LLM Stochastic Bias

Testing whether LLMs can sample from probability distributions or just pick the most likely option. 5.6M API calls across 4 OpenAI models. Spoiler: they can't.

5,600,000 API calls 4 models 2 experiments 0 models pass

The Question

When you tell an LLM "pick a state with 20% probability," does it actually produce that state ~20% of the time across many calls? Or does it just pick the most likely option every time?

This matters for any application using LLMs for decision-making, game AI, simulation, or agentic workflows where stochastic behavior is expected. We tested this with two experiments across four OpenAI models, making 5.6 million API calls total.

Part 1: Flat Distribution (10 States)

Each model is given a probability distribution over 10 states and asked to "pick one" per call. No code execution, no random library — just the model's own sense of probability. 1,000,000 independent calls per model, temperature=1.0.

The Target Distribution

graph LR
  subgraph dist [Each call: pick one state according to these probabilities]
    S1["S1\n3%"]
    S2["S2\n5%"]
    S3["S3\n8.5%"]
    S4["S4\n11.5%"]
    S5["S5\n15%"]
    S6["S6\n20%"]
    S7["S7\n15%"]
    S8["S8\n10%"]
    S9["S9\n7%"]
    S10["S10\n5%"]
  end

Part 1 Results: Model Comparison

The percentage shows how often each model picked State 6 (the 20% mode) out of 1,000,000 calls:

GPT-5.4-mini

Fast · $0.75/1M in

99.94%

State 6 (expected 20%)

Near-Collapse

GPT-5.4

Flagship · $2.50/1M in

55.75%

State 6 (expected 20%)

Distributes

GPT-5-mini

Reasoning · $0.25/1M in

100.0%

State 6 (expected 20%)

Total Collapse

GPT-4.1

Production · $2.00/1M in

100.0%

State 6 (expected 20%)

Total Collapse

Distribution Comparison (1M iterations each)

Deviation from Expected (%)

State	Expected	GPT-5.4-mini	GPT-5.4	GPT-5-mini	GPT-4.1

Distribution Shape (Radar)

GPT-5.4 Detail (only model with spread)

Statistical Divergence (lower = better)

Part 1 Key Findings

GPT-5.4 is the only model with distributional spread. It picked all 10 states, with State 6 at 55.75%. Still heavily biased, but the shape is recognizable.
Three models show total mode collapse. GPT-4.1 and GPT-5-mini picked State 6 in 100.00% of 1M calls. GPT-5.4-mini picked it 99.94%, with only 600 outliers to State 7.
Understanding ≠ sampling. The models clearly understand the distribution (they identify the mode correctly). But understanding a distribution and sampling from one are fundamentally different capabilities.
The PRNG baseline is perfect. Python's random.choices() deviates by <0.07% across all runs. The baseline is trustworthy.

Part 2: Multi-Layer State Transitions

Can the models follow conditional probability distributions across a 3-layer hierarchical state machine? Each iteration traces a path from root to leaf. Two modes: Chained (3 API calls, one per layer) and Single-shot (1 call, full tree shown). 100,000 paths per model per mode.

The 3-Layer State Tree

graph TD
  Root((" "))
  Root -->|"15%"| S1["S1 Input"]
  Root -->|"30%"| S2["S2 Compute"]
  Root -->|"25%"| S3["S3 Validate"]
  Root -->|"20%"| S4["S4 Optimize"]
  Root -->|"10%"| S5["S5 Output"]

  S1 -->|"40%"| S11["S1.1 Tokenize"]
  S1 -->|"35%"| S12["S1.2 Parse"]
  S1 -->|"25%"| S13["S1.3 Normalize"]

  S2 -->|"20%"| S21["S2.1 Arithmetic"]
  S2 -->|"30%"| S22["S2.2 Logic"]
  S2 -->|"35%"| S23["S2.3 Pattern Match"]
  S2 -->|"15%"| S24["S2.4 Heuristic"]

  S3 -->|"45%"| S31["S3.1 Type Check"]
  S3 -->|"30%"| S32["S3.2 Bounds Check"]
  S3 -->|"25%"| S33["S3.3 Constraints"]

  S4 -->|"45%"| S41["S4.1 Cache"]
  S4 -->|"25%"| S42["S4.2 Recompute"]
  S4 -->|"18%"| S43["S4.3 Prune"]
  S4 -->|"12%"| S44["S4.4 Parallelize"]

  S5 -->|"55%"| S51["S5.1 Format"]
  S5 -->|"45%"| S52["S5.2 Serialize"]

  S11 -->|"..."| L3a["Layer 3\n2-4 leaves each"]
  S23 -->|"..."| L3b["Layer 3\n2-4 leaves each"]
  S41 -->|"..."| L3c["Layer 3\n2-4 leaves each"]

Part 2 — Layer 1 Distribution (100K paths per model)

Chained Mode (3 API calls per path)

Single-Shot Mode (1 API call per path)

Each Model's "Favorite" State (highlights non-argmax bias)

Model	Expected Argmax	Chained Favorite	Chained %	Single Favorite	Single %

Part 2 — L1 Chi-Squared by Model & Mode (lower = better)

Part 2 Key Findings

Models don't always pick the argmax. GPT-5.4-mini favored S3 (25%) over S2 (30%) in chained mode, and S4 (20%) in single-shot. The bias is label/position dependent, not purely probability-driven.
Chained vs Single-shot produces completely different distributions from the same model. Prompt structure matters more than probability values.
GPT-5.4 is again the best distributer (chi-squared 5-12x lower than others). It spreads across all 5 states in both modes.
S5 "Output" (10%) is universally ignored. Probabilities below ~10% are treated as zero by all models — a consistent "probability floor" effect.

Model Behavior Taxonomy

Based on 5.6M API calls across both experiments:

Model	Behavior	Description
GPT-5.4	Approximate distributer	Distributes across all states with heavy mode bias. Best probabilistic intuition of any model tested. Chi-squared 4.5x lower than others.
GPT-5.4-mini	Non-argmax collapser	Collapses to a single state, but NOT always the highest-probability one. Bias is label and prompt-structure dependent.
GPT-4.1	Strict argmax	Reliably picks the highest-probability option. 100% deterministic in flat distributions. Predictable but useless as a sampler.
GPT-5-mini	Strict argmax / variable	Argmax in chained mode, but single-shot shows unexpected variety. Reasoning tokens may give more deliberation.

Implications

For agentic systems: Only the most capable models show any distributional awareness. Using smaller models as "random choice makers" in agent pipelines will produce deterministic behavior.
For game AI / simulation: LLMs cannot substitute for PRNGs. Even GPT-5.4's best effort is off by 35.75% on its worst state.
For benchmarking: "Probabilistic intuition" could serve as a novel benchmark dimension for evaluating model capabilities beyond reasoning and knowledge.