# Experiment Conclusions

Running log of findings as experiments accumulate. Each section corresponds to a test run.

---

## Run 1 -- GPT-4o, 100 LLM iterations (2026-03-21)

**Configuration:** `gpt-4o`, 100 LLM calls, 1,000,000 PRNG samples, temperature=1.0, max_tokens=5

### Raw Numbers

| State | Expected | LLM Actual | LLM Delta | PRNG Actual | PRNG Delta |
|-------|----------|-----------|-----------|-------------|------------|
| State 1 | 3.0% | 0.00% | -3.00% | 2.97% | -0.03% |
| State 2 | 5.0% | 0.00% | -5.00% | 5.03% | +0.03% |
| State 3 | 8.5% | 0.00% | -8.50% | 8.48% | -0.02% |
| State 4 | 11.5% | 0.00% | -11.50% | 11.49% | -0.01% |
| State 5 | 15.0% | 0.00% | -15.00% | 15.00% | -0.00% |
| State 6 | 20.0% | **100.00%** | **+80.00%** | 19.97% | -0.03% |
| State 7 | 15.0% | 0.00% | -15.00% | 15.04% | +0.04% |
| State 8 | 10.0% | 0.00% | -10.00% | 10.00% | +0.00% |
| State 9 | 7.0% | 0.00% | -7.00% | 7.01% | +0.01% |
| State 10 | 5.0% | 0.00% | -5.00% | 5.00% | -0.00% |

### Statistics

| Metric | LLM | PRNG |
|--------|-----|------|
| Max deviation | 80.00% | 0.04% |
| Mean absolute deviation | 16.00% | 0.02% |
| Chi-squared | 400.00 | 5.66 |
| p-value | 0.0000 | 0.7751 |
| Verdict | **FAIL** | PASS |

### Key Observations

1. **Total mode collapse.** GPT-4o selected State 6 in 100 out of 100 calls (100.0%). Not a single other state was ever chosen. This is not a sampling bias -- it is a complete deterministic collapse onto the highest-probability option.

2. **The argmax problem.** When asked to "pick a state according to a distribution," GPT-4o interprets this as "pick the most likely state." This is consistent with how language models fundamentally work: they are trained to maximize the likelihood of the next token, and without explicit randomness injection at the application level, they default to the mode of any distribution they are presented with. Even with `temperature=1.0`, the model's understanding of "pick according to probability" is semantic, not stochastic -- it *knows* State 6 is the most probable and picks it every time.

3. **No evidence of probabilistic intuition at this scale.** The experiment's hypothesis was that an LLM might over-represent the mode while still showing some spread (the "bell curve bias"). The actual result is far more extreme: there is zero spread. The model does not simulate randomness at all -- it performs deterministic selection of the argmax.

4. **The PRNG baseline is near-perfect.** With 1,000,000 iterations, `random.choices()` deviates by at most 0.04% from the expected distribution. The chi-squared p-value of 0.7751 indicates the PRNG results are statistically indistinguishable from the theoretical distribution. This confirms the baseline is trustworthy.

5. **Latency pattern.** Mean response latency was ~1.0s per call. The first batch (iterations 0-49) averaged ~1.9s; the second batch (50-99) averaged ~0.7s, suggesting connection pooling or server-side caching after warmup. All 100 responses parsed successfully (0 unparsed).

### Interpretation

This result does not mean GPT-4o "doesn't understand probability." It clearly does -- it correctly identified State 6 as the most probable. The issue is that **understanding a distribution and sampling from one are fundamentally different capabilities.** The model can describe, analyze, and reason about the distribution perfectly, but when asked to *act as* a sampler, it collapses to a point estimate (the mode).

This has implications for any application that asks an LLM to make "random" or "probabilistic" choices -- NPC behavior in games, Monte Carlo simulations, decision-making under uncertainty. Without external randomness, the LLM's choices will be systematically biased toward the most likely outcome.

### Open Questions for Future Runs

- Does increasing iterations to 1,000 or 10,000 change the behavior, or is State 6 always selected?
- Does a different model (GPT-4.1, GPT-4o-mini, Claude) show any spread?
- Does rephrasing the prompt (e.g., "simulate a random draw" vs. "pick one state") affect the distribution?
- Does injecting a seed phrase or asking the model to "be surprising" introduce any variability?
- At what `temperature` setting (if any) does the model begin to show spread across states?

---

## Part 2: State Transitions -- Full Run (GPT-4.1, 10,000 iterations, 2026-03-21)

**Configuration:** `gpt-4.1`, 10,000 paths per mode, 1,000,000 PRNG paths, batch_size=200

- Chained mode: 10,000 valid / 0 errors, completed in 522.7s (~19 paths/s)
- Single-shot mode: 10,000 valid / 0 errors, completed in 171.2s (~58 paths/s)

### Layer 1 Distribution

| State | Expected | Chained | Single-Shot | PRNG |
|-------|----------|---------|-------------|------|
| S1 Input | 15.0% | **0.00%** | 4.19% | 14.99% |
| S2 Compute | 30.0% | **99.40%** | **50.47%** | 30.01% |
| S3 Validate | 25.0% | 0.60% | 21.83% | 25.02% |
| S4 Optimize | 20.0% | 0.00% | 23.51% | 19.96% |
| S5 Output | 10.0% | **0.00%** | **0.00%** | 10.03% |

### Statistics

| Metric | Chained | Single-Shot |
|--------|---------|-------------|
| L1 chi-squared | 22,935.97 | 3,277.57 |
| L1 p-value | 0.0000 | 0.0000 |
| Verdict | FAIL | FAIL |

### Key Findings

1. **Near-total mode collapse in chained mode.** With 10,000 iterations, gpt-4.1 selected S2 (Compute, 30% expected) in 99.40% of chained paths. States S1, S4, and S5 were never selected. Only S3 appeared occasionally at 0.60%. This is even more extreme than Part 1's argmax behavior -- the model locks onto the single highest-probability option almost without exception.

2. **Single-shot mode distributes more, but with systematic distortion.** When the full tree is shown in one prompt, the model spreads across 4 of 5 states but still over-represents S2 (50.47% vs 30% expected) and completely ignores S5 (0% vs 10% expected). S4 is slightly over-represented (23.51% vs 20%), suggesting the model partially "fills in" missing probability mass from S5 into S4.

3. **S5 "Output" is invisible to the model.** Across both modes, the model never selected S5 (10% expected). The model appears to have a floor below which a probability is treated as zero. S1 (15%) was also ignored in chained mode but appeared at 4.19% in single-shot, suggesting the threshold is somewhere between 10-15% in chained mode and lower in single-shot mode.

4. **Chained mode amplifies argmax bias.** The sequential structure of chained mode appears to strengthen the model's tendency to pick the mode. When presented with just 5 options at Layer 1, the model has no "tree context" to remind it that variety is expected. In contrast, single-shot mode shows the full tree, which may cue the model that distributional variety is needed.

5. **Chi-squared values confirm catastrophic failure.** Chained L1 chi2 = 22,935.97 and single L1 chi2 = 3,277.57, both with p = 0.0000. The PRNG baseline (not shown in stats) is near-perfect as expected. The single-shot mode is ~7x better than chained by chi2, but still astronomically far from the expected distribution.

6. **Zero parse failures across 40,000 API calls.** All 10,000 chained paths (30,000 API calls) and 10,000 single-shot paths parsed successfully. The prompts and response format are robust at scale.

### Interpretation

The hierarchical experiment reveals a spectrum of probabilistic failure:
- **Chained mode** is essentially a deterministic argmax selector at each layer, producing a "greedy path" through the tree
- **Single-shot mode** shows the model has some awareness of distributional shape when it can see the full picture, but still dramatically distorts the distribution

This has direct implications for agentic AI systems that use LLMs for decision-making through state machines or workflow routing. Without external randomness injection, any state machine driven by LLM "choices" will collapse into a narrow set of dominant paths, ignoring low-probability branches entirely.

### Open Questions

- Do newer models (gpt-5.4, gpt-5.4-mini) show any improvement in distributional spread?
- Does the conditional distribution within Layer 2 (given S2 was picked) match the expected sub-state probabilities?
- Would prompt engineering (e.g., "you must sometimes pick unlikely states") reduce the argmax bias?
- Does temperature > 1.0 (if supported) help distribute across states?

---

## Part 1: Multi-Model Comparison -- 1,000,000 Iterations Each (2026-03-21)

**Configuration:** 4 models x 1,000,000 LLM calls each, 1,000,000 PRNG samples per model, temperature=1.0, batch_size=500

Total: **4,000,000 API calls**, 0 errors across all models.

### Per-State Distribution Across Models

| State | Expected | gpt-5.4-mini | gpt-5.4 | gpt-5-mini | gpt-4.1 |
|-------|----------|-------------|---------|-----------|---------|
| State 1 (3%) | 3.0% | 0.00% | 0.00% | 0.00% | 0.00% |
| State 2 (5%) | 5.0% | 0.00% | 1.77% | 0.00% | 0.00% |
| State 3 (8.5%) | 8.5% | 0.00% | 1.68% | 0.00% | 0.00% |
| State 4 (11.5%) | 11.5% | 0.00% | 7.11% | 0.00% | 0.00% |
| State 5 (15%) | 15.0% | 0.00% | 5.53% | 0.00% | 0.00% |
| **State 6 (20%)** | **20.0%** | **99.94%** | **55.75%** | **100.00%** | **100.00%** |
| State 7 (15%) | 15.0% | 0.06% | 13.76% | 0.00% | 0.00% |
| State 8 (10%) | 10.0% | 0.00% | 10.59% | 0.00% | 0.00% |
| State 9 (7%) | 7.0% | 0.00% | 2.33% | 0.00% | 0.00% |
| State 10 (5%) | 5.0% | 0.00% | 1.49% | 0.00% | 0.00% |

### Summary Statistics

| Model | State 6 % | Max Deviation | Mean |D| | Chi-squared | Runtime |
|-------|-----------|---------------|---------|-------------|---------|
| gpt-5.4-mini | 99.94% | 79.94% | 16.00% | 3,993,864 | 10,654s |
| **gpt-5.4** | **55.75%** | **35.75%** | **7.27%** | **878,295** | 9,676s |
| gpt-5-mini | 100.00% | 80.00% | 16.00% | 3,999,808 | 8,942s |
| gpt-4.1 | 100.00% | 80.00% | 16.00% | 4,000,000 | 9,422s |

All PRNG baselines: max deviation < 0.07%, p-values > 0.28 (statistically perfect).

### Key Findings

1. **GPT-5.4 is the only model with distributional spread.** Across 1,000,000 calls, gpt-5.4 distributed across all 10 states, with State 6 at 55.75% (vs 20% expected). While still heavily biased, it picked every state at least 1.49% of the time. Its chi-squared (878,295) is ~4.5x lower than the other models, indicating meaningfully better probabilistic behavior.

2. **Three models show complete or near-complete mode collapse:**
   - **gpt-4.1**: 100.00% State 6 across 1M calls. Zero spread.
   - **gpt-5-mini**: 100.00% State 6 (with 3 unparsed responses out of 1M). Zero spread.
   - **gpt-5.4-mini**: 99.94% State 6, with exactly 600 picks of State 7 (0.06%) and nothing else. Near-zero spread.

3. **Model intelligence does not predict probabilistic ability.** gpt-5.4 (the most capable/expensive model) is the only one with spread. gpt-5-mini and gpt-4.1 (both strong models) show identical 100% collapse. This suggests probabilistic sampling is a distinct capability that scales with model size/architecture but not in a simple linear way.

4. **gpt-5.4's distribution shape roughly matches the expected shape.** While gpt-5.4 dramatically over-represents State 6, the relative ordering of other states approximately matches expectations: States 7 and 8 (highest after 6) get the most representation (13.76% and 10.59%), while tail states (1, 2, 10) get the least (0-1.77%). The model "understands" the shape of the distribution but compresses it toward the mode.

5. **The 0.06% bleed in gpt-5.4-mini is interesting.** Out of 1M calls, exactly 600 went to State 7 (the second-highest probability at 15%). This tiny leakage suggests the model's "argmax certainty" is not perfectly deterministic -- there is an infinitesimal probability mass assigned to the runner-up state. This may be an artifact of the temperature=1.0 setting interacting with the model's output logits.

6. **Zero unparsed responses (except gpt-5-mini).** gpt-5.4-mini, gpt-5.4, and gpt-4.1 all returned 1,000,000/1,000,000 valid responses. gpt-5-mini had 3 unparsed responses out of 1,000,000 (0.0003%), which is statistically negligible.

### Interpretation

The headline finding is a clear capability hierarchy:

```
gpt-5.4  >>  gpt-5.4-mini  >>  gpt-5-mini = gpt-4.1
  (spread)    (0.06% bleed)     (total collapse)
```

gpt-5.4 appears to have been trained with some degree of stochastic sampling capability -- it can approximate (poorly, but nonzero) a probability distribution when asked to. The other models treat the task as a classification problem and return the argmax.

This has important implications:
- **For agentic systems:** Only the most capable models show any distributional awareness. Using smaller models as "random choice makers" in agent pipelines will produce deterministic behavior.
- **For game AI / simulation:** LLMs cannot substitute for PRNGs. Even gpt-5.4's best effort is off by 35.75% on its worst state.
- **For benchmarking:** "Probabilistic intuition" could serve as a novel benchmark dimension for evaluating model capabilities beyond reasoning and knowledge.

---

## Part 2: Multi-Model State Transitions -- 100,000 Iterations Each (2026-03-21)

**Configuration:** 4 models, 100,000 paths per mode (chained + single-shot), 1,000,000 PRNG paths, batch_size=200

Total: ~1,600,000 API calls across all models (300K chained + 100K single per model).

### Layer 1 Distribution: All Models Compared

| State | Expected | 5.4-mini Ch | 5.4-mini Sg | 5.4 Ch | 5.4 Sg | 5-mini Ch | 5-mini Sg | 4.1 Ch | 4.1 Sg |
|-------|----------|------------|------------|--------|--------|----------|----------|--------|--------|
| S1 (15%) | 15.0% | 0.00% | 0.00% | 5.40% | 2.22% | 0.00% | 0.03% | 0.00% | 4.21% |
| S2 (30%) | 30.0% | 21.99% | 15.64% | **39.64%** | **35.61%** | **97.84%** | 31.92% | **99.14%** | **51.06%** |
| S3 (25%) | 25.0% | **64.24%** | 6.94% | 23.31% | 23.52% | 2.12% | **58.46%** | 0.84% | 21.90% |
| S4 (20%) | 20.0% | 13.77% | **77.42%** | **28.73%** | **38.66%** | 0.05% | 9.59% | 0.02% | 22.83% |
| S5 (10%) | 10.0% | 0.00% | 0.00% | 2.93% | 0.00% | 0.00% | 0.00% | 0.00% | 0.01% |

Ch = Chained mode, Sg = Single-shot mode.

### Summary Statistics

| Model | Mode | Valid | Errors | L1 chi2 | Runtime |
|-------|------|-------|--------|---------|---------|
| gpt-5.4-mini | Chained | 99,971 | 29 | 90,647 | 2,271s |
| gpt-5.4-mini | Single | 100,000 | 0 | 209,807 | 882s |
| gpt-5.4 | Chained | 99,999 | 1 | 18,162 | 2,413s |
| gpt-5.4 | Single | 100,000 | 0 | 39,445 | 1,649s |
| gpt-5-mini | Chained | 100,000 | 0 | 219,255 | 3,604s |
| gpt-5-mini | Single | 89,165 | 10,835 | 67,089 | 2,804s |
| gpt-4.1 | Chained | 99,993 | 7 | 227,637 | 2,249s |
| gpt-4.1 | Single | 100,000 | 0 | 33,314 | 770s |

### Key Findings

1. **Every model picks a different "favorite" state -- and it's NOT always the argmax.** This is the most surprising finding. In Part 1 (flat distribution), every model locked onto State 6 (the 20% mode). But in the 5-state transition tree:
   - **gpt-5.4-mini chained** favors **S3 (64.24%)** -- the second-highest probability state (25%), not S2 (30%)
   - **gpt-5.4-mini single** favors **S4 (77.42%)** -- the third-highest (20%)
   - **gpt-5-mini chained** favors **S2 (97.84%)** -- the actual argmax
   - **gpt-5-mini single** favors **S3 (58.46%)** -- not the argmax
   - **gpt-4.1** behaves like Part 1: chained 99.14% S2 (argmax), single 51.06% S2

   This means the argmax hypothesis from Part 1 was **only partially correct**. With different state labels and fewer options (5 vs 10), models develop idiosyncratic biases that aren't purely based on probability magnitude.

2. **gpt-5.4 is again the best distributer, by a large margin.** Its chi-squared values (18,162 chained, 39,445 single) are 5-12x lower than other models. In chained mode, it spreads across all 5 states (5.40%, 39.64%, 23.31%, 28.73%, 2.93%). It still over-represents S2 and under-represents S1/S5, but the shape is recognizable.

3. **Chained vs Single-shot produces radically different distributions within the same model.** gpt-5.4-mini picks S3 64% of the time in chained mode but S4 77% of the time in single-shot mode. The same model, same probabilities, completely different bias depending on whether it sees one layer at a time or the full tree. This suggests the model's "choice" is heavily influenced by prompt structure, not just probability values.

4. **S5 (Output, 10%) is universally ignored.** Across all 4 models and both modes, S5 received 0-2.93% (expected 10%). Only gpt-5.4 in chained mode gave it any meaningful representation (2.93%). This confirms the "probability floor" effect from the 10K run: probabilities below ~10-15% are effectively treated as zero.

5. **S1 (Input, 15%) is nearly invisible.** Only gpt-5.4 gives S1 significant representation (5.40% chained, 2.22% single). All other models assign it 0-4.21%. Despite being 15% expected, the model perceives it as a "low probability" option and suppresses it.

6. **gpt-5-mini single-shot had 10,835 parse failures (10.8%).** The reasoning model struggled with the single-shot format -- it needed to output a 3-part path ("S2 > S2.3 > S2.3.1") but frequently produced malformed responses. All other models had < 0.03% error rates.

### Model Behavior Taxonomy

Based on both Part 1 and Part 2, we can classify the models:

| Model | Behavior | Description |
|-------|----------|-------------|
| **gpt-5.4** | Approximate distributer | Distributes across all states with heavy mode bias. Best probabilistic intuition. |
| **gpt-5.4-mini** | Non-argmax collapser | Collapses to a single state but NOT always the highest probability one. Bias is label/position dependent. |
| **gpt-4.1** | Strict argmax | Reliably picks the highest-probability option. Deterministic and predictable. |
| **gpt-5-mini** | Strict argmax (chained) / Variable (single) | Argmax in chained mode, but in single-shot shows unexpected variety -- perhaps because reasoning tokens give it more "deliberation." |

### Interpretation

The transition experiment reveals that LLM probabilistic bias is far more nuanced than simple argmax:

- **Prompt structure matters more than probability values.** The same model produces wildly different distributions in chained vs single-shot mode, even though the underlying probabilities are identical.
- **Label semantics may influence choices.** gpt-5.4-mini's preference for "Validate" (S3) over "Compute" (S2) in chained mode might reflect semantic associations rather than probability reasoning.
- **Only gpt-5.4 attempts genuine distributional sampling.** It is the only model where the output distribution is recognizably related to the input distribution across both experiments.
- **The "probability floor" (~10%) is consistent.** States below 10% probability are systematically ignored across all models, suggesting a hard threshold in how models perceive and act on small probabilities.

---

*All planned experiments complete. Total API calls across all runs: ~5.6 million.*