# Optimal grid search over `conf.toml` space

This directory contains a small pipeline for mapping the **phase diagram** of
the foraging simulator: which team wins as you vary two `conf.toml` parameters.
It is built for the specific experiment *red aggressive vs. blue pacifist,
varying `view_dist` and the ball weapon `radius`*, but generalises to any pair
of numeric fields.

## The idea

A naive sweep would run a fixed, large number of seeds for **every** grid point.
Most of that work is wasted: a point deep in red territory is decided after a
handful of seeds, while only points near the phase boundary need many.

The pipeline therefore combines three things:

1. **Sequential stopping.** Each grid point is evaluated in parallel batches of
   seeds. After every batch we test the *n*-sigma stopping rule and stop as soon
   as the outcome is certain (or a hard cap is hit). Decisive points cost ~64
   seeds; only boundary points pay for the full budget.
2. **Adaptive refinement.** After a coarse grid, cells whose four corners do not
   agree on a winner are refined (centre + edge midpoints) so extra samples are
   spent where the boundary actually is. Repeatable.
3. **Common random numbers.** Every point is evaluated with the *same* seed
   sequence `0, 1, 2, …`, which removes seed-to-seed noise from comparisons
   between neighbouring grid points and sharpens the apparent boundary.

## The statistic and the stopping rule

Each seed produces scores `(red, blu)` summing to at most `food_amount`. Define
the per-seed outcome

```
d = (red - blu) / food_amount          # in [-1, 1], 0 == even match
```

`d > 0` means red out-foraged blue, `d < 0` means blue did. Across `N` seeds we
track the running mean `m`, sample standard deviation `s`, and standard error
`SE = s / sqrt(N)`. The stopping rule is

```
|m| / SE >= sigma      (default sigma = 3)
```

i.e. the RED-BLU food-share difference is at least `sigma` standard errors from
zero. The winning side is the sign of `m`. A point that reaches
`--max-samples` without satisfying the rule is reported as **`tie`**
(undecided) — that is exactly the phase-boundary band.

> **Caveat — optional stopping.** Looking at the data repeatedly and stopping at
> `n` sigma inflates the true type-I error (the "look-elsewhere" effect), so the
> nominal confidence is optimistic. For a phase *diagram* this is fine and
> standard; if you need calibrated error bars, raise `--sigma`, or use the
> per-point `n`, `sd_diff` and `sigma` columns to apply a sequential correction
> (e.g. a Bonferroni or alpha-spending adjustment) afterwards.

## Architecture

| Piece | Role |
| --- | --- |
| `simulator --sequential` | New flag on the Rust binary (in `src/main.rs`). Runs seeds in parallel batches, applies the n-sigma rule, prints one full-precision JSON line and exits. Additive: the GUI and fixed-count headless paths are untouched, and all existing tests still pass. |
| `tools/sweep.py` | Grid driver. Renders a `conf.toml` per point, fans points out over a thread pool (each simulator process pinned to 1 rayon thread), does coarse + adaptive passes, and appends resumable JSONL + CSV. |
| `tools/plot_phase.py` | Matplotlib phase diagram: discrete winner regions, the `mean_diff = 0` boundary, and the continuous difference field. |

## Usage

```bash
# build once
cargo build --release --manifest-path simulator/Cargo.toml

# coarse grid only (fast)
python3 tools/sweep.py --view-dist 100:550:25 --radius 0:250:12.5

# coarse + one adaptive refinement pass, then plot
python3 tools/sweep.py \
    --view-dist 100:550:25 --radius 0:250:12.5 \
    --sigma 3 --max-samples 2000 \
    --refine 1 --refine-max-samples 2500 --seed-budget 3000000 \
    --workers 16 --rayon-threads 1 \
    --out sweep_out/results.jsonl
python3 tools/plot_phase.py sweep_out/results.jsonl -o sweep_out/phase.png

# preview the grid without running
python3 tools/sweep.py --dry-run

# pin other fields with --set (repeatable)
python3 tools/sweep.py --set kills_user=false --set food_amount=50
```

Axes accept either a range (`start:stop:step`) or an explicit list
(`0,10,20,30`).

### Key options

| Flag | Meaning |
| --- | --- |
| `--view-dist`, `--radius` | the two swept axes |
| `--sigma` | n-sigma stopping threshold for the coarse pass (default 3) |
| `--min-samples` / `--max-samples` | lower bound before testing / hard cap |
| `--batch` | seeds per parallel batch between rule checks |
| `--refine` | number of adaptive refinement passes |
| `--refine-sigma` / `--refine-max-samples` | stopping settings for refined points |
| `--seed-budget` | global safety valve: stop scheduling once this many seeds have run |
| `--workers` / `--rayon-threads` | points in parallel / simulator threads per point |
| `--set KEY=VALUE` | pin any other `conf.toml` field |
| `--out` | JSONL path (a `.csv` is written alongside; both are resumable) |

### Output columns

`view_dist, weapon_radius, n, sigma, winner, stopped, mean_red, mean_blu,
mean_diff, sd_diff, se_diff, red_wins, blu_wins, ties, weapon_kills_user,
food_amount`

- `n` — seeds actually run for this point
- `sigma` — `|mean_diff| / se_diff` achieved
- `winner` — `red` / `blu` (sign of `mean_diff`)
- `stopped` — `sigma` (decided) or `max_samples` (undecided → `tie` in the plot)

## Result of the bundled experiment

`conf.toml` (red aggressive, blue pacifist, `forwardball`, `kills_user = true`),
coarse grid `view_dist ∈ [100, 550] × 25`, `radius ∈ [0, 250] × 12.5`, one
refinement pass, `sigma = 3`: **714 points, ~650k seeds** (a fixed 2000-seed
sweep over the same grid would have been ~1.4M seeds).

`phase.png` shows a clear transition:

- **Low `view_dist` (≲ 140):** blue wins almost everywhere. Red starts far from
  the food and, with little sensing, forages no better than the pacifist while
  paying for an unused weapon.
- **Higher `view_dist`:** red wins above a curved `radius` threshold and blue
  below it. The boundary is *non-monotonic*: the radius red needs dips to a
  minimum near `view_dist ≈ 250` and rises again towards 550, because the
  `forwardball` sits at `view_end` (`view_dist` ahead of the robot), so a large
  `view_dist` places the blast far ahead and needs a larger radius to connect.
- The boundary band (`tie` points) is where `|mean_diff| / SE < 3` even after the
  sample cap.

## Extending

- **Other axes:** point `--view-dist`/`--radius` at any numeric field, or add a
  third axis. For a third axis, wrap `sweep.py` in a loop over the extra value
  and use separate output files, then plot a slice montage.
- **Other statistics:** the sequential mode currently tracks the food-share
  difference. Win-rate (`red_wins`/`blu_wins`) is already recorded if you prefer
  a Bernoulli statistic.
- **Finer boundary:** more `--refine` passes and/or a larger `--seed-budget`.