147 lines
6.7 KiB
Markdown
147 lines
6.7 KiB
Markdown
# Optimal grid search over `conf.toml` space
|
||
|
||
This directory contains a small pipeline for mapping the **phase diagram** of
|
||
the foraging simulator: which team wins as you vary two `conf.toml` parameters.
|
||
It is built for the specific experiment *red aggressive vs. blue pacifist,
|
||
varying `view_dist` and the ball weapon `radius`*, but generalises to any pair
|
||
of numeric fields.
|
||
|
||
## The idea
|
||
|
||
A naive sweep would run a fixed, large number of seeds for **every** grid point.
|
||
Most of that work is wasted: a point deep in red territory is decided after a
|
||
handful of seeds, while only points near the phase boundary need many.
|
||
|
||
The pipeline therefore combines three things:
|
||
|
||
1. **Sequential stopping.** Each grid point is evaluated in parallel batches of
|
||
seeds. After every batch we test the *n*-sigma stopping rule and stop as soon
|
||
as the outcome is certain (or a hard cap is hit). Decisive points cost ~64
|
||
seeds; only boundary points pay for the full budget.
|
||
2. **Adaptive refinement.** After a coarse grid, cells whose four corners do not
|
||
agree on a winner are refined (centre + edge midpoints) so extra samples are
|
||
spent where the boundary actually is. Repeatable.
|
||
3. **Common random numbers.** Every point is evaluated with the *same* seed
|
||
sequence `0, 1, 2, …`, which removes seed-to-seed noise from comparisons
|
||
between neighbouring grid points and sharpens the apparent boundary.
|
||
|
||
## The statistic and the stopping rule
|
||
|
||
Each seed produces scores `(red, blu)` summing to at most `food_amount`. Define
|
||
the per-seed outcome
|
||
|
||
```
|
||
d = (red - blu) / food_amount # in [-1, 1], 0 == even match
|
||
```
|
||
|
||
`d > 0` means red out-foraged blue, `d < 0` means blue did. Across `N` seeds we
|
||
track the running mean `m`, sample standard deviation `s`, and standard error
|
||
`SE = s / sqrt(N)`. The stopping rule is
|
||
|
||
```
|
||
|m| / SE >= sigma (default sigma = 3)
|
||
```
|
||
|
||
i.e. the RED-BLU food-share difference is at least `sigma` standard errors from
|
||
zero. The winning side is the sign of `m`. A point that reaches
|
||
`--max-samples` without satisfying the rule is reported as **`tie`**
|
||
(undecided) — that is exactly the phase-boundary band.
|
||
|
||
> **Caveat — optional stopping.** Looking at the data repeatedly and stopping at
|
||
> `n` sigma inflates the true type-I error (the "look-elsewhere" effect), so the
|
||
> nominal confidence is optimistic. For a phase *diagram* this is fine and
|
||
> standard; if you need calibrated error bars, raise `--sigma`, or use the
|
||
> per-point `n`, `sd_diff` and `sigma` columns to apply a sequential correction
|
||
> (e.g. a Bonferroni or alpha-spending adjustment) afterwards.
|
||
|
||
## Architecture
|
||
|
||
| Piece | Role |
|
||
| --- | --- |
|
||
| `simulator --sequential` | New flag on the Rust binary (in `src/main.rs`). Runs seeds in parallel batches, applies the n-sigma rule, prints one full-precision JSON line and exits. Additive: the GUI and fixed-count headless paths are untouched, and all existing tests still pass. |
|
||
| `tools/sweep.py` | Grid driver. Renders a `conf.toml` per point, fans points out over a thread pool (each simulator process pinned to 1 rayon thread), does coarse + adaptive passes, and appends resumable JSONL + CSV. |
|
||
| `tools/plot_phase.py` | Matplotlib phase diagram: discrete winner regions, the `mean_diff = 0` boundary, and the continuous difference field. |
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# build once
|
||
cargo build --release --manifest-path simulator/Cargo.toml
|
||
|
||
# coarse grid only (fast)
|
||
python3 tools/sweep.py --view-dist 100:550:25 --radius 0:250:12.5
|
||
|
||
# coarse + one adaptive refinement pass, then plot
|
||
python3 tools/sweep.py \
|
||
--view-dist 100:550:25 --radius 0:250:12.5 \
|
||
--sigma 3 --max-samples 2000 \
|
||
--refine 1 --refine-max-samples 2500 --seed-budget 3000000 \
|
||
--workers 16 --rayon-threads 1 \
|
||
--out sweep_out/results.jsonl
|
||
python3 tools/plot_phase.py sweep_out/results.jsonl -o sweep_out/phase.png
|
||
|
||
# preview the grid without running
|
||
python3 tools/sweep.py --dry-run
|
||
|
||
# pin other fields with --set (repeatable)
|
||
python3 tools/sweep.py --set kills_user=false --set food_amount=50
|
||
```
|
||
|
||
Axes accept either a range (`start:stop:step`) or an explicit list
|
||
(`0,10,20,30`).
|
||
|
||
### Key options
|
||
|
||
| Flag | Meaning |
|
||
| --- | --- |
|
||
| `--view-dist`, `--radius` | the two swept axes |
|
||
| `--sigma` | n-sigma stopping threshold for the coarse pass (default 3) |
|
||
| `--min-samples` / `--max-samples` | lower bound before testing / hard cap |
|
||
| `--batch` | seeds per parallel batch between rule checks |
|
||
| `--refine` | number of adaptive refinement passes |
|
||
| `--refine-sigma` / `--refine-max-samples` | stopping settings for refined points |
|
||
| `--seed-budget` | global safety valve: stop scheduling once this many seeds have run |
|
||
| `--workers` / `--rayon-threads` | points in parallel / simulator threads per point |
|
||
| `--set KEY=VALUE` | pin any other `conf.toml` field |
|
||
| `--out` | JSONL path (a `.csv` is written alongside; both are resumable) |
|
||
|
||
### Output columns
|
||
|
||
`view_dist, weapon_radius, n, sigma, winner, stopped, mean_red, mean_blu,
|
||
mean_diff, sd_diff, se_diff, red_wins, blu_wins, ties, weapon_kills_user,
|
||
food_amount`
|
||
|
||
- `n` — seeds actually run for this point
|
||
- `sigma` — `|mean_diff| / se_diff` achieved
|
||
- `winner` — `red` / `blu` (sign of `mean_diff`)
|
||
- `stopped` — `sigma` (decided) or `max_samples` (undecided → `tie` in the plot)
|
||
|
||
## Result of the bundled experiment
|
||
|
||
`conf.toml` (red aggressive, blue pacifist, `forwardball`, `kills_user = true`),
|
||
coarse grid `view_dist ∈ [100, 550] × 25`, `radius ∈ [0, 250] × 12.5`, one
|
||
refinement pass, `sigma = 3`: **714 points, ~650k seeds** (a fixed 2000-seed
|
||
sweep over the same grid would have been ~1.4M seeds).
|
||
|
||
`phase.png` shows a clear transition:
|
||
|
||
- **Low `view_dist` (≲ 140):** blue wins almost everywhere. Red starts far from
|
||
the food and, with little sensing, forages no better than the pacifist while
|
||
paying for an unused weapon.
|
||
- **Higher `view_dist`:** red wins above a curved `radius` threshold and blue
|
||
below it. The boundary is *non-monotonic*: the radius red needs dips to a
|
||
minimum near `view_dist ≈ 250` and rises again towards 550, because the
|
||
`forwardball` sits at `view_end` (`view_dist` ahead of the robot), so a large
|
||
`view_dist` places the blast far ahead and needs a larger radius to connect.
|
||
- The boundary band (`tie` points) is where `|mean_diff| / SE < 3` even after the
|
||
sample cap.
|
||
|
||
## Extending
|
||
|
||
- **Other axes:** point `--view-dist`/`--radius` at any numeric field, or add a
|
||
third axis. For a third axis, wrap `sweep.py` in a loop over the extra value
|
||
and use separate output files, then plot a slice montage.
|
||
- **Other statistics:** the sequential mode currently tracks the food-share
|
||
difference. Win-rate (`red_wins`/`blu_wins`) is already recorded if you prefer
|
||
a Bernoulli statistic.
|
||
- **Finer boundary:** more `--refine` passes and/or a larger `--seed-budget`.
|