batch plotter via deepseek
This commit is contained in:
parent
764536be7a
commit
db234a0a53
4 changed files with 691 additions and 0 deletions
4
.gitignore
vendored
Normal file
4
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
# grid-search / phase-diagram outputs
|
||||||
|
sweep_out/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
147
tools/README.md
Normal file
147
tools/README.md
Normal file
|
|
@ -0,0 +1,147 @@
|
||||||
|
# Optimal grid search over `conf.toml` space
|
||||||
|
|
||||||
|
This directory contains a small pipeline for mapping the **phase diagram** of
|
||||||
|
the foraging simulator: which team wins as you vary two `conf.toml` parameters.
|
||||||
|
It is built for the specific experiment *red aggressive vs. blue pacifist,
|
||||||
|
varying `view_dist` and the ball weapon `radius`*, but generalises to any pair
|
||||||
|
of numeric fields.
|
||||||
|
|
||||||
|
## The idea
|
||||||
|
|
||||||
|
A naive sweep would run a fixed, large number of seeds for **every** grid point.
|
||||||
|
Most of that work is wasted: a point deep in red territory is decided after a
|
||||||
|
handful of seeds, while only points near the phase boundary need many.
|
||||||
|
|
||||||
|
The pipeline therefore combines three things:
|
||||||
|
|
||||||
|
1. **Sequential stopping.** Each grid point is evaluated in parallel batches of
|
||||||
|
seeds. After every batch we test the *n*-sigma stopping rule and stop as soon
|
||||||
|
as the outcome is certain (or a hard cap is hit). Decisive points cost ~64
|
||||||
|
seeds; only boundary points pay for the full budget.
|
||||||
|
2. **Adaptive refinement.** After a coarse grid, cells whose four corners do not
|
||||||
|
agree on a winner are refined (centre + edge midpoints) so extra samples are
|
||||||
|
spent where the boundary actually is. Repeatable.
|
||||||
|
3. **Common random numbers.** Every point is evaluated with the *same* seed
|
||||||
|
sequence `0, 1, 2, …`, which removes seed-to-seed noise from comparisons
|
||||||
|
between neighbouring grid points and sharpens the apparent boundary.
|
||||||
|
|
||||||
|
## The statistic and the stopping rule
|
||||||
|
|
||||||
|
Each seed produces scores `(red, blu)` summing to at most `food_amount`. Define
|
||||||
|
the per-seed outcome
|
||||||
|
|
||||||
|
```
|
||||||
|
d = (red - blu) / food_amount # in [-1, 1], 0 == even match
|
||||||
|
```
|
||||||
|
|
||||||
|
`d > 0` means red out-foraged blue, `d < 0` means blue did. Across `N` seeds we
|
||||||
|
track the running mean `m`, sample standard deviation `s`, and standard error
|
||||||
|
`SE = s / sqrt(N)`. The stopping rule is
|
||||||
|
|
||||||
|
```
|
||||||
|
|m| / SE >= sigma (default sigma = 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
i.e. the RED-BLU food-share difference is at least `sigma` standard errors from
|
||||||
|
zero. The winning side is the sign of `m`. A point that reaches
|
||||||
|
`--max-samples` without satisfying the rule is reported as **`tie`**
|
||||||
|
(undecided) — that is exactly the phase-boundary band.
|
||||||
|
|
||||||
|
> **Caveat — optional stopping.** Looking at the data repeatedly and stopping at
|
||||||
|
> `n` sigma inflates the true type-I error (the "look-elsewhere" effect), so the
|
||||||
|
> nominal confidence is optimistic. For a phase *diagram* this is fine and
|
||||||
|
> standard; if you need calibrated error bars, raise `--sigma`, or use the
|
||||||
|
> per-point `n`, `sd_diff` and `sigma` columns to apply a sequential correction
|
||||||
|
> (e.g. a Bonferroni or alpha-spending adjustment) afterwards.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
| Piece | Role |
|
||||||
|
| --- | --- |
|
||||||
|
| `simulator --sequential` | New flag on the Rust binary (in `src/main.rs`). Runs seeds in parallel batches, applies the n-sigma rule, prints one full-precision JSON line and exits. Additive: the GUI and fixed-count headless paths are untouched, and all existing tests still pass. |
|
||||||
|
| `tools/sweep.py` | Grid driver. Renders a `conf.toml` per point, fans points out over a thread pool (each simulator process pinned to 1 rayon thread), does coarse + adaptive passes, and appends resumable JSONL + CSV. |
|
||||||
|
| `tools/plot_phase.py` | Matplotlib phase diagram: discrete winner regions, the `mean_diff = 0` boundary, and the continuous difference field. |
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# build once
|
||||||
|
cargo build --release --manifest-path simulator/Cargo.toml
|
||||||
|
|
||||||
|
# coarse grid only (fast)
|
||||||
|
python3 tools/sweep.py --view-dist 100:550:25 --radius 0:250:12.5
|
||||||
|
|
||||||
|
# coarse + one adaptive refinement pass, then plot
|
||||||
|
python3 tools/sweep.py \
|
||||||
|
--view-dist 100:550:25 --radius 0:250:12.5 \
|
||||||
|
--sigma 3 --max-samples 2000 \
|
||||||
|
--refine 1 --refine-max-samples 2500 --seed-budget 3000000 \
|
||||||
|
--workers 16 --rayon-threads 1 \
|
||||||
|
--out sweep_out/results.jsonl
|
||||||
|
python3 tools/plot_phase.py sweep_out/results.jsonl -o sweep_out/phase.png
|
||||||
|
|
||||||
|
# preview the grid without running
|
||||||
|
python3 tools/sweep.py --dry-run
|
||||||
|
|
||||||
|
# pin other fields with --set (repeatable)
|
||||||
|
python3 tools/sweep.py --set kills_user=false --set food_amount=50
|
||||||
|
```
|
||||||
|
|
||||||
|
Axes accept either a range (`start:stop:step`) or an explicit list
|
||||||
|
(`0,10,20,30`).
|
||||||
|
|
||||||
|
### Key options
|
||||||
|
|
||||||
|
| Flag | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `--view-dist`, `--radius` | the two swept axes |
|
||||||
|
| `--sigma` | n-sigma stopping threshold for the coarse pass (default 3) |
|
||||||
|
| `--min-samples` / `--max-samples` | lower bound before testing / hard cap |
|
||||||
|
| `--batch` | seeds per parallel batch between rule checks |
|
||||||
|
| `--refine` | number of adaptive refinement passes |
|
||||||
|
| `--refine-sigma` / `--refine-max-samples` | stopping settings for refined points |
|
||||||
|
| `--seed-budget` | global safety valve: stop scheduling once this many seeds have run |
|
||||||
|
| `--workers` / `--rayon-threads` | points in parallel / simulator threads per point |
|
||||||
|
| `--set KEY=VALUE` | pin any other `conf.toml` field |
|
||||||
|
| `--out` | JSONL path (a `.csv` is written alongside; both are resumable) |
|
||||||
|
|
||||||
|
### Output columns
|
||||||
|
|
||||||
|
`view_dist, weapon_radius, n, sigma, winner, stopped, mean_red, mean_blu,
|
||||||
|
mean_diff, sd_diff, se_diff, red_wins, blu_wins, ties, weapon_kills_user,
|
||||||
|
food_amount`
|
||||||
|
|
||||||
|
- `n` — seeds actually run for this point
|
||||||
|
- `sigma` — `|mean_diff| / se_diff` achieved
|
||||||
|
- `winner` — `red` / `blu` (sign of `mean_diff`)
|
||||||
|
- `stopped` — `sigma` (decided) or `max_samples` (undecided → `tie` in the plot)
|
||||||
|
|
||||||
|
## Result of the bundled experiment
|
||||||
|
|
||||||
|
`conf.toml` (red aggressive, blue pacifist, `forwardball`, `kills_user = true`),
|
||||||
|
coarse grid `view_dist ∈ [100, 550] × 25`, `radius ∈ [0, 250] × 12.5`, one
|
||||||
|
refinement pass, `sigma = 3`: **714 points, ~650k seeds** (a fixed 2000-seed
|
||||||
|
sweep over the same grid would have been ~1.4M seeds).
|
||||||
|
|
||||||
|
`phase.png` shows a clear transition:
|
||||||
|
|
||||||
|
- **Low `view_dist` (≲ 140):** blue wins almost everywhere. Red starts far from
|
||||||
|
the food and, with little sensing, forages no better than the pacifist while
|
||||||
|
paying for an unused weapon.
|
||||||
|
- **Higher `view_dist`:** red wins above a curved `radius` threshold and blue
|
||||||
|
below it. The boundary is *non-monotonic*: the radius red needs dips to a
|
||||||
|
minimum near `view_dist ≈ 250` and rises again towards 550, because the
|
||||||
|
`forwardball` sits at `view_end` (`view_dist` ahead of the robot), so a large
|
||||||
|
`view_dist` places the blast far ahead and needs a larger radius to connect.
|
||||||
|
- The boundary band (`tie` points) is where `|mean_diff| / SE < 3` even after the
|
||||||
|
sample cap.
|
||||||
|
|
||||||
|
## Extending
|
||||||
|
|
||||||
|
- **Other axes:** point `--view-dist`/`--radius` at any numeric field, or add a
|
||||||
|
third axis. For a third axis, wrap `sweep.py` in a loop over the extra value
|
||||||
|
and use separate output files, then plot a slice montage.
|
||||||
|
- **Other statistics:** the sequential mode currently tracks the food-share
|
||||||
|
difference. Win-rate (`red_wins`/`blu_wins`) is already recorded if you prefer
|
||||||
|
a Bernoulli statistic.
|
||||||
|
- **Finer boundary:** more `--refine` passes and/or a larger `--seed-budget`.
|
||||||
146
tools/plot_phase.py
Normal file
146
tools/plot_phase.py
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Plot the red/blue phase diagram from a `sweep.py` result file.
|
||||||
|
|
||||||
|
Reads the JSONL produced by `tools/sweep.py` and renders:
|
||||||
|
|
||||||
|
1. the winning side per (view_dist, radius) point, with the estimated phase
|
||||||
|
boundary (mean RED-BLU food-share difference == 0) overlaid, and
|
||||||
|
2. the continuous mean-difference field with a diverging colour map.
|
||||||
|
|
||||||
|
Markers are sized by the number of seeds the sequential test needed, so it is
|
||||||
|
easy to see where the search had to work hardest.
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
python3 tools/plot_phase.py sweep_out/results.jsonl -o sweep_out/phase.png
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import matplotlib
|
||||||
|
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
import numpy as np
|
||||||
|
from matplotlib.colors import TwoSlopeNorm
|
||||||
|
from matplotlib.lines import Line2D
|
||||||
|
|
||||||
|
RED = "#d62728"
|
||||||
|
BLU = "#1f77b4"
|
||||||
|
TIE = "#9e9e9e"
|
||||||
|
|
||||||
|
|
||||||
|
def load(path: Path) -> dict:
|
||||||
|
if path.suffix == ".csv":
|
||||||
|
import csv
|
||||||
|
with path.open() as f:
|
||||||
|
rows = list(csv.DictReader(f))
|
||||||
|
else:
|
||||||
|
rows = []
|
||||||
|
with path.open() as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
rows.append(json.loads(line))
|
||||||
|
cols = {
|
||||||
|
"vd": np.array([float(r["view_dist"]) for r in rows]),
|
||||||
|
"rad": np.array([float(r["weapon_radius"]) for r in rows]),
|
||||||
|
"diff": np.array([float(r["mean_diff"]) for r in rows]),
|
||||||
|
"sigma": np.array([float(r["sigma"]) for r in rows]),
|
||||||
|
"n": np.array([float(r["n"]) for r in rows]),
|
||||||
|
"winner": [r["winner"] for r in rows],
|
||||||
|
"stopped": [r["stopped"] for r in rows],
|
||||||
|
}
|
||||||
|
return cols
|
||||||
|
|
||||||
|
|
||||||
|
def outcome(winner: str, stopped: str) -> str:
|
||||||
|
return "tie" if stopped != "sigma" else winner
|
||||||
|
|
||||||
|
|
||||||
|
def marker_sizes(n: np.ndarray, lo: float = 18.0, hi: float = 260.0) -> np.ndarray:
|
||||||
|
if n.max() <= n.min():
|
||||||
|
return np.full_like(n, (lo + hi) / 2)
|
||||||
|
scaled = (n - n.min()) / (n.max() - n.min())
|
||||||
|
return lo + scaled * (hi - lo)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("results", type=Path, help="results.jsonl or results.csv")
|
||||||
|
ap.add_argument("-o", "--out", type=Path, default=Path("sweep_out/phase.png"))
|
||||||
|
ap.add_argument("--title", default="Adversarial foraging: red (aggressive) vs blue (pacifist)")
|
||||||
|
ap.add_argument("--dpi", type=int, default=180)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
d = load(args.results)
|
||||||
|
vd, rad, diff, sigma, n = d["vd"], d["rad"], d["diff"], d["sigma"], d["n"]
|
||||||
|
labels = [outcome(w, s) for w, s in zip(d["winner"], d["stopped"])]
|
||||||
|
color_of = {"red": RED, "blu": BLU, "tie": TIE}
|
||||||
|
colors = [color_of[l] for l in labels]
|
||||||
|
sizes = marker_sizes(n)
|
||||||
|
|
||||||
|
# Regular grid for a smooth background field, if the points are lattice-like.
|
||||||
|
fig, axes = plt.subplots(1, 2, figsize=(15, 6.2), sharex=True, sharey=True)
|
||||||
|
|
||||||
|
# ---- Panel 1: discrete winner regions -------------------------------- #
|
||||||
|
ax = axes[0]
|
||||||
|
ax.scatter(vd, rad, s=sizes, c=colors, marker="s", edgecolors="black",
|
||||||
|
linewidths=0.6, alpha=0.9, zorder=3)
|
||||||
|
# Boundary contour from the continuous field.
|
||||||
|
try:
|
||||||
|
ax.tricontour(vd, rad, diff, levels=[0.0], colors="black",
|
||||||
|
linewidths=2.0, zorder=4)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
ax.set_title("Winner (marker size ∝ seeds needed)")
|
||||||
|
ax.set_xlabel("view_dist")
|
||||||
|
ax.set_ylabel("weapon radius")
|
||||||
|
ax.set_axisbelow(True)
|
||||||
|
ax.grid(alpha=0.25)
|
||||||
|
handles = [
|
||||||
|
Line2D([], [], marker="s", linestyle="", color=RED, markersize=11, label="red wins"),
|
||||||
|
Line2D([], [], marker="s", linestyle="", color=BLU, markersize=11, label="blue wins"),
|
||||||
|
Line2D([], [], marker="s", linestyle="", color=TIE, markersize=11, label="undecided (hit cap)"),
|
||||||
|
Line2D([], [], color="black", linewidth=2, label="boundary: mean diff = 0"),
|
||||||
|
]
|
||||||
|
ax.legend(handles=handles, loc="best", framealpha=0.9)
|
||||||
|
|
||||||
|
# ---- Panel 2: continuous mean-difference field ----------------------- #
|
||||||
|
ax = axes[1]
|
||||||
|
vmax = max(abs(diff).max(), 1e-6)
|
||||||
|
norm = TwoSlopeNorm(vmin=-vmax, vcenter=0.0, vmax=vmax)
|
||||||
|
sc = ax.scatter(vd, rad, s=sizes, c=diff, cmap="coolwarm", norm=norm,
|
||||||
|
marker="s", edgecolors="black", linewidths=0.6, zorder=3)
|
||||||
|
try:
|
||||||
|
cs = ax.tricontour(vd, rad, diff, levels=[0.0], colors="black",
|
||||||
|
linewidths=2.0, zorder=4)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# A faint sigma=3 contour shows where confidence is marginal vs decisive.
|
||||||
|
try:
|
||||||
|
ax.tricontour(vd, rad, sigma, levels=[3.0], colors="black",
|
||||||
|
linestyles="dotted", linewidths=1.2, zorder=4)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
cb = fig.colorbar(sc, ax=ax, fraction=0.046, pad=0.04)
|
||||||
|
cb.set_label("mean RED-BLU food-share difference")
|
||||||
|
ax.set_title("Mean difference (dotted: |diff|/SE = 3)")
|
||||||
|
ax.set_xlabel("view_dist")
|
||||||
|
ax.set_axisbelow(True)
|
||||||
|
ax.grid(alpha=0.25)
|
||||||
|
|
||||||
|
fig.suptitle(args.title, fontsize=13)
|
||||||
|
fig.tight_layout(rect=(0, 0, 1, 0.96))
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fig.savefig(args.out, dpi=args.dpi)
|
||||||
|
print(f"wrote {args.out}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
394
tools/sweep.py
Normal file
394
tools/sweep.py
Normal file
|
|
@ -0,0 +1,394 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Adaptive grid search over the simulator's `conf.toml` space.
|
||||||
|
|
||||||
|
For every point on a (view_dist x weapon-radius) grid this driver asks the
|
||||||
|
simulator for a *sequential* evaluation: seeds are run in batches and the run
|
||||||
|
stops as soon as the RED-BLU food-share difference is `--sigma` standard
|
||||||
|
errors away from zero, or `--max-samples` is hit. The per-point statistics are
|
||||||
|
appended to a JSONL file (resumable) and a flat CSV.
|
||||||
|
|
||||||
|
The grid is *adaptive*: after the coarse pass, cells whose four corners do not
|
||||||
|
agree on a winner are refined (centre + edge midpoints) with a stricter sigma,
|
||||||
|
so extra samples are spent where the phase boundary actually is.
|
||||||
|
|
||||||
|
Only one thing about the base config is required: red aggressive, blue
|
||||||
|
pacifist. `conf.toml` already satisfies that.
|
||||||
|
|
||||||
|
Examples
|
||||||
|
--------
|
||||||
|
# coarse pass only
|
||||||
|
python3 tools/sweep.py --view-dist 100:550:25 --radius 0:250:12.5
|
||||||
|
|
||||||
|
# coarse + two refinement passes
|
||||||
|
python3 tools/sweep.py --view-dist 100:550:50 --radius 0:250:25 --refine 2
|
||||||
|
|
||||||
|
# see the grid without running anything
|
||||||
|
python3 tools/sweep.py --dry-run
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parent.parent
|
||||||
|
DEFAULT_BINARY = REPO / "simulator" / "target" / "release" / "simulator"
|
||||||
|
DEFAULT_BASE = REPO / "simulator" / "conf.toml"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Grid specification
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def parse_axis(spec: str) -> list[float]:
|
||||||
|
"""Parse an axis spec into a sorted list of unique floats.
|
||||||
|
|
||||||
|
Accepts either a comma/space separated list (`0,10,20`) or a
|
||||||
|
`start:stop:step` range (`0:250:12.5`, inclusive of `stop` when it lands).
|
||||||
|
"""
|
||||||
|
spec = spec.strip()
|
||||||
|
if ":" in spec and "," not in spec:
|
||||||
|
parts = [p.strip() for p in spec.split(":")]
|
||||||
|
if len(parts) != 3:
|
||||||
|
raise ValueError(f"range must be start:stop:step, got {spec!r}")
|
||||||
|
start, stop, step = (float(p) for p in parts)
|
||||||
|
if step == 0:
|
||||||
|
raise ValueError("range step must be non-zero")
|
||||||
|
out: list[float] = []
|
||||||
|
v = start
|
||||||
|
# Tolerate float drift when deciding whether `stop` is reached.
|
||||||
|
eps = abs(step) * 1e-9
|
||||||
|
if step > 0:
|
||||||
|
while v <= stop + eps:
|
||||||
|
out.append(round(v, 9))
|
||||||
|
v += step
|
||||||
|
else:
|
||||||
|
while v >= stop - eps:
|
||||||
|
out.append(round(v, 9))
|
||||||
|
v += step
|
||||||
|
return out
|
||||||
|
return sorted({round(float(p), 9) for p in re.split(r"[,\s]+", spec) if p})
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Config templating
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def render_config(base_text: str, overrides: dict[str, float | int | bool]) -> str:
|
||||||
|
"""Replace `key = value` lines in the TOML text for each override."""
|
||||||
|
text = base_text
|
||||||
|
for key, value in overrides.items():
|
||||||
|
if isinstance(value, bool):
|
||||||
|
rendered = "true" if value else "false"
|
||||||
|
elif isinstance(value, int):
|
||||||
|
rendered = str(value)
|
||||||
|
else:
|
||||||
|
rendered = repr(float(value))
|
||||||
|
pattern = re.compile(rf"^(\s*){re.escape(key)}(\s*)=.*$", re.M)
|
||||||
|
text, n = pattern.subn(rf"\g<1>{key}\g<2>= {rendered}", text)
|
||||||
|
if n == 0:
|
||||||
|
raise ValueError(f"key {key!r} not found in base config")
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def parse_value(raw: str) -> float | int | bool:
|
||||||
|
"""Parse a `--set KEY=VALUE` value into a bool/int/float."""
|
||||||
|
low = raw.strip().lower()
|
||||||
|
if low in ("true", "false"):
|
||||||
|
return low == "true"
|
||||||
|
try:
|
||||||
|
return int(raw)
|
||||||
|
except ValueError:
|
||||||
|
return float(raw)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_set(items: list[str]) -> dict[str, float | int | bool]:
|
||||||
|
out: dict[str, float | int | bool] = {}
|
||||||
|
for item in items or []:
|
||||||
|
if "=" not in item:
|
||||||
|
raise ValueError(f"--set expects KEY=VALUE, got {item!r}")
|
||||||
|
key, _, raw = item.partition("=")
|
||||||
|
out[key.strip()] = parse_value(raw)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Sequential evaluation of a single grid point
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
class Evaluator:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
binary: Path,
|
||||||
|
base_text: str,
|
||||||
|
rayon_threads: int,
|
||||||
|
timeout: float,
|
||||||
|
extra: dict[str, float | int | bool] | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.binary = binary
|
||||||
|
self.base_text = base_text
|
||||||
|
self.rayon_threads = rayon_threads
|
||||||
|
self.timeout = timeout
|
||||||
|
self.extra = extra or {}
|
||||||
|
|
||||||
|
def __call__(self, point: tuple[float, float, dict]) -> dict:
|
||||||
|
view_dist, radius, opts = point
|
||||||
|
overrides = {"view_dist": view_dist, "radius": radius, **self.extra}
|
||||||
|
text = render_config(self.base_text, overrides)
|
||||||
|
# Unique temp file per call so concurrent threads never race.
|
||||||
|
fd, path = tempfile.mkstemp(suffix=".toml", dir=tempfile.gettempdir())
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w") as f:
|
||||||
|
f.write(text)
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["RAYON_NUM_THREADS"] = str(self.rayon_threads)
|
||||||
|
cmd = [
|
||||||
|
str(self.binary),
|
||||||
|
"--sequential",
|
||||||
|
"--config", path,
|
||||||
|
"--sigma", str(opts["sigma"]),
|
||||||
|
"--min-samples", str(opts["min_samples"]),
|
||||||
|
"--max-samples", str(opts["max_samples"]),
|
||||||
|
"--batch", str(opts["batch"]),
|
||||||
|
]
|
||||||
|
proc = subprocess.run(
|
||||||
|
cmd, capture_output=True, text=True, env=env, timeout=self.timeout
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(path)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"simulator failed for view_dist={view_dist} radius={radius}: "
|
||||||
|
f"{proc.stderr.strip() or proc.stdout.strip()}"
|
||||||
|
)
|
||||||
|
|
||||||
|
line = next(
|
||||||
|
(ln for ln in reversed(proc.stdout.splitlines()) if ln.strip().startswith("{")),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if line is None:
|
||||||
|
raise RuntimeError(f"no JSON in simulator output:\n{proc.stdout}")
|
||||||
|
result = json.loads(line)
|
||||||
|
# Normalise to the grid coordinates we asked for (JSON round-trip of
|
||||||
|
# the floats should match, but this makes the key exact).
|
||||||
|
result["view_dist"] = view_dist
|
||||||
|
result["weapon_radius"] = radius
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Persistence
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def result_key(r: dict) -> tuple[float, float]:
|
||||||
|
return (round(float(r["view_dist"]), 9), round(float(r["weapon_radius"]), 9))
|
||||||
|
|
||||||
|
|
||||||
|
def load_results(path: Path) -> dict[tuple[float, float], dict]:
|
||||||
|
results: dict[tuple[float, float], dict] = {}
|
||||||
|
if path.exists():
|
||||||
|
with path.open() as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
r = json.loads(line)
|
||||||
|
results[result_key(r)] = r
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def write_csv(path: Path, results: dict[tuple[float, float], dict]) -> None:
|
||||||
|
if not results:
|
||||||
|
return
|
||||||
|
cols = [
|
||||||
|
"view_dist", "weapon_radius", "n", "sigma", "winner", "stopped",
|
||||||
|
"mean_red", "mean_blu", "mean_diff", "sd_diff", "se_diff",
|
||||||
|
"red_wins", "blu_wins", "ties", "weapon_kills_user", "food_amount",
|
||||||
|
]
|
||||||
|
rows = sorted(results.values(), key=result_key)
|
||||||
|
with path.open("w", newline="") as f:
|
||||||
|
import csv
|
||||||
|
w = csv.DictWriter(f, fieldnames=cols, extrasaction="ignore")
|
||||||
|
w.writeheader()
|
||||||
|
for r in rows:
|
||||||
|
w.writerow(r)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Adaptive refinement
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def outcome(r: dict) -> str:
|
||||||
|
"""Collapse a result to the label used for refinement decisions.
|
||||||
|
|
||||||
|
A point that hit the sample cap without reaching sigma is reported as
|
||||||
|
`tie`, so the boundary around genuinely undecided regions is refined too.
|
||||||
|
"""
|
||||||
|
return "tie" if r.get("stopped") != "sigma" else r["winner"]
|
||||||
|
|
||||||
|
|
||||||
|
def find_refinement_points(
|
||||||
|
results: dict[tuple[float, float], dict],
|
||||||
|
vds: list[float],
|
||||||
|
rads: list[float],
|
||||||
|
) -> set[tuple[float, float]]:
|
||||||
|
"""Midpoints of cells whose four corners disagree on the outcome."""
|
||||||
|
def label(vd: float, rad: float) -> str | None:
|
||||||
|
r = results.get((round(vd, 9), round(rad, 9)))
|
||||||
|
return outcome(r) if r else None
|
||||||
|
|
||||||
|
new: set[tuple[float, float]] = set()
|
||||||
|
for i in range(len(vds) - 1):
|
||||||
|
for j in range(len(rads) - 1):
|
||||||
|
corners = [
|
||||||
|
label(vds[i], rads[j]),
|
||||||
|
label(vds[i + 1], rads[j]),
|
||||||
|
label(vds[i], rads[j + 1]),
|
||||||
|
label(vds[i + 1], rads[j + 1]),
|
||||||
|
]
|
||||||
|
if None in corners or len(set(corners)) == 1:
|
||||||
|
continue
|
||||||
|
mid_vd = round((vds[i] + vds[i + 1]) / 2, 9)
|
||||||
|
mid_rad = round((rads[j] + rads[j + 1]) / 2, 9)
|
||||||
|
new.update({
|
||||||
|
(mid_vd, rads[j]),
|
||||||
|
(mid_vd, rads[j + 1]),
|
||||||
|
(vds[i], mid_rad),
|
||||||
|
(vds[i + 1], mid_rad),
|
||||||
|
(mid_vd, mid_rad),
|
||||||
|
})
|
||||||
|
return new
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Main
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--binary", type=Path, default=DEFAULT_BINARY)
|
||||||
|
ap.add_argument("--base", type=Path, default=DEFAULT_BASE,
|
||||||
|
help="base config; red must be aggressive and blue pacifist")
|
||||||
|
ap.add_argument("--view-dist", default="100:550:25", help="axis spec (default 100:550:25)")
|
||||||
|
ap.add_argument("--radius", default="0:250:12.5", help="axis spec (default 0:250:12.5)")
|
||||||
|
ap.add_argument("--out", type=Path, default=Path("sweep_out/results.jsonl"))
|
||||||
|
ap.add_argument("--sigma", type=float, default=3.0)
|
||||||
|
ap.add_argument("--min-samples", type=int, default=32)
|
||||||
|
ap.add_argument("--max-samples", type=int, default=2000)
|
||||||
|
ap.add_argument("--batch", type=int, default=64)
|
||||||
|
ap.add_argument("--refine", type=int, default=1, help="number of adaptive refinement passes")
|
||||||
|
ap.add_argument("--refine-sigma", type=float, default=3.0)
|
||||||
|
ap.add_argument("--refine-max-samples", type=int, default=2500)
|
||||||
|
ap.add_argument("--refine-batch", type=int, default=128)
|
||||||
|
ap.add_argument("--seed-budget", type=int, default=4_000_000,
|
||||||
|
help="stop scheduling work once this many seeds have run (safety valve)")
|
||||||
|
ap.add_argument("--workers", type=int, default=os.cpu_count() or 4)
|
||||||
|
ap.add_argument("--rayon-threads", type=int, default=1,
|
||||||
|
help="threads the simulator may use per point (keep at 1: we parallelise over points)")
|
||||||
|
ap.add_argument("--timeout", type=float, default=600.0)
|
||||||
|
ap.add_argument("--set", action="append", default=[], metavar="KEY=VALUE",
|
||||||
|
help="pin any other conf.toml field (repeatable), e.g. --set kills_user=false")
|
||||||
|
ap.add_argument("--dry-run", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if not args.binary.exists():
|
||||||
|
ap.error(f"binary not found: {args.binary} (build with `cargo build --release`)")
|
||||||
|
base_text = args.base.read_text()
|
||||||
|
|
||||||
|
vds = parse_axis(args.view_dist)
|
||||||
|
rads = parse_axis(args.radius)
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
csv_path = args.out.with_suffix(".csv")
|
||||||
|
|
||||||
|
results = load_results(args.out)
|
||||||
|
if results:
|
||||||
|
print(f"resuming: {len(results)} points already in {args.out}", file=sys.stderr)
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
print(f"view_dist ({len(vds)}): {vds}")
|
||||||
|
print(f"radius ({len(rads)}): {rads}")
|
||||||
|
print(f"coarse grid: {len(vds) * len(rads)} points; "
|
||||||
|
f"{args.refine} refinement pass(es); out={args.out}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
ev = Evaluator(args.binary, base_text, args.rayon_threads, args.timeout,
|
||||||
|
extra=parse_set(args.set))
|
||||||
|
base_opts = dict(sigma=args.sigma, min_samples=args.min_samples,
|
||||||
|
max_samples=args.max_samples, batch=args.batch)
|
||||||
|
|
||||||
|
out_lock = threading.Lock()
|
||||||
|
out_fh = args.out.open("a")
|
||||||
|
state = {"seeds": sum(int(r["n"]) for r in results.values())}
|
||||||
|
if state["seeds"]:
|
||||||
|
print(f"resuming: {state['seeds']} seeds already spent", file=sys.stderr)
|
||||||
|
|
||||||
|
def evaluate(points: list[tuple[float, float]], opts: dict, label: str) -> bool:
|
||||||
|
"""Evaluate `points`, appending results as they finish.
|
||||||
|
|
||||||
|
Returns False if the seed budget ran out (caller should stop refining).
|
||||||
|
"""
|
||||||
|
todo = [
|
||||||
|
(vd, rad, opts) for vd, rad in points
|
||||||
|
if (round(vd, 9), round(rad, 9)) not in results
|
||||||
|
]
|
||||||
|
if not todo:
|
||||||
|
print(f"[{label}] nothing new to evaluate", flush=True)
|
||||||
|
return True
|
||||||
|
print(f"[{label}] evaluating {len(todo)} point(s) "
|
||||||
|
f"(sigma={opts['sigma']}, max={opts['max_samples']}, "
|
||||||
|
f"seeds={state['seeds']}/{args.seed_budget})", flush=True)
|
||||||
|
chunk = max(args.workers * 2, 8)
|
||||||
|
done = 0
|
||||||
|
with ThreadPoolExecutor(max_workers=args.workers) as ex:
|
||||||
|
for i in range(0, len(todo), chunk):
|
||||||
|
if state["seeds"] >= args.seed_budget:
|
||||||
|
print(f"[{label}] seed budget exhausted; stopping", flush=True)
|
||||||
|
return False
|
||||||
|
for r in ex.map(ev, todo[i:i + chunk]):
|
||||||
|
key = result_key(r)
|
||||||
|
with out_lock:
|
||||||
|
results[key] = r
|
||||||
|
out_fh.write(json.dumps(r) + "\n")
|
||||||
|
out_fh.flush()
|
||||||
|
state["seeds"] += int(r["n"])
|
||||||
|
done += 1
|
||||||
|
print(f" [{done}/{len(todo)}] vd={r['view_dist']:>6} "
|
||||||
|
f"rad={r['weapon_radius']:>6} n={r['n']:>5} "
|
||||||
|
f"sigma={r['sigma']:.2f} -> {outcome(r)} ({r['stopped']})",
|
||||||
|
flush=True)
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Coarse pass.
|
||||||
|
evaluate([(vd, rad) for vd in vds for rad in rads], base_opts, "coarse")
|
||||||
|
|
||||||
|
# Adaptive refinement passes.
|
||||||
|
for p in range(args.refine):
|
||||||
|
vds = sorted({k[0] for k in results})
|
||||||
|
rads = sorted({k[1] for k in results})
|
||||||
|
new = find_refinement_points(results, vds, rads)
|
||||||
|
if not new:
|
||||||
|
print("no boundary cells left to refine")
|
||||||
|
break
|
||||||
|
opts = dict(sigma=args.refine_sigma, min_samples=args.min_samples,
|
||||||
|
max_samples=args.refine_max_samples, batch=args.refine_batch)
|
||||||
|
if not evaluate(sorted(new), opts, f"refine {p + 1}"):
|
||||||
|
break
|
||||||
|
|
||||||
|
out_fh.close()
|
||||||
|
write_csv(csv_path, results)
|
||||||
|
|
||||||
|
red = sum(1 for r in results.values() if outcome(r) == "red")
|
||||||
|
blu = sum(1 for r in results.values() if outcome(r) == "blu")
|
||||||
|
tie = sum(1 for r in results.values() if outcome(r) == "tie")
|
||||||
|
print(f"\ndone: {len(results)} points -> red={red} blu={blu} tie={tie}")
|
||||||
|
print(f"wrote {args.out} and {csv_path}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Loading…
Reference in a new issue