comments: the scripts say what they do and how to run them

Each header is a one-line summary, the environment it reads and its
usage; the history, measurements and record numbers go. Only comment
lines change: with comments stripped the scripts are byte-identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sam
2026-09-23 19:23:35 +07:00
co-authored by Claude Opus 5.5
parent 449324b7be
commit b484fe0d27
10 changed files with 123 additions and 375 deletions
+11 -36
View File
@@ -1,21 +1,15 @@
#!/usr/bin/env bash
# Phase-5 visual A/B render harness driver (decisions 0008 + 0010).
# Visual A/B harness: capture deterministic screenshots and diff them against a baseline.
#
# Captures deterministic benilla screenshots (one process per scenario, server-less + clutter-off +
# pinned camera, game clock AND frame step — see the `capture` module) and diffs them with
# `benilla-visual`. `selfcheck` is the proof that "deterministic" is true. The intended
# loop: capture `baseline` on the current pipeline, make a render change, then `diff` — the diff is the
# machine check that catches a regression before the director's eye (decision 0008).
# One client process per scenario: server-less, clutter off, a pinned camera, a fixed game clock and
# frame step (the `capture` module); `benilla-visual` diffs the PNGs. Opens a real window. Captures
# hold Blizzard-derived imagery, so they stay under the gitignored target/visual/.
#
# Captures contain Blizzard-derived imagery, so they live under the gitignored target/visual/ and are
# NEVER committed. macOS/local only: capture opens a real window (no headless GPU path here).
#
# Usage:
# scripts/visual.sh list # print scenario names
# scripts/visual.sh capture <dir> # capture every scenario into <dir>/
# scripts/visual.sh baseline # capture into target/visual/baseline/
# scripts/visual.sh diff [--fail <mae>] # capture into target/visual/candidate/, diff vs baseline/
# scripts/visual.sh selfcheck # capture the sweep TWICE from ONE build; demand identical
# scripts/visual.sh diff [--fail <mae>] # capture target/visual/candidate/, diff vs baseline/
# scripts/visual.sh selfcheck [<mae>] # capture twice from one build; demand identical
set -euo pipefail
cd "$(dirname "$0")/.."
@@ -23,8 +17,7 @@ VIS=target/visual
build() { cargo build -q -p benilla -p benilla-visual; }
# The scenario list is owned by the binary (capture/mod.rs), printed via WOW_CAPTURE=list — so this driver
# never drifts from the code.
# The binary owns the scenario list (`capture/mod.rs`) and prints it under WOW_CAPTURE=list.
scenarios() { WOW_CAPTURE=list cargo run -q -p benilla; }
capture_into() {
@@ -60,32 +53,14 @@ case "$cmd" in
./target/debug/benilla-visual diff-dir "$VIS/baseline" "$VIS/candidate" --out "$VIS/diff" "$@"
;;
selfcheck) # selfcheck [<mae>] — the tolerance defaults to 0 (bit-identical)
# The harness's own tripwire (decision 0723). A golden diff is evidence ONLY if the harness is
# deterministic, so measure that directly: capture the whole sweep twice from one unchanged
# build and demand a bit-identical result (--fail 0 passes only at MAE exactly 0). Before 0723
# this failed at MAE 0.009 / max delta 180 — the same band a real render change lands in, which
# is how 0721's flame pixels ended up unreadable and how 0719 read signal out of churn. Run it
# after touching anything the capture clock feeds, and whenever a diff looks like noise.
# A diff is evidence only if capture is deterministic: two sweeps from one build must match bit
# for bit. Run it after touching anything the capture clock feeds.
build
capture_into "$VIS/self-a"
capture_into "$VIS/self-b"
echo "--- selfcheck: self-a vs self-b, one build, two runs ---"
# The bar is 0 — bit-identical — and the four blessed scenarios meet it (0817 set the bar, 1183
# cut the sweep to two director-framed spots x two day times). Scenarios that could not hold it
# left the sweep rather than being carried as slack in the bar (the old 0.001 was cover for two
# overlook cells at 421 px / max delta 43).
#
# What makes a cell flake is a real renderer defect, not harness drift: draw order follows spawn
# order, and spawn order varies because asset loads complete on a thread pool (0723 "Open, named"
# -> 0815 Open -> 1182, which measured it). Both states are perfectly STABLE, so no amount of
# settling or waiting fixes it. **Do not respond to a failure here by widening the bar or the
# waits** — that mistake has cost several sessions. Anything above 0 is either that defect
# surfacing in a blessed cell or a genuine harness regression; pass a number to probe.
#
# 1182 is also the correction to the obvious first guess: the tie is NOT always a coplanar pair.
# In the canal cells (deleted by 1183) it was two surfaces INTERSECTING, 0.62 yd apart, tying at
# one MSAA sample — invisible with WOW_MSAA=off. Run WOW_PICK through the differing pixels and
# read the PERPENDICULAR gap before assuming which geometry you are chasing.
# Keep the bar at 0 and add no waits: a nonzero result is a harness regression or a renderer
# defect, such as draw order following spawn order, which varies with thread-pool asset loads.
./target/debug/benilla-visual diff-dir "$VIS/self-a" "$VIS/self-b" --out "$VIS/self-diff" \
--fail "${2:-0}"
echo "selfcheck OK — the harness clock is a pure function of the build"