Skill for e2e testing

2026-07-14 15:00:05 -05:00 · 2026-07-14 15:00:05 -05:00 · 7fc95e8a70
commit 7fc95e8a70
parent 08d781a67b
3 changed files with 179 additions and 1 deletions
--- a/.cursor/skills/idle-spectator-e2e/SKILL.md
+++ b/.cursor/skills/idle-spectator-e2e/SKILL.md
@ -0,0 +1,124 @@
+---
+name: idle-spectator-e2e
+description: >-
+  Run IdleSpectator WorldBox E2E tests via the in-mod agent harness
+  (scripts/harness-run.sh, critical_smoke, .harness commands).
+  Use when adding or fixing IdleSpectator features, playtesting the mod,
+  writing harness scenarios, or verifying spectator/focus/tip/settings behavior.
+---
+
+# IdleSpectator E2E harness
+
+Always validate IdleSpectator changes with the in-game harness.
+Do not rely on OS mouse/keyboard automation (Wayland-flaky).
+Do not call `Application.Quit()` (can hang Steam on Linux).
+
+## Mandatory workflow
+
+After any feature or bugfix in `IdleSpectator/`:
+
+1. Bump `IdleSpectator/mod.json` version (NML recompiles on game load).
+2. Extend or add a scenario in `IdleSpectator/HarnessScenarios.cs` when the change is new behavior.
+3. Run:
+   ```bash
+   ./scripts/harness-run.sh --repeat 3 critical_smoke
+   ```
+4. If you added a focused scenario, also run it (example: `./scripts/harness-run.sh tip_match`).
+5. Iterate until **all repeats PASS**. Do not stop after one flaky pass.
+6. Report: scenario name, PASS/FAIL counts, and any new asserts.
+
+For tiny doc-only edits, harness is optional.
+For anything that touches camera, spectator, tips, settings, discovery, or input exit, harness is required.
+
+## Runner
+
+```bash
+# Default: critical_smoke (starts WorldBox if needed)
+./scripts/harness-run.sh
+
+./scripts/harness-run.sh critical_smoke
+./scripts/harness-run.sh --repeat 3 critical_smoke
+./scripts/harness-run.sh tip_match
+./scripts/harness-run.sh smoke
+./scripts/harness-run.sh --no-launch critical_smoke
+./scripts/harness-run.sh --cmd '{"id":"1","action":"snapshot"}'
+```
+
+Env overrides:
+
+- `IDLE_SPECTATOR_MOD` - mod folder (default: repo `IdleSpectator/`)
+- `WORLDBOX_LOG` - Player.log path
+- `HARNESS_TIMEOUT` / `HARNESS_BOOT_TIMEOUT`
+- `WORLDBOX_APP_ID` - Steam app id (default `1206560`)
+
+Paths:
+
+- Mod: `IdleSpectator/`
+- Runner: `scripts/harness-run.sh`
+- Inbox/results: `IdleSpectator/.harness/` (gitignored)
+- Log: `~/.config/unity3d/mkarpenko/WorldBox/Player.log`
+
+The runner:
+
+- Starts Steam/WorldBox when not running
+- Touches `.harness/reset` + rewinds offset so commands are re-read
+- Enables `.harness/auto-dismiss` (closes Welcome Back)
+- Waits for `last-result.json` status `PASS` / `FAIL`
+
+A loaded world is still required.
+`wait_world` + auto-dismiss handle Welcome Back and empty maps.
+
+## Built-in scenarios
+
+| Name | Purpose |
+|------|---------|
+| `critical_smoke` | Default regression: settings gate, idle health, tip match, retarget, manual exit |
+| `smoke` | Short enable/focus health check |
+| `tip_match` | Tip asset must match focused unit |
+
+Prefer `pick_unit` (living map unit) over miracle `spawn` in critical paths.
+Spawned crabs/sheep often die on bad tiles and flake tests.
+
+## Adding coverage for a new fix
+
+1. Add asserts/steps to `critical_smoke` when the bug is core behavior.
+2. Or add a named scenario and run both it and `critical_smoke`.
+3. Use `asset=auto` / `label="New species: {asset}"` after `pick_unit`.
+4. Keep InterestDirector frozen while `AgentHarness.Busy` so ambient does not steal focus mid-test.
+5. `wait_world` must stay front-of-queue on retry (never enqueue behind later steps).
+
+## Command cheat sheet
+
+JSONL lines in `.harness/commands.jsonl` (also used by scenarios):
+
+- `wait_world`, `dismiss_windows`, `wait`
+- `spectator` (`value=on|off`, optional `expect=blocked`)
+- `pick_unit`, `spawn`, `focus`, `watch`, `trigger_interest`
+- `simulate_input` (camera-drag manual exit)
+- `set_setting` (`expect=enabled`, `value=true|false`)
+- `assert` expects: `idle`, `health`, `power_bar`, `no_bad`, `tip_matches_unit`, `unit_asset`, `tip_contains`, `enabled`
+- `snapshot`, `reset_counters`, `scenario`
+
+Machine output:
+
+- `[IdleSpectator][HARNESS]` / `[ASSERT]` / `[SNAP]` in Player.log
+- `IdleSpectator/.harness/last-result.json`
+- `IdleSpectator/.harness/results.jsonl`
+
+## Process safety
+
+- Kill only the real game binary (`.../common/worldbox/worldbox`), never `pkill -f` patterns that match the agent shell.
+- Prefer `./scripts/harness-run.sh` for launch; avoid casual `Application.Quit`.
+- After code changes, restart/relaunch so NML recompiles (version bump helps confirm load).
+
+## Definition of done
+
+A change is done only when:
+
+- Relevant harness scenario(s) PASS
+- For core spectator/focus/tip/settings work: `critical_smoke` passes **3 times** (`--repeat 3`)
+- Failures were fixed by hardening the mod or scenario, not by lowering asserts
+
+## More detail
+
+- Command and scenario notes: [reference.md](reference.md)
--- a/.cursor/skills/idle-spectator-e2e/reference.md
+++ b/.cursor/skills/idle-spectator-e2e/reference.md
@ -0,0 +1,52 @@
+# IdleSpectator harness reference
+
+## Architecture
+
+```text
+scripts/harness-run.sh
+  -> starts WorldBox if needed
+  -> writes IdleSpectator/.harness/commands.jsonl
+  -> touches reset + auto-dismiss
+AgentHarness.Update() (in-game)
+  -> executes commands on Unity main thread
+  -> writes results.jsonl + last-result.json
+  -> logs [HARNESS]/[ASSERT]/[SNAP]
+```
+
+## Why not OS input?
+
+Wayland makes `xdotool`/`ydotool` unreliable.
+Truth lives in engine state (`MoveCamera`, tips, settings), so drive the mod API directly.
+
+## Hard-won failure modes
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Welcome closes, idle never starts | commands.jsonl rewritten same length; offset at EOF | runner writes `offset=0` + `reset` |
+| Steps run before world ready | `wait_world` requeued to back | `RequeueFront` |
+| Tip says crab, focus is angle | watch fell back to nearest unit | require matching asset; use `pick_unit`/`asset=auto` |
+| Spawn then "no unit for asset" | miracle spawns die on bad tiles | prefer `pick_unit` for critical paths |
+| Flaky health/no_bad mid-scenario | ambient retarget while harness busy | `InterestDirector` returns early when `AgentHarness.Busy` |
+| Steam hang after tests | `Application.Quit` | leave game running |
+
+## Useful asserts
+
+- `health` - idle + focus + power bar hidden + `bad==0`
+- `tip_matches_unit` - tip asset equals focus unit asset; if tip starts with `New species:`, label species must match
+- `no_bad` - StateProbe bad counter is zero
+- `enabled` - mod setting `enabled` bool
+
+## Extending scenarios
+
+Edit `IdleSpectator/HarnessScenarios.cs`:
+
+1. Add steps with `Step(id, action, ...)`.
+2. Register name in `Build`.
+3. Run `./scripts/harness-run.sh <name>` then `--repeat 3 critical_smoke`.
+
+Keep critical_smoke deterministic:
+
+- `dismiss_windows` / `wait_world` first
+- `pick_unit` before tip/focus asserts
+- `reset_counters` after spectator+focus before health
+- `simulate_input` last for manual-exit coverage
--- a/.gitignore
+++ b/.gitignore
@ -1,5 +1,7 @@
 # Cursor / local agent config (machine-specific; do not publish)
-.cursor/
+.cursor/*
+!.cursor/skills/
+!.cursor/skills/**/

 # Local one-shot spectator enable flag
 IdleSpectator/.force-on