41 lines
1.9 KiB
Markdown
41 lines
1.9 KiB
Markdown
# Event E2E suite (maintainable)
|
|
|
|
Goal: every catalog event is covered by automation that stays cheap when new ids are added.
|
|
|
|
## Tracks
|
|
|
|
| Track | Scenario | What it proves | How new events join |
|
|
|-------|----------|----------------|---------------------|
|
|
| **Inventory audits** | nested in `event_coverage` | Live library vs authored catalog | Add JSON row / live seed; audit fails until matched |
|
|
| **A inject coverage** | `event_inject_coverage` / `ec30` | Every **camera-worthy** id can register an interest key + label | Auto via `EventCatalog.*.AuthoredIds` |
|
|
| **Live outcome** | `event_live_outcome_smoke` | Real API/Beh + `EventOutcome` confirm (curated) | Add a deterministic harness step when a patch exists |
|
|
|
|
Full suite: `event_suite` = `event_coverage` + `event_live_outcome_smoke`.
|
|
Regression nests `event_suite`.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
./scripts/harness-run.sh event_inject_coverage
|
|
./scripts/harness-run.sh event_live_outcome_smoke
|
|
./scripts/harness-run.sh event_suite
|
|
./scripts/harness-run.sh regression
|
|
```
|
|
|
|
Artifact: `IdleSpectator/.harness/event-inject-coverage.tsv`
|
|
(`domain`, `id`, `camera`, `result`, `key`, `notes`).
|
|
|
|
## Design rules
|
|
|
|
1. **Do not** hand-write one harness step per catalog id for inject - loop `AuthoredIds`.
|
|
2. B-only rows (`CreatesInterest=false`, Ambient happiness) are `skip_b`, not failures.
|
|
3. Inject coverage ≠ live outcome. Inject proves wiring; live proves confirm predicates.
|
|
4. Expand live outcome slowly (family, status apply, happiness apply, …) as wrappers exist.
|
|
5. Libraries: only camera-worthy live-seeded ids are injected (can be large; Signal-gated).
|
|
6. After each successful inject, remove the key from `InterestRegistry`. The registry caps at 96 candidates; leaving them in causes false `register_failed` on later ids.
|
|
|
|
## Code
|
|
|
|
- `EventInjectCoverageHarness.Run` - data-driven register loop
|
|
- Assert expect `event_inject_coverage` in `AgentHarness`
|
|
- Observe→confirm docs: `event-observation.md`
|