89 lines
4.3 KiB
Markdown
89 lines
4.3 KiB
Markdown
# Event E2E suite (maintainable)
|
||
|
||
Goal: every catalog event is covered by automation that stays cheap when new ids are added.
|
||
|
||
Live accuracy roadmap: [`event-live-verification-plan.md`](event-live-verification-plan.md).
|
||
|
||
Observe → confirm → emit: [`event-observation.md`](event-observation.md).
|
||
|
||
## Tracks
|
||
|
||
| Track | Scenario | What it proves | How new events join |
|
||
|-------|----------|----------------|---------------------|
|
||
| **Inventory audits** | nested in `event_coverage` | Live library vs authored catalog | Add JSON row / live seed; audit fails until matched |
|
||
| **A inject coverage** | `event_inject_coverage` / `ec30` | Every **camera-worthy** id can register an interest key + label | Auto via `EventCatalog.*.AuthoredIds` |
|
||
| **Live apply loop** | `event_live_apply_loop` | Happiness Signal + status gains via real apply | Auto loop; ledger `id` / `id_drop` / `fail` |
|
||
| **Live relationship** | `event_live_relationship` | Vanilla setters/Beh + join negative; leave/`family_removed`/baby live fires | Shared runner + ledger claims |
|
||
| **Live pipelines** | `event_live_pipelines` | Traits/decisions/WorldLog/wars/eras/books/plots + libraries (worldLaw/item/subspecies_trait/power/gene/phenotype/meta traits); spell/biome best-effort unsupported; boat/combat best-effort | Auto loops + honest gaps |
|
||
| **Live outcome smoke** | `event_live_outcome_smoke` | Short family + sample apply | Curated |
|
||
| **Live coverage ledger** | `event_live_coverage` | Honest `none|feed|pipeline|id_drop|id|…` TSV | Seeded from catalogs; claimed by live runners |
|
||
|
||
## Suite split (weight-aware)
|
||
|
||
| Gate | Includes | When |
|
||
|------|----------|------|
|
||
| `event_suite` | audits + inject + apply loop + relationship + outcome smoke | Day-to-day event gate |
|
||
| `regression` | `event_suite` + `event_live_pipelines` (+ other product smokes) | Full gate |
|
||
| `critical_smoke` ×3 | Spectator/settings/tip/exit | PR gate |
|
||
|
||
`event_live_pipelines` stays **regression-only**.
|
||
|
||
Flake budget is proven (`--repeat 3` PASS); it remains heavier than day-to-day.
|
||
|
||
## Coverage levels
|
||
|
||
Recorded in `.harness/event-live-coverage.tsv` (gitignored; regenerated each live run).
|
||
|
||
| Level | Meaning | Verified? |
|
||
|-------|---------|-----------|
|
||
| `none` | Inject only; no live fire this run | No |
|
||
| `feed` | Production feed entry with synthetic payload | Partial |
|
||
| `pipeline` | Vanilla fire proved the shared hook family; this id may not have fired | Partial |
|
||
| `id_drop` | Real fire; authored drop matched | Yes (accurate drop) |
|
||
| `id` | Real fire + expected note/interest | Yes |
|
||
| `unsupported` | No deterministic fire API; reason in `notes` | Honest gap |
|
||
| `fail` | Unexpected miss/drop | Broken |
|
||
|
||
Inject / `domain_feed` never writes a live level other than leaving `none`.
|
||
|
||
## Busy policy
|
||
|
||
Most feeds early-out on `AgentHarness.Busy`.
|
||
|
||
Live runners set `ForceLiveEventFeeds` (relationship also sets `ForceRelationshipFeeds`).
|
||
|
||
`LiveFeedsAllowed` = `!Busy || ForceLiveEventFeeds || ForceRelationshipFeeds`.
|
||
|
||
Ledger `notes` should record `busy_bypass=ForceLiveEventFeeds` (or the specific flag) when used.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
./scripts/harness-run.sh event_inject_coverage
|
||
./scripts/harness-run.sh event_live_apply_loop
|
||
./scripts/harness-run.sh event_live_relationship
|
||
./scripts/harness-run.sh event_live_pipelines
|
||
./scripts/harness-run.sh event_suite
|
||
./scripts/harness-run.sh regression
|
||
```
|
||
|
||
Artifacts (under `IdleSpectator/.harness/`, gitignored):
|
||
|
||
- `event-inject-coverage.tsv` - inject wiring
|
||
- `event-live-coverage.tsv` - **source of truth for live proof**
|
||
- `event-trigger-audit.tsv` - design / call-site audit (not live proof)
|
||
|
||
## Design rules
|
||
|
||
1. Loop `AuthoredIds` - no per-id harness steps for inject/apply/traits.
|
||
2. Inject ≠ live. `domain_feed` is never live.
|
||
3. `ForceLiveEventFeeds` / `LiveFeedsAllowed` bypass Busy during live runners only.
|
||
4. Coverage levels must not over-claim (see plan).
|
||
5. Registry max-96: remove/clear during mass live loops so trim does not false-fail.
|
||
6. Assert fails on `fail > 0`; do not fail on camera `none` until a budget is explicitly tightened.
|
||
|
||
## Code
|
||
|
||
- `EventInjectCoverageHarness`, `EventLiveApplyLoopHarness`, `EventLiveRelationshipHarness`, `EventLivePipelinesHarness`
|
||
- `EventLiveCoverageLedger`
|
||
- Asserts: `event_inject_coverage`, `event_live_apply_loop`, `event_live_relationship`, `event_live_pipelines`, `event_live_coverage`
|