worldbox-observer-mod/docs/event-live-verification-plan.md

Event live verification plan

Goal: every catalog event is accurate and not bugged in production, with honest coverage tracking. Inject stays; full sim grows by hook family, then loops ids where one apply API covers many.

Do not over-claim. Weaker proof gets a weaker live_level.

What “works” means

Failure mode	Required proof
False positive (fires when claim is false)	Full sim + EventOutcome / drop log (negative case required)
False negative (never fires)	Full sim through vanilla API (not inject / not domain_feed)
Bad prose / wrong subject	Inject + reason; sim when label needs live fields
Catalog drift	Inventory audits + inject
Registry wiring	Inject (event_inject_coverage)
Expected policy drop	Apply ran + authored drop reason matched (id_drop)

Coverage levels

Record per id in .harness/event-live-coverage.tsv (regenerated each run; do not commit).

Level	Meaning	Counts as “verified”?
none	Inject only; no live fire this run	No
feed	Called production feed entry with synthetic payload (skips Harmony / some Busy/world wiring)	Partial - glue inside feed only
pipeline	Vanilla fire proved this id’s shared hook family; this id was not necessarily fired	Partial - siblings may share predicate/API
id_drop	Real apply/fire ran; result was an authored drop (reason matched)	Yes for “accurate drop”
id	This id went through real fire and asserted note/interest as expected	Yes
unsupported	No deterministic fire API; explicit reason in notes	Honest gap
fail	Fire attempted; unexpected miss/drop/wrong key	Broken

Claiming rules (strict)

1. id / id_drop: only the id that was actually fired this run.
2. pipeline: only when (a) vanilla entry was used, and (b) siblings share the same confirm predicate and same fire API. Note must include shared=<pipeline_id>. Never mark an id pipeline merely because another id in the domain passed.
3. feed: never promote to pipeline or id without a vanilla path.
4. domain_feed / inject: never write any live level except leaving none.
5. Ledger is computed from the last run’s results + catalog seed. No hand-maintained allowlist of “known smokes.”

Ambient / CreatesInterest=false: not camera-live goals. Optional B-log loop later; default leave at none unless explicitly asserted as id_drop / ActivityLog-only.

Current baseline (done)

• Inventory audits in event_coverage
• event_inject_coverage - all camera-worthy ids register
• event_live_outcome_smoke - family lovers/parent, one status, one happiness (proof exists; ledger will record from run, not a static list)
• Observe→confirm for relationship Beh paths
• Docs: event-e2e.md, event-observation.md

Phase 0 - Coverage ledger + Busy policy

Deliverable: harness writes .harness/event-live-coverage.tsv every live/coverage run.

Columns: domain, id, camera, inject, live_level, pipeline, notes

• Seed rows from the same AuthoredIds loops as inject (camera / inject from that run or mirror rules)
• live_level defaults to none; overwrite only from live runners’ results
• Suite prints: id=N id_drop=N pipeline=N feed=N none=N unsupported=N fail=N
• Fail the assert on fail > 0 once live runners exist; do not fail on none until Phase 3 exit tightening

Busy bypass (required before Phase 2/3 interest asserts)

Most feeds early-out on AgentHarness.Busy. Define per domain:

Domain	During live fire	Primary assert
Happiness	HappinessSourceHook.Harness / existing apply path	Router notes + log (interest optional)
Status	ActivityLog primary; optional force-interest flag later	Status counters + log
Relationship	ForceRelationshipFeeds (already used for parent)	Interest key + drop log
WorldLog / war / plot / book / trait / decision / …	Explicit Force*Feeds or harness “allow live feeds” mode	Interest key and/or domain log

Document the flag in notes when used (busy_bypass=ForceRelationshipFeeds).

Maintainability: new catalog ids appear as none automatically until a runner claims them.

Phase 1 - Shared apply loops (highest confidence / hour)

Loop camera-worthy ids through existing vanilla-ish apply APIs (happiness_apply, status_apply).

1a Happiness

For each Signal id in EventCatalog.Happiness.AuthoredIds (skip Ambient unless doing an optional B-log pass):

1. Reset notes / clear subject counters
2. happiness_apply with that id
3. Classify:
   • expected note / LastEffectId → id
   • authored drop reason matched → id_drop
   • else → fail
4. pipeline field: happiness_apply

Do not require camera interest while Busy unless a force path is on.

1b Status

For each CreatesInterest status (gains):

1. Clear status + activity status counters
2. status_apply
3. Assert gain + log prose → id; authored skip/drop → id_drop; else fail
4. status_remove / egg hatch as a separate pipeline (status_loss / hatch) with its own ids

Exit: every camera-worthy happiness Signal and status gain id is id, id_drop, or fail (no silent none for those sets). Suite fails on fail.

Phase 2 - Confirm-gated relationship families

One full sim per EventOutcome predicate (and separate setter smokes where there is no confirm helper).

Confirm smokes (predicate + negative case)

Pipeline	Sim	Assert
GainedLover / bond	happiness_bond_lovers	notes + log; exercised id → id
parent / add_child	relationship_set_parent	rel:add_child; id → id
LostLover / clear_lover	bond then clear via game API	emit or authored drop
GainedFamily / family_group_join	join when family exists and when not	key vs outcome_unconfirmed
LostFamily / leave-remove	leave/remove when applicable	key vs unconfirmed
StillSeekingLover / find_lover	seeking Beh path	confirm rules

Negative tests are mandatory for confirm predicates (especially join-with-no-family).

Setter smokes (no shared confirm sibling claim)

new_family, family_group_new, become_alpha, baby_created, etc.: each needs its own vanilla fire for id. Do not mark them pipeline off add_child or join.

Siblings may get pipeline only under Claiming rule 2 (shared=<predicate>).

Exit: every relationship camera id is id, pipeline (with valid shared=), id_drop, unsupported, or fail - never unexplained none.

Phase 3 - Remaining hook families

Prefer vanilla mutation. If only feed-invoke is feasible, record feed and keep improving to vanilla.

Pipeline	Minimum vanilla sim	Id loop?	Also cover
WorldLog	Post/construct real world history message the patch sees	Yes if asset id is the parameter
Disasters	Through paired WorldLog / disaster trigger	Via paired worldLog
War types	WarManager.newWar (or equivalent)	Per type if API takes type
Eras	Force age change	Per era if possible
Plots	setPlot / finish; phases as separate pipelines	Per plot id if API takes id
Books	Create/burn vanilla path	Per book id if API takes id
Traits	addTrait(id) / remove	Yes
Decisions	Decision cooldown / real decision entry for interesting ids	Yes for camera decisions
Combat	Existing combat interest path / attack-target gate	Pipeline smoke + known gates
Boat	Unload/board path the patch uses	Pipeline smoke
Meta (city/kingdom)	Real civic create if possible; else unsupported/feed
Building	Destroy/damage path	Pipeline smoke
Libraries	Only with real gain/cast API; else unsupported	Loop when API exists

Rules:

1. Do not call domain_feed / inject and call it live.
2. No fire API → unsupported + reason, not FAIL and not fake pipeline.
3. After each fire: teardown (expire interest keys, clear statuses/wars/plots as needed).
4. Assert: expected note/key or expected drop; flag unexpected outcome_unconfirmed.
5. Busy bypass per Phase 0 table; record in notes.

Exit: every camera domain has ids in id / id_drop / pipeline / feed / unsupported - zero unexplained none for camera rows. Then optionally tighten CI to fail if camera none + feed exceed a budget.

Phase 4 - Wire into suite (weight-aware)

Scenario	Role	When
event_inject_coverage	Wiring gate	Always / event_suite
event_live_apply_loop	Phase 1 happiness+status	event_suite + regression
event_live_relationship	Phase 2 confirms + negatives	event_suite (stable)
event_live_pipelines	Phase 3	Regression only (flake budget proven 3/3)
event_live_coverage	Write ledger + summary assert (fail==0)	End of whichever live scenarios ran
event_suite	audits + inject + apply loop + relationship + outcome smoke	Day-to-day event gate
regression	event_suite + event_live_pipelines (+ other product smokes)	Full gate

Keep event_live_outcome_smoke as a short smoke nested in event_suite.

Phase 5 - Docs sync

• Update docs/event-e2e.md with levels, Busy policy, suite split
• Update event-observation.md “adding an event”: name confirm predicate, fire API, and expected live level
• Source of truth: harness-generated event-live-coverage.tsv for live proof; event-trigger-audit.tsv stays design/call-site audit - do not duplicate live_verified into both unless generating one from the other

Implementation status

Phase	Status
0 Ledger + Busy notes	Done - EventLiveCoverageLedger, assert event_live_coverage
1a Happiness Signal apply loop	Done - EventLiveApplyLoopHarness
1b Status gain apply loop	Done - same harness; spawn human for emotions
2 Relationship confirms	Done - EventLiveRelationshipHarness (leave via Beh/setFamily(null), remove with Prefix emit + anchor hint, baby via createBabyActorFromData; join often pipeline via setFamily)
3 Other pipelines	Done - EventLivePipelinesHarness (traits/decisions/WorldLog/…; building destroy; boat/combat best-effort; libraries unsupported)
4 Suite wiring	Done - event_suite nests apply+relationship; event_live_pipelines in regression; flake budget 3/3
5 Docs sync	Done - event-e2e.md, event-observation.md

Scenarios: event_live_apply_loop + event_live_relationship nest in event_suite.

event_live_pipelines runs in regression (flake budget proven).

Non-goals

• One hand-written harness step per catalog id
• Calling inject / domain_feed “live”
• Promoting feed to “fully verified”
• Full meteorology sim for every disaster when a shared WorldLog pipeline covers the feed (still record pipeline/feed honestly)
• Failing CI on camera none before Phase 3 exit
• Committing generated coverage TSV
• Marking sibling ids pipeline without shared predicate and shared fire API

Success criteria

• Camera happiness Signal + status gains: each id, id_drop, or fixed fail
• Relationship: every confirm predicate has positive and negative sim; setters not piggybacked
• Every other camera domain: id / id_drop / pipeline / feed / unsupported - no unexplained none
• fail == 0 on live runs
• New catalog ids show up as none until a runner claims them
• Coverage vocabulary never equates inject or feed with production-accurate trigger proof