--- name: data-provenance-program-complete description: "The FO3+NV data-provenance cleanup — what it taught. Model-estimated game data had been falsely stamped as wiki-sourced; the fix established that guards must pin NUMBERS not citation labels, that UNVERIFIED data is left alone rather than guessed, and that a source-of-record doc is a hypothesis, not truth." metadata: node_type: memory type: project originSessionId: 1060dea9-0dc3-42b4-958d-2a6f7a6c0c58 --- *Rewritten 2026-07-20: test counts, suite counts, cache revs, APP_VERSION, commit hashes and the itemized shipped inventory were removed — all derivable from git log, `CHANGELOG.md` and the source. The lessons and rulings are what memory is for.* ## HOW IT STARTED — the incident A single fake FO3 karma warning ("ENCLAVE HIT SQUAD") turned out not to be isolated ([[fo3-karma-rebuild]]). It was a whole **CLASS** of defect: game data had been **model-estimated, then falsely stamped with a `// Source: fallout.wiki` comment.** The citation was decoration. A provenance sweep and fix pass then ran across **both games, every data table**, with each fix **independently audited** and spot-checked against the LIVE wiki — never trusting the sweep or the implementer's own report. ## ⭐ THE LESSONS (these are the deliverable) **A citation label proves nothing. Guards must pin the NUMBERS.** The golden-master approach that came out of this pins every *verified* stat value, plus **per-game plausibility range bands** that would have tripped the wildly-wrong values on their own. Checking that a citation comment exists is theatre; checking the value is the test. **Never pin UNVERIFIED values as canonical.** Where a value could not be confirmed at the wiki, it was **left byte-identical rather than guessed**, and covered only by the band guard. Guessing to make a table "complete" is how the original rot happened. **A source-of-record doc is a HYPOTHESIS, not truth — verify at the wiki in BOTH the build and the audit.** This paid off repeatedly and concretely: the provenance sweep itself was wrong more than once (it claimed a weapon was a New Vegas item when it is actually from FO76; it flagged items that were correct as-is; it missed two fake perks that were actually from the other game), and the **builds caught the sweep's errors** because they re-verified rather than trusting the document handed to them. **The independent audit earned its keep.** **⭐ The owner's main game got more care.** The New Vegas registry (perks/traits/books/magazines) came back **CLEAN**. The rot was concentrated in the numeric stat tables, not the registries — and it was worse in FO3 than in NV. Expect his primary game to be the more trustworthy one, and weight scrutiny accordingly. **Errors are not uniformly biased.** An early "~10× inflation" headline turned out to be a small-sample fluke; the real errors were **scattered in both directions**. Don't extrapolate a systematic story from the first few data points. ## RE-SOURCING DECISIONS WORTH KEEPING - Rows belonging to a **different game** are deleted from the wrong game's table, but kept where they genuinely belong (several items were real in the other title). - **Invented items are deleted outright**, not corrected — there is nothing to correct them to. - **Duplicate rows get removed AND aliased**, so the old names still resolve through the real lookup path. Without the alias, one name fuzzy-matched a *different* real item — worse than a miss. - Where a doc and the wiki disagree, **trust the wiki and proceed.** ## STILL GENUINELY OPEN — and the ruling on it **NV bestiary numbers are level-scaled and likely have no single canonical wiki value — this may be uncloseable.** Names were verified clean; the numbers are honestly left UNVERIFIED under a band-only guard. Do not "fix" them by picking a number. ## ⚠ DISPATCH OPERATIONAL NOTES - **A dispatched code session that calls AskUserQuestion HANGS** — the orchestrator's `send_message` does not dismiss its modal, and the session sits at the same turn through repeated messages and long waits. **Fix: abort it and re-dispatch a fresh session with the decision pre-baked**, plus an explicit "do NOT call AskUserQuestion; if a doc contradicts the wiki, trust the wiki and proceed." That worked cleanly. - **`start_code_task` can time out while still spawning the session in the background.** Check the session list before retrying — retrying blind once produced **three duplicate sweeps**. Fallback that worked: reuse an idle session via `send_message` (warm context, no creation). - **WebFetch is 403-blocked on fallout.wiki** — verification has to go through the Chrome browser tools or web search. Related: [[fo3-karma-rebuild]], [[reframing-review-2026-07-13]] (the citation-guard lesson), [[deep-systems-review-2026-07-13]], [[dispatch-autonomy-preference]].