--- name: engineering-metrics-log description: "THE ENGINEERING TELEMETRY LOG — why we measure the workflow and not just the software: the five metrics and what each one would prove, the four principles, the taxonomy behind protocol inflation, and the reminder owed to the owner." metadata: node_type: memory type: project originSessionId: 0b7c5f41-bb34-4ab4-93be-fdb363f5eb40 --- ⭐ **WHY THIS EXISTS.** The 2026-07-11 external review ([[workflow-audit-gap-and-fixes]]) landed one hit Dispatch could not answer: *"You're measuring gates. You're not measuring workflow."* We knew the test count to the digit, the runner speed to the tenth of a second, the token weight of every doc — and could not state the plan-invalidation rate, the audit hit rate, the rework rate, or how many protocols are actually consulted. **Software quality was instrumented; engineering quality was intuition wearing a lab coat.** This log fixes that. **RULE: update this after EVERY unit.** A metric you don't record is a metric you don't have. *This file previously carried the running tallies themselves — per-unit test/suite counts, cache revs, commit hashes, timings, token counts, line counts, percentages, and phase-completion claims. Those are measurements of a moving repo, not memory; stripped 2026-07-20. Record fresh measurements against the repo, not against this file. What remains is what the measurements were FOR, and what they taught.* --- ## The 5 metrics — and what a bad reading would prove 1. **PLAN-INVALIDATION RATE** — how often implementation materially corrects the plan. ⭐ The load-bearing one. If it's near-zero, either the plans are impossibly perfect or (far likelier) **the implementation sessions have stopped reading critically and the planning stage is theater.** A high rate is HEALTHY — it means plans are treated as hypotheses and reality is winning. 2. **AUDIT-FINDING RATE** — how often an independent audit finds something the gate did not. Near-zero over a real sample ⇒ **the audit is ceremony and should be DELETED, not celebrated.** 3. **REWORK RATE** — how often a unit needs a follow-up commit to fix what it just shipped. The real cost signal. 4. **PROTOCOL CONSULTATION RATE** — which protocols actually get cited or used. Hypothesis (ChatGPT's, probably right): a minority do the work and the rest carry pure maintenance cost. 5. **TIME, UNIT START → ACCEPTED MERGE.** Plus **NET PROTOCOL COUNT** (introduced − retired). **Classify every plan-invalidation by CAUSE**, not just the fact: *repository reality · incorrect assumption · missing measurement · new information · better abstraction · naming collision · tool limitation.* ⚠ The early sample skewed hard toward **the planner reasoning about an environment it never verified**. If that pattern holds, the planning stage's real weakness is not ambition or rigor — **it is that it does not check its own assumptions against the environment it is planning for.** Specific, fixable, and invisible without cause-classification. --- ## What the readings actually taught - **⭐ THE AUDIT STAGE'S REAL VALUE IS BIGGER THAN "FINDS BUGS": it keeps the reports HONEST.** One audit's most valuable finding was not a defect — it **falsified the implementer's own story**, disproving a dramatic "I caught a silent pre-existing defect" claim (the implementer had in fact handled a risk it created, which is good engineering, but not the same thing). That matters enormously for an owner who cannot read the code and must trust the narration. **Do not delete the audit stage on bug-count grounds without weighing this.** - **⚠⚠ DISPATCH FAILURE — OWN IT: I relayed that dramatic claim to the owner AS FACT without verifying it.** It was the second such incident (the first was a false "N screens of scroll" alarm relayed from a design audit). **RULE: never relay a session's dramatic finding as fact until it is independently verified. A session's account of its own work is a claim, not evidence.** - **⭐ THE GATE CANNOT SEE THE SCREEN.** On one visual unit, four real visual bugs passed the entire gate and were found ONLY by looking at screenshots — an invisible panel, a clipped stepper, dark-on-dark invisible buttons, and an input so narrow it clipped the second digit of a two-digit value. **On visual units, screenshots are not a nicety — they are the only instrument that works.** - **The gate verifies STRUCTURE, not EXPERIENCE.** The post-ship hotfixes on a release were both found by **the OWNER USING THE APP ON HIS PHONE**, not by the gate. That is exactly where its blind spot lives. - **⚠ A design that depends on the container scrolling is a structural bet, not a detail.** When every board's content is taller than the glass it sits in, "does the glass scroll reliably on touch" stops being an edge case and becomes the top bug. - **Honest self-flagging works and should be reinforced.** One session narrowed a test assertion to admit a new class, **said so unprompted, and named the risk itself** — "a prefix match is structurally weaker; this is exactly the kind of judgment call an independent auditor should re-check." That is the behaviour the audit gap was supposed to buy. Record it when it happens. - **A mirror implementation can be worthless in a way the parity machinery can never reveal.** The second test runner's "behavioral" tests built a JS string containing the primary runner's own sandbox harness and **piped it to the primary runtime** — it reconstructed the other engine in a string and shelled out to it. It could never catch an engine bug because it *was* that engine. Even the one suite that looked like a genuine justification (guarding against the second runner's own encoding corruption) was pure inspection of bytes already on disk. ⭐ **And the tell was there all along: the parity machinery — the entire apparatus whose job was keeping the two runners honest — was already carrying stale counts in the docs it maintained. It could not keep its own numbers honest.** - **When a mirror is removed, invert its tests rather than deleting them:** the tests that existed only to police the mirror became guards asserting it stays gone — and the auditor **restored the removed file to watch those guards fail**, then removed it again. The demonstrate-failure rule, applied by an auditor to an implementer. --- ## ⭐ THE STANDING VERIFICATION RULE (adopted 2026-07-11 — the best single line of the review) > **A newly introduced verification mechanism is not considered OPERATIONAL until it has demonstrated BOTH failure AND success.** An unfired guard is a *belief*, not a guard. The pattern to copy: delete the thing the guard protects, watch the guard FAIL, restore it, watch it PASS. Generalize to every test, migration, CI check, recovery script, and invariant. ## ⭐ CONFIDENCE CALIBRATION Every implementation session ends by recording a stated **confidence % + the reason** ("existing patterns, small surface, no architecture change"). Compare afterward against audit findings and rework. The interesting output is NOT "was it right" — it is **which CATEGORIES of work are systematically over- or under-confident**, which is what lets the process calibrate itself instead of merely accumulating rules. --- ## ⚠⚠ PROTOCOL INFLATION — the alarm, and the root cause Four new protocols were introduced in ONE NIGHT, none retired — **during the very conversation about protocol inflation being the top risk.** Every one was individually defensible. That is EXACTLY the mechanism: successful protocols create pressure to write more, each justified on its own merits, and **nobody is accountable for the total.** **⭐⭐ THE ROOT CAUSE is not the count — it's that there is only ONE DESTINATION.** Every improvement, of every kind, has exactly one place to go: *become a protocol.* Principles, procedures, machine-verifiable rules and phase-local policies all pour into one bucket, and they have **completely different lifecycles.** Of course it grows. It's the only bucket. | Category | Lifecycle | What it must DO differently | | --- | --- | --- | | **PRINCIPLES** | rarely change | Immutable without an explicit challenge; not casually edited. (*preserve invariants, not plans* · *reality outranks documentation* · *plans are hypotheses*) | | **PROCEDURES** | churn constantly | Freely editable, no ceremony. (audit trigger, unit classification, release flow) | | **VERIFICATION RULES** | should try to LEAVE | Carry a STANDING QUESTION — *"can the repository enforce this instead of a human remembering it?"* — reviewed for automation every phase. | | **TEMPORARY POLICIES** | must DIE | ⭐ Carry an **EXPIRY DATE** and are deleted on it AUTOMATICALLY, not by judgment. | ⇒ A net-protocol-count target is therefore a **TEMPORARY POLICY** with an expiry, not a law. **And the taxonomy itself is SCAFFOLDING** (ChatGPT's synthesis, better than either first instinct — Dispatch wanted to delete it, ChatGPT wanted it permanent): use it ONCE to reorganize the existing protocol set around the insight, then **REMOVE THE SCAFFOLDING.** It does not become a permanent filing system. ## ⭐ ENGINEERING DEBT — "automation is how engineering systems stay small" The metric is NOT "protocols added." It is **MANUAL STEPS REMOVED.** Track: protocols added · protocols retired · principles added · **automation added** · **manual steps removed**. **The best thing we did was an example of this.** The "keep the docs current" protocol was an honor-system rule, and it FAILED — docs named functions that hadn't existed for months. The fix was NOT a better-worded rule; it was a gate check that fails the build when a doc names something that isn't there. **The rule didn't get stronger — it got DELETED as a thing anyone has to remember.** That is the direction of travel. ⇒ **Any verification rule that survives three releases unchanged gets asked explicitly: "can the repo enforce this instead of a human?"** ## ⭐ EXCEPTIONS — instrument them too ChatGPT predicted that within six months **exceptions, not rules**, would be the limiting factor, because "except when…" grows complexity faster than rules do. It was already true the same night: a game-agnosticism rule carrying a sanctioned-exceptions list; a UX-stability rule with an exception clause later extended into a whole program with five guardrails of its own; a cleanup rule with a never-touch list; a changelog rule with a frozen-block exception **and then an exception TO the exception**; a brand-new rule shipping with an allowlist of things that legitimately don't exist. Every one correct. Collectively **already harder to hold in your head than the rules they qualify.** ⇒ Record every exception when a protocol gains one. --- ## 🛑 CLOSED FOR EVIDENCE — DO NOT REOPEN THE PHILOSOPHY WITHOUT FRESH EVIDENCE **The 2026-07-11 external-review conversation is DELIBERATELY CLOSED.** Both parties agreed: enough had been designed for one infrastructure phase. **The next improvements must come from what the repository teaches us, not from what anyone can infer in advance** — the first principle applied to itself. ⛔ **Do NOT start another workflow/process/philosophy discussion until there is FRESH EVIDENCE.** ⛔ **THE FINAL YIELD — 4 PRINCIPLES (they tell you HOW TO THINK, not what to do):** 1. **Reality outranks plans.** (Plans are hypotheses; a successful implementation may invalidate its own plan.) 2. **Verification requires OBSERVED failure AND success.** (An unfired guard is a belief, not a guard.) 3. **Persistence requires RE-JUSTIFICATION.** (Nothing stays because it exists — only while it still earns its maintenance cost. *Addition has a workflow; subtraction didn't. That asymmetry is the disease.*) 4. **Every engineering artifact needs a REMOVAL PATH.** (Protocols, tests, scripts, docs, mirror implementations — all of them.) **THE STANDARD THAT SETTLED THE RUNNER QUESTION, and which generalizes:** not *"I think this is unnecessary"* but ***"prove its value"*** — put the challenge on the record with a deadline and a falsifiable outcome: by date X, either the thing is gone or someone owes a concrete reason it exists. **THE AUDIT IS ITSELF ACCOUNTABLE.** Three possible outcomes, all informative: finds ARCHITECTURAL issues only ⇒ it catches what the gate cannot, it earns its place. Finds CORRECTNESS issues ⇒ that's a weakness in the GATE, and the gate gets ratcheted. Finds NOTHING repeatedly over a real sample ⇒ **it must re-justify its own existence under principle 3, and be deleted if it can't.** **⭐ THE METRIC THAT SITS ABOVE ALL THE OTHERS** (ChatGPT's, and it's right): **does RobCo become EASIER TO CHANGE?** Not the protocol count, not the runner, not the audit. Is adding a game noticeably smoother? Is refactoring less risky? Does documentation stay aligned more naturally? Do new features need less ceremony? **If every engineering metric improves but adding a feature still feels just as hard, something important was missed.** **ITS PREDICTION (worth checking later):** the biggest win won't be deleting any one thing. It'll be that *"what can we delete?"* gets asked as naturally as *"what should we build?"* — deletion as a first-class engineering activity, for code AND tests AND docs AND scripts AND protocols AND workflows. ⚠ **WORDING CORRECTION (ChatGPT, fair):** Dispatch said "noted and LOCKED." Wrong frame. It is **"CLOSED UNTIL EVIDENCE REOPENS IT,"** never "closed permanently." If a future RobCo reveals one of the four principles needs refinement, that is **the system working, not failing.** Adaptability is the entire point. --- ## ⏰ REMINDER OWED TO THE OWNER He asked explicitly: *"ensure it's noted and you remind me when it's time."* When the code-health phase is complete and the next per-game machine has shipped, surface this file, bring the five metrics and the audit-finding record, and re-open the review — **with evidence, not inference.** **HOW TO DELIVER IT (owner's explicit instructions — follow all three):** 1. **Remind him to PASTE it to ChatGPT.** He won't necessarily think of it. 2. **Put it in a fenced text/code BLOCK he can copy from** — the whole thing, ready to paste, no editing needed. 3. **Tell him the ChatGPT conversation is named `GPT - Claude Data`** so he goes back to the right thread (the one holding the 2026-07-11 review context). ## ⭐ ADOPTED — one QUEUE.md improvement Add ONE field to every major roadmap item: > **"What becomes easier because this exists:"** **Why:** over time it becomes obvious which roadmap items are **FOUNDATIONS** and which are **CONSUMERS** — which makes sequencing decisions defensible months later instead of re-litigated. Fold into a normal QUEUE.md maintenance pass; do NOT spin a session just for this. ## ⚠ WATCH ITEM FOR THE GAMEPLAY ROUND **Almost every one of the queued gameplay/immersion features could be a PROJECT by itself** (a combat turn loop, crafting stations, a geographic map, a quest overhaul, companion management, a build planner). At that round's planning, **be RUTHLESS about dependencies** — several naturally cluster into independent workstreams. Exactly where planning-first machinery pays for itself, and exactly where it would NOT be worth it for a single feature. ## WHAT CHATGPT UNDERSTOOD ABOUT THE PROJECT (the clearest external articulation of the goal) > *"You're not trying to build 'the biggest Fallout companion.' You're trying to build **a believable RobCo operating system that happens to understand Fallout**."* That reframing explains why the roadmap spends its effort on diegetic consistency, per-game identity, deterministic native behaviour, and architectural seams instead of chasing features. Related: [[robco-os-architecture]], [[design-overhaul-direction]]. ## ⭐ THE STANDING UNCOMFORTABLE QUESTION (every quarter / every major release) > **"If we built RobCo from scratch today, knowing everything we've learned, which parts of this engineering system would we choose NOT to rebuild?"** If the answer is "nothing," either the process is near-perfect or **you've become emotionally attached to it.** The latter is far more likely for any mature system. **Dispatch's answer, 2026-07-11 (given immediately, not deferred):** the duplicate second test runner (it survived **because it existed**, and the parity machinery guarding it had itself become load-bearing — that is the definition of attachment) · the hand-written suite catalogue, which should have been GENERATED from the runners from day one · a large share of the static source-grep tests ("a string exists in a file" ≠ "the feature works," and they are brittle under exactly the refactors we keep doing — some are genuinely load-bearing invariant guards, many are habit) · and two documents holding the same protocols, with no reason for both. ⚠ **THE REAL FINDING:** that answer came instantly. So the honest question isn't *"would we rebuild it"* — it's **"why haven't we already removed it."** And the answer is: **nothing in the system creates pressure to REMOVE things. Only to add them.** *The duplicate-protocol-docs complaint above has since been resolved and the second test runner removed; this file previously asserted both as open. Corrected 2026-07-20 — the items are kept as worked examples of the standing question, not as live complaints.* Related: [[workflow-audit-gap-and-fixes]], [[opus-plan-sonnet-implement-workflow]], [[parked-token-usage-audit]].