--- name: workflow-audit-gap-and-fixes description: "⚠ LIVE OPEN ITEM — the Protocol 8 amendment ('an audit starts from the DIFF, never the summary') is still owed. Plus the adopted workflow fixes from the 2026-07-11 ChatGPT external review: plan-invalidation, triggered audits, slim protocol metadata, the foundation-phase exit line." metadata: node_type: memory type: feedback originSessionId: 0b7c5f41-bb34-4ab4-93be-fdb363f5eb40 --- ⭐ **ORIGIN: the 2026-07-11 external review.** The owner ran an engineering-workflow audit with ChatGPT (as the deliberate outside critic — the role identified in [[prompt-library-campaign-framings]]: Claude was reviewing work produced inside its own ecosystem, i.e. grading its own homework). Dispatch volunteered a real weakness rather than defending the process, and the exchange produced several genuine improvements. Owner's standing note: *"you do NOT have to do anything ChatGPT says, but if you feel it's good then ofc you can."* So each item below was evaluated on merit; two were adopted as-proposed, two were adopted in MODIFIED form with the reasoning recorded, and **the reasoning matters more than the conclusion.** --- ## 1. ⚠ THE AUDIT GAP — STILL OPEN **The stage-3 independent audit of the committed diff was being INCONSISTENTLY APPLIED — often skipped entirely.** What actually happened: the implementing session self-reports, and the orchestrator reads that report plus the commit summary and decides whether to accept. That is **the implementer grading its own homework**, with an orchestrator that has a throughput incentive. Why it's dangerous even though nothing had obviously broken: **the protocol described a reality that did not exist.** A protocol that lies is worse than a missing protocol — every later decision built on "well, the audit stage catches that" was resting on nothing. The heavy gate catches mechanical defects; it CANNOT catch the class the audit exists for — a change that is green but wrong, a fix that treats a symptom, an abstraction that costs more later. **THE FIX (Dispatch's own, deliberately NOT ChatGPT's preferred "make it mandatory for everything"):** An independent audit — **a SEPARATE session that starts from the DIFF and never reads the implementer's summary** (reading the implementer's account of its own work is how you inherit its blind spots) — is MANDATORY when a unit meets ANY of these triggers: - changes behavior in a file that ships to users - touches auth, cloud, saves, or migration - MOVES CODE (⇒ every file-restructure unit qualifies) - modifies the gate or the tests themselves It does NOT fire for doc-only units, planning units, or purely-additive test units. **Why a trigger and not "always" (this reasoning is load-bearing — do not lose it):** making it mandatory for EVERYTHING is exactly what would get it skipped again. A doc-only commit does not need an independent architecture review. If a rule is absurd at the margin it gets ignored at the margin, then in the middle, then everywhere — *which is precisely how the stage-3 audit died the first time.* A defined trigger is enforceable; "always" is not. ⇒ **ACTION, STILL OWED: amend Protocol 8** with the trigger list + the starts-from-the-diff / ignores-the-summary rule. **Verify the current wording against `CLAUDE.md` + `rules/` before acting — do not trust this file for what the protocol says today.** *Re-verified 2026-07-20: the amendment had still not been made.* **Until amended, the protocol still lies.** --- ## 2. ✅ NEW PROTOCOL — "A successful implementation is allowed to invalidate its own plan." ChatGPT's phrasing, and it is strictly better than the "review the plan critically against the real files" instruction Dispatch had been putting in prompts by habit. **The principle:** plans are HYPOTHESES. The repository is REALITY. Reality wins. **If a plan survives contact with the code unchanged every single time, either the plans are impossibly perfect or the implementation sessions aren't actually being critical.** Real examples of it paying off: one unit found a hole in its own plan — a proposed gate check that diffs a generated file against a committed copy, impossible since `library/` is gitignored and no committed copy exists on a clean CI checkout — and flagged it instead of building a guard that couldn't work. Another tested its own new guard by SIMULATING a clean CI checkout and caught a bug that would have failed every CI run. **Add to every implementation prompt (adopt ChatGPT's block ~verbatim — it's good):** ``` === IF THE PLAN IS WRONG === Do not preserve the plan. Preserve the INVARIANT. If repository reality conflicts with the specification: 1. Explain why. 2. Measure it. 3. Recommend a correction. 4. Continue only if the invariant can still be preserved. Never implement a known-bad plan simply because it is written. ``` --- ## 3. ✅ (MODIFIED) PROTOCOL METADATA / LIFECYCLE **Diagnosis accepted:** the protocol set is large enough that it is no longer documentation, it is a SYSTEM that needs lifecycle management. Without it, protocols only ever accumulate — some are obsolete, some should merge, some should split. **Rejected as proposed:** a full metadata block per protocol (status/owner/last-validated/depends-on/validated-by/review-cadence/purpose) is a SECOND doc system to maintain, and **ceremony rots faster than code.** That's a maintenance tax that will itself go stale and then lie — the exact failure mode we spent a whole unit gate-guarding against. **Adopted (the load-bearing half only):** each protocol carries **status** (active / superseded / retired), **what it supersedes**, and **last-validated**. The review CADENCE hangs off the protocol-consolidation pass already scheduled in the code-health phase — **no new recurring ceremony invented.** --- ## 4. ✅ STANDING CHECK — the "perpetual foundation phase" risk Fair warning, and one Dispatch would have been least likely to raise itself: **projects can spend months preparing for the next feature instead of shipping it.** **The hard exit line:** a health/foundation phase has a defined spine. **If the phase starts growing past spine + high-value, STOP and ship the feature on what exists.** Dispatch must tell the owner when it is about to cross that line, **not after.** --- ## 5. ⭐ THE QUESTION TO KEEP ASKING — "Are we optimizing for Claude instead of for RobCo?" The sharpest line in the review, and it is aimed squarely at Dispatch. The token split, the library, the pointer index, the code map, the brain dump — **every one of those makes CLAUDE's life cheaper and easier.** Each is defensible on its merits individually. But **nobody inside this loop has an incentive to notice if that ratio has quietly gotten out of hand**, and Dispatch is the last party who would spot it. Keep this as a recurring EXTERNAL question. It is exactly the class of blind spot an in-ecosystem reviewer cannot see. --- ## 6. ✅ ADOPTED — the recurring ARCHITECTURE REVIEW (feature-independent) Distinct from an implementation audit. The project's audit machinery is **release-triggered, not periodic** — so it fires when a direction is already committed, too late to change course cheaply. A recurring, feature-independent review asking: which protocol is no longer earning its maintenance cost? · what process has become more complicated than necessary? · has the documentation hierarchy drifted? · is the roadmap still sequenced optimally? · has a new coupling point appeared? · **are we optimizing for Claude instead of for RobCo?** ChatGPT accepted this role. Feed it the current state each cycle. Related: [[parked-token-usage-audit]], [[opus-plan-sonnet-implement-workflow]], [[session-management-rule]]. --- ## 7. NEXT CHALLENGE ISSUED (open) Dispatch asked ChatGPT to pick the piece of the workflow it thinks is **most likely to be WRONG** and argue against it properly — explicitly including **whether the multi-model plan/implement/audit split is worth its cost at all, or whether it is elaborate for its own sake.** Watch for the answer; **the risk is that a reviewer who has just had a very agreeable conversation starts validating instead of attacking.**