# VISUAL UPLOAD → ON-DEVICE OCR — Implementation Plan > **Protocol 8, Opus (diagnose & plan) stage.** Report-only; no edits, no commits. > A Sonnet session reviews + implements after this plan is approved. > Branch: `dev` @ `fbea0c4` (clean). Owner-approved AI→native conversion of VISUAL UPLOAD. **Locked owner decisions (do not re-litigate):** - Visual Upload goes **native**: on-device OCR is the truth source (player-control principle). - The **only hard rule is FREE**. **Tesseract.js (Apache-2.0, free) is APPROVED.** The "zero-byte / lightweight" ethos is **not** a hard rule here — a free dependency is fine. - Model is **HYBRID (Protocol 32):** on-device OCR **primary**; AI-vision (existing Gemini path) **fallback**. Both behind the remote kill-switch with graceful fallback (Protocol 32/33). - Owner is **mobile-primary** (360/412). Preview/confirm before any state write (Protocol 24). --- ## 1. CONFIRMED CURRENT BEHAVIOR (Protocol 27 — reproduced by reading the live code) ### 1.1 The attach → transmit → AI-vision pipeline **Attach (no OCR today — the image is stashed for the AI):** - `index.html:3796` — composer `[+]` button → `onclick="triggerImageUpload()"`. - `js/ui-saves.js:791` — `triggerImageUpload()` clicks the hidden `#imageInput`. - `index.html:3839` — `<input type="file" id="imageInput" accept="image/*" onchange="handleImageSelection(event)">`. - `js/ui-saves.js:794-806` — `handleImageSelection(event)`: ```js attachedImageMimeType = file.type; const reader = new FileReader(); reader.onload = e => { attachedImageData = e.target.result; // data-URL string const preview = document.getElementById('imagePreview'); preview.src = attachedImageData; preview.style.display = 'block'; // #imagePreviewContainer shows }; reader.readAsDataURL(file); ``` - The two globals live at `js/ui-core.js:1-2`: `let attachedImageData = null; let attachedImageMimeType = null;` **Send (image rides the Gemini request):** `submitCommandInput()` → `transmitMessage()` (`js/api.js:2045`): - `2051` — early-return unless there is text **or** an attached image. - `2062` — `displayUserText = attachedImageData ? '[VISUAL DATA UPLOADED] ' + userText : userText`. - `2068` — **native command router is skipped when an image is attached** (`if (!attachedImageData && _routeNativeCommand(userText))`). - `2130-2137` — adds the `vats-scanning` class to `#imagePreviewContainer` + a `playClack()` scan-loop (the green "scanning" animation). - `2140-2160` — token triage keeps `inventory` in the payload when the text contains `[VISUAL]` (among other keywords). - `2174-2178` — **the actual AI-vision call**: pushes the base64 image (data-URL prefix stripped) as an `inlineData` part on the last user message: ```js lastUserMsg.parts.push({ inlineData: { mimeType: attachedImageMimeType, data: attachedImageData.split(',')[1] }, }); ``` - `2202-2219` — POST to `generativelanguage.googleapis.com …:generateContent`, `responseMimeType: 'application/json'`. The reply is Tri-Node JSON. - `2316-2318` — `parsedNode.state` → `autoImportState(JSON.stringify(parsedNode.state))` (explicit field mapping, additive merge — Protocol 24). - `js/api.js:2034-2037` (`_resetTransmitUI`) — clears `attachedImageData` / `attachedImageMimeType`, hides `#imagePreview`, resets `#imageInput.value`. **The AI directive that shapes the parse** (`js/api.js:142`, `_directiveEconomy`-adjacent): > *Visual Upload Override: Execute `> [VISUAL UPLOAD: CATEGORY]` on a screenshot. You MUST > update ONLY the parsed category. You are STRICTLY FORBIDDEN from deleting un-pictured items > from other categories (e.g., if uploading Weapons, do NOT delete Armor or Junk).* - Advertised in `COMMAND_REGISTRY` (`js/ui-core.js:5365`): `{ cmd: '[VISUAL UPLOAD: X]', desc: 'Parse a screenshot into inventory (Wpn / App / Msc).' }`. ### 1.2 True use case (confirmed) The feature is an **image → state parse**. The directive frames it around **inventory categories** (Weapons / Apparel / Misc), and the AI returns a `state` node that `autoImportState()` merges additively. Nothing in the code restricts it to inventory — the AI *could* also return SPECIAL / skills / HP / caps — but the shipped framing and the `Wpn / App / Msc` registry copy make **inventory the dominant real case**, with **stat-panel screenshots (SPECIAL / skills / level / caps / HP)** the clear secondary case. **⇒ The native parser must handle BOTH: (a) inventory list lines, and (b) a stat panel.** ### 1.3 What "native" changes Exactly like `[USE]` stopped calling `transmitMessage()` (Part A of the prior plan), the visual path stops sending the image to Gemini by default. The `attachedImageData` global + the `transmitMessage()` `inlineData` branch are **kept intact as the AI-vision fallback** (Protocol 22 — reuse, don't fork), invoked only when OCR is unavailable/declined. --- ## 2. TESSERACT.JS INTEGRATION (free, self-hosted, lazy) ### 2.1 Why self-hosted (not CDN) The strict CSP (`index.html:21`) forbids a CDN for this: - `script-src 'self' 'unsafe-inline' https://www.gstatic.com https://www.google.com https://apis.google.com` — **no jsdelivr/unpkg**; the Tesseract JS must be same-origin. - `worker-src 'self'` — **no `blob:` workers** (Tesseract's default). The worker must be a same-origin file. - `connect-src 'self' …` — `'self'` is present, so fetching the wasm/traineddata from same-origin is allowed; a CDN `connect-src` is **not**. - `img-src 'self' data: blob:` — canvas/preprocessing is fine. Self-hosting also makes the feature **offline-capable** (PWA-intact) and avoids adding third- party origins to the CSP. ### 2.2 Vendored files (committed, pinned version) Pin a single Tesseract.js release (e.g. v5.x) and vendor the minimal set: | File | Approx size | Where | Precache? | |---|---|---|---| | `js/vendor/tesseract.min.js` (main API) | ~50 KB | new `js/vendor/` | **yes** (ASSETS) | | `js/vendor/tesseract-worker.min.js` (`workerPath`) | ~0.5 MB | `js/vendor/` | **yes** (ASSETS) | | `js/vendor/tesseract-core.wasm.js` + `.wasm` (`corePath`) | ~3–4 MB | `js/vendor/` | **NO — runtime best-effort** | | `assets/ocr/eng.traineddata.gz` (`langPath`, gzipped) | ~4 MB (≈11 MB unzipped) | new `assets/ocr/` | **NO — runtime best-effort** | - Add an **`Apache-2.0` attribution** note (a `js/vendor/LICENSE-tesseract` + a README/tech-stack line). Free-license compliance. - **Repo size grows ~7–8 MB** (gzipped core+lang). Flag to owner (§7). Binary blobs are `ignore`d from `repomix.config.json` (Protocol 37) so AI-context packs stay lean. ### 2.3 Lazy load (never on boot) - **Do NOT** add Tesseract to the `<script>` chain in `index.html` and **do NOT** import it in any boot phase. It loads **only when Visual Upload is actually used**. - New `js/ocr.js` exposes `async _ensureTesseract()`: - Idempotent (a module-scope `_tesseractPromise` guard) — injects `<script src="js/vendor/tesseract.min.js">` once, resolves when `window.Tesseract` exists. - Configures **explicit same-origin paths** so nothing hits a CDN or a blob: ```js Tesseract.createWorker('eng', 1, { workerPath: 'js/vendor/tesseract-worker.min.js', corePath: 'js/vendor/', // resolves tesseract-core.wasm(.js) langPath: 'assets/ocr/', // resolves eng.traineddata.gz workerBlobURL: false, // CSP: worker-src 'self' forbids blob workers gzip: true, }); ``` - First run pays the network cost for core+lang; thereafter served cache-first (see §2.5). ### 2.4 CSP addition (the load-bearing change) Tesseract compiles WebAssembly. Under CSP, `WebAssembly.instantiate` requires **`'wasm-unsafe-eval'`** in `script-src` (Chrome/modern browsers) — `'unsafe-inline'` does **not** cover wasm. **Add `'wasm-unsafe-eval'` to `script-src`** in `index.html:21`: ``` script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval' https://www.gstatic.com … ``` - This is the **one genuinely risky CSP change**. It must be re-verified live on **both** hosts (Cloudflare staging + GitHub Pages prod) and inside the **worker context** (dedicated same- origin workers inherit the document CSP; confirm the wasm compiles inside the worker, not just the main thread). See §7 hard-parts. - Suite 55 (CSP Stage-2 guards) must be updated: the `unsafe-inline` **tripwire** (55.10–55.11) asserts *no* sha256/nonce is added (which would silently disable `unsafe-inline`); `'wasm-unsafe-eval'` is a **keyword source**, not a hash/nonce, so it does **not** trip that rule — but add an explicit positive guard that `'wasm-unsafe-eval'` is present and a note that it is intentional. ### 2.5 Service-worker caching (offline-safe, install-safe) - `sw.js` `install` uses **`cache.addAll(ASSETS)` — all-or-nothing** (`sw.js:44-45`). Putting the multi-MB wasm/traineddata there would **bloat install and risk install failure** if a byte is missing. **Do NOT.** - **ASSETS gets only the small, safe files:** `./js/ocr.js`, `./js/vendor/tesseract.min.js`, `./js/vendor/tesseract-worker.min.js`. - **The heavy core+lang are runtime best-effort cached at first OCR use** — the exact established `CHANGELOG.md` pattern (`sw.js:53`, `cache.add(...).catch(()=>{})`): ```js // in _ensureTesseract(), after a successful load: if ('caches' in window) { caches.open(_CURRENT_CACHE).then(c => c.addAll(['js/vendor/tesseract-core.wasm.js', 'js/vendor/tesseract-core.wasm', 'assets/ocr/eng.traineddata.gz']).catch(() => {}) ); } ``` (Cache name read from the active `caches.keys()` per the SYSTEM STATUS precedent — never a duplicated `CACHE_NAME` literal.) The cache-first fetch handler (`sw.js:84-89`) then serves them offline on every subsequent use. - **Offline reality:** the **first** Visual Upload needs network to fetch core+lang; **every use after that is fully offline.** This is the honest offline story — call it out in the UX and CHANGELOG (§5, §7). --- ## 3. ON-DEVICE PIPELINE — image → OCR → native parser → validated apply `js/ocr.js` (new module; layering per Protocol 23 — pure parse logic + orchestration; state writes go through existing setters, rendering through ui-render). Boot: appended to the `<script>` chain (after `ui-render.js`, before `ui-core.js`), added to `sw.js` ASSETS and the `tests/test.html` boot chain (Protocol 40). ### 3.1 Flow 1. `handleImageSelection(event)` (rewired, `js/ui-saves.js:794`): if `isFeatureEnabled('visualOcr')` → `runVisualOcr(file)`; else fall to the hybrid path (§4). 2. `runVisualOcr(file)`: - Show the existing `#imagePreview` + a diegetic "OPTICAL SCAN IN PROGRESS…" state (reuse the `vats-scanning` class already wired for the scan look). - `await _ensureTesseract()`; on load failure → hybrid fallback (§4) + `_recordFeatureFailure`. - Preprocess (see §3.2) → `worker.recognize(canvas)` → raw `text`. - `const parsed = _parseOcrText(text, getGameContext())`. - `renderVisualParsePreview(parsed, file)` — the confirm step (§3.4). 3. User reviews/edits the preview → **CONFIRM** → `applyVisualParse(parsed)` writes state through existing native setters (§3.5). **Nothing is written before confirm.** ### 3.2 Preprocessing (accuracy on green-phosphor game fonts) Pip-Boy screenshots are low-contrast phosphor-on-dark. In a `<canvas>` (img-src allows it): - Downscale huge phone screenshots to a sane max dimension; **upscale** tiny text ~2×. - Grayscale + contrast stretch; **binary threshold**; **invert** if background is darker than text (detect by mean luminance) so Tesseract sees dark-on-light. - Optionally region-crop is **out of scope** for v1 (whole-image OCR + robust parse instead). ### 3.3 The deterministic parser — `_parseOcrText(text, ctx)` (pure, game-agnostic) Returns a validated, structured preview object — **never touches state**: ```js { inventory: [ { raw, name, qty, wgt, val, type, matched } … ], stats: { special:{s:..,p:..}, skills:{guns:..}, scalars:{hp,caps,level,rads,xp} }, unparsed: [ …lines the parser could not classify… ] } ``` **Inventory lines** (dominant case): - Split into lines; for each, extract a leading/trailing **quantity** (`(\d+)\s*x?` / `x\s*(\d+)` / `\(\d+\)`), strip noise, take the item name. - **Cross-reference `lookupItemInDb(name)`** (`db_nv.js:813` / `db_fo3.js:469`) — the fuzzy DB lookup returns canonical `{wgt, val, type}`. This is what turns noisy OCR ("Stimpakk") into a real item. `matched=true` when the DB resolves it; unresolved names are still previewable but flagged for the user to confirm/correct (Protocol 24). - Game-agnostic by construction: `lookupItemInDb` reads whatever DB the **active game** loaded (Protocol 38) — no game literals in the parser. **Stat panel lines:** - Labeled patterns → `_resolveStatToken(label)` (`js/api.js:1463`). That resolver already maps SPECIAL aliases, scalar aliases (hp/caps/level/xp/karma/rads), **and per-game skill keys via `getSkillKeys()`** (so FNV `guns` vs FO3 `small_guns`/`big_guns` resolve correctly with **zero hardcoded skill list**). Reusing it keeps the parser Protocol-38 clean and Protocol-22 non-forked. - Extract the numeric value adjacent to each resolved label; clamp happens at apply time (§3.5). ### 3.4 Preview / confirm UI — `renderVisualParsePreview(parsed, file)` (ui-render.js) - Reuses `openModal()` (`ui-core.js:4797`) + the `#sysModal` shell — the **native LOOT modal pattern** (`renderLoot`/`renderLootList`, `ui-render.js:3550`). - Two sections: **DETECTED INVENTORY** (rows: qty · name · resolved wgt/val/type · a keep/discard toggle + editable qty; unmatched names flagged amber) and **DETECTED STATS** (rows: stat label · parsed value · keep/discard toggle). Matches the tracker-row / button conventions (Protocol UI-3/UI-5). - A **"NOTHING DETECTED" empty state** with a one-tap **"TRY AI VISION"** button (hybrid, §4) and a **"MANUAL ENTRY"** fallback (jump to the CARGO MANIFEST add-item form). - Footer: **CONFIRM & APPLY** (calls `applyVisualParse`) + **CANCEL** (`confirmAction`/close). - Mobile-first (owner is mobile-primary): 360/412 verified; ≥16px inputs, ≥28px tap targets (Protocol 17), no horizontal overflow (Protocol 10). ### 3.5 Validated apply — `applyVisualParse(parsed)` (additive, confirm-gated) - **Inventory:** reuse the `addItem()` merge idiom (`ui-render.js:1-52`) / `_lootAdd` (`ui-render.js:3613`) — **additive**, dedup by lowercased name, bump qty, backfill `wgt/val/type` from the DB, **never delete un-pictured items** (mirrors the AI directive's own no-clobber rule, now enforced in code). Emit `item.added` per row (Protocol 22 — same signal the manual/LOOT paths emit; feeds the #18 MANIFEST PUNCH animation for free). - **Stats:** route each kept stat through `_resolveStatToken` → `_applyStatToken` (`js/api.js:1501`) → the `_nativeSet*` setters (`ui-core.js:5534+`). Those already **clamp** (SPECIAL 1–10, skills 0–100, HP≤max, rads≤maxRads, level≤`MAX_PLAYER_LEVEL`, XP to band) and call `saveState()` + emit `stat.change` — **one choke point per stat, never re-implemented** (Protocol 22/24). Discarded rows write nothing. - `renderInventory()` / `updateMath()` / `saveState()` once at the end; a plain-English `appendToChat('> [OPTICAL SCAN] Applied N items, M stat edits.', 'sys')` receipt. - **No new campaign-state field** ⇒ **Protocol 4 is NOT triggered.** Everything writes through existing `state.inventory` / `state.ammo` / the native setters, which are already covered by `sanitizeImportedContainer()` / `migrateState()` and ride the cloud save (Protocol 34 serialized-whole). State this explicitly in the commit. --- ## 4. HYBRID + KILL-SWITCH (Protocol 32/33) ### 4.1 Two flags (register in `js/cloud.js:56` `_featureFlags`, fail-open) ```js visualOcr: true, // on-device OCR (primary) visualAiVision: true, // Gemini-vision fallback (existing pipeline) ``` - Boot merges LKG from `robco_feature_flags` (already wired, `cloud.js:73`) then the remote `/config/flags` doc (`loadRemoteConfig`, `cloud.js:97`). Read via `isFeatureEnabled()` (`cloud.js:127`) — **fail-open / last-known-good** (Protocol 33). The whole check is non-blocking and never black-screens (Protocol 33). ### 4.2 Routing (all fail-safe) | Condition | Behavior | |---|---| | `visualOcr` on | Run on-device OCR (primary). | | OCR load/recognize fails, **or** user taps "TRY AI VISION" | If `visualAiVision` **and** `aiChat` on **and** a key is present → the **existing AI-vision path**: re-stash `attachedImageData`/`attachedImageMimeType`, then `transmitMessage('[VISUAL UPLOAD]')` (reuses `api.js:2174-2178` verbatim — Protocol 22). | | `visualOcr` off (killed) | Skip OCR entirely; go straight to the AI-vision fallback if allowed, else the manual-entry message. | | Both off / offline / no key | Graceful: *"Optical scan and Director vision are both offline — add items via CARGO MANIFEST."* App stays fully usable (Protocol 33). | - After `FAIL_THRESHOLD` (3) OCR failures, `_recordFeatureFailure('visualOcr', …)` (`cloud.js:88`) auto-disables OCR into the AI-vision fallback (Protocol 35 client analogue) — no reload, no black screen. - **Retain the AI-vision code path unchanged** so the hybrid is real, not a rewrite. The directive @142 is **reframed** (not deleted): "native OCR is primary; only produce a Visual-Upload parse when explicitly asked via the AI-vision fallback." Protocol 14: add/adjust a directive-schema test in the same commit (Suite 14-class). --- ## 5. UX (mobile-first, player-authored) - Entry unchanged (Protocol 25 UX stability): composer `[+]` → file picker. Muscle memory kept. - On pick: preview thumbnail + "OPTICAL SCAN IN PROGRESS…" → the **preview/confirm modal** (§3.4) is the player-authored gate. **State is written only on CONFIRM.** - Unmatched OCR rows are shown and flagged, editable, keep/discard per row — the user is the authority, OCR is a suggestion (Protocol 24). - Empty/low-confidence result → "NOTHING DETECTED" with **TRY AI VISION** + **MANUAL ENTRY**. - Diegetic, plain-English chat receipt after apply. - Verified live at 360 / 412 / desktop (Protocol 10/17/26); `render-check.mjs` for overflow. --- ## 6. FILES / FUNCTIONS / TESTS / DOCS ### 6.1 Code | File | Change | |---|---| | **`js/ocr.js`** (NEW) | `_ensureTesseract()` (lazy loader + runtime cache), `runVisualOcr(file)`, canvas preprocessing, **pure `_parseOcrText(text, ctx)`**, orchestration + hybrid routing. | | `js/ui-render.js` | `renderVisualParsePreview(parsed, file)` + `applyVisualParse(parsed)` (reuse LOOT-modal + addItem/`_lootAdd` idioms + native stat setters). | | `js/ui-saves.js:794` | Rewire `handleImageSelection()` → gated `runVisualOcr()`; keep `triggerImageUpload()`. | | `js/cloud.js:56` | Add `visualOcr` + `visualAiVision` flags. | | `js/api.js:142` | Reframe the Visual-Upload directive to "native OCR primary; AI vision on explicit fallback only." Keep the `inlineData` branch (`2174`) as the fallback. `_routeNativeCommand`: optionally add `[VISUAL UPLOAD]` / `[OCR]` token → opens the file picker natively. | | `js/ui-core.js:5365` | Update `COMMAND_REGISTRY` copy (now native/offline). | | `index.html:21` | **Add `'wasm-unsafe-eval'` to `script-src`.** Cache-bump note. | | `sw.js:8` | ASSETS += `./js/ocr.js`, `./js/vendor/tesseract.min.js`, `./js/vendor/tesseract-worker.min.js`. Heavy core+lang = runtime best-effort cache (NOT ASSETS). **Bump `CACHE_NAME` → `-r125`.** | | `js/state.js:17` | *(Optional)* `robco_visual_engine` MetaStore pref (`'ocr'`\|`'ai'`) if a user toggle is wanted. **No new campaign-state field — Protocol 4 not triggered.** | | `js/vendor/*`, `assets/ocr/eng.traineddata.gz`, `js/vendor/LICENSE-tesseract` | Vendored, pinned, Apache-2.0 attributed. | | `repomix.config.json` | `ignore` the binary blobs; `include` `js/ocr.js` (Protocol 37). | ### 6.2 Tests (both runners at parity — Protocol 15) - **NEW Suite 205 — VISUAL UPLOAD OCR** (`robco-diagnostics.js` + `.ps1`, exact parity): - **Static:** `js/ocr.js` exists + in `sw.js` ASSETS + in `index.html` boot chain (after ui-render, before ui-core) + in `tests/test.html` boot chain (Protocol 40); vendored shims in ASSETS; `'wasm-unsafe-eval'` present in the CSP; `visualOcr`/`visualAiVision` in `_featureFlags`; the OCR entry is **gated on `isFeatureEnabled('visualOcr')`**; the AI-vision fallback gated on `visualAiVision`; `runVisualOcr` writes nothing before confirm; the heavy core/lang are **NOT** in the all-or-nothing ASSETS (install-safety guard); parser is **game-agnostic** (no `FNV`/`FO3`/`Fallout` literals; uses `lookupItemInDb`/`getSkillKeys`); Tesseract is **lazy** (no `<script>` tag / no boot import). - **Behavioral (Node `vm`):** run the real `_parseOcrText` against sample noisy OCR strings (an inventory list + a SPECIAL/skills panel) → assert inventory rows resolve via `lookupItemInDb`, stats resolve via `_resolveStatToken`, and the **apply round-trip clamps** (SPECIAL 1–10, skill 0–100, HP≤max) — the Protocol 14/20/24 validation guard. - **Update existing:** Suite 55 (CSP directive — add the `wasm-unsafe-eval` positive guard; re-confirm the unsafe-inline tripwire still holds); Suite 111.11 + Suite 113 macro/registry lists + the `COMMAND_REGISTRY` ↔ `NATIVE_COMMAND_ROUTER` ↔ `[FEATURES]` consistency (Suite 113); `tests/test.html` boot chain + `Suites: N` marker (Protocol 40) if a suite is added there. - **Protocol 2a:** sync the test-count everywhere (CLAUDE.md ×5, RULES.md, README ×4, ARCHITECTURE, CHANGELOG `[Unreleased]` header, both runners' per-suite comments, `test.html`). Run the `Select-String` drift scan. ### 6.3 Docs (Protocol 2/2a/21) - `CHANGELOG.md` `[Unreleased]` — plain-English "Visual Upload now reads screenshots on your device (offline after first use); Director vision is the fallback." - `ARCHITECTURE.md` — new `js/ocr.js` module + boot order, the CSP `wasm-unsafe-eval` rationale, the Native-Input-Path table (OCR is a new native write path), the hybrid/kill-switch. - `README.md` — tech-stack (Tesseract.js dependency), file structure (`js/vendor/`, `assets/ocr/`), script load order, Current-State + feature table. --- ## 7. SCOPE / RISK CALL — this is genuinely large; STAGE it Owner-flagged as *"the largest of the remaining conversions."* Confirmed. **Recommend three staged Protocol-8 units, one push each (Protocol 19 within a unit, Protocol 12 no concurrent):** - **Stage 1 — Infra proof (highest risk first).** Vendor + lazy-load Tesseract, the `wasm-unsafe-eval` CSP change, SW caching, and a bare "OCR → dump raw text into a modal" (no parse/apply). This isolates and **verifies the genuinely hard parts** before any product logic: 1. **CSP + wasm in a same-origin worker** — the `worker-src 'self'` + `wasm-unsafe-eval` interaction must be verified **live on Cloudflare staging AND GitHub Pages** (worker- inherited CSP is browser-nuanced; the test gate cannot catch it — like Protocol 29's real- device rule). **This is the make-or-break.** 2. **Offline story** — first-use network fetch of core+lang, then best-effort cache, then fully offline. Verify the cache actually populates and survives a reload offline. - **Stage 2 — Parser + preview + native apply** (inventory first, then the stat panel). The deterministic `_parseOcrText`, the confirm/preview modal, additive apply through existing setters. Bulk of the product value; low infra risk once Stage 1 lands. - **Stage 3 — Hybrid + kill-switch + failure auto-fallback + full docs/tests.** Wire `visualOcr`/`visualAiVision`, the "TRY AI VISION" path, `_recordFeatureFailure` auto-disable, and finish Protocol 2/2a/13/14 docs+tests. **Genuinely hard / flag to owner:** - **OCR accuracy on green-phosphor Pip-Boy fonts is the real product risk.** Tesseract's default `eng` model is trained on print, not stylized game UI. Preprocessing (§3.2) helps but will not be perfect. **The mandatory preview/confirm step (§3.4) is the mitigation** — OCR is a suggestion the player edits, never an authority (Protocol 24). Set expectations accordingly; consider it "assisted entry," not "magic import." - **Repo weight +~7–8 MB** (vendored wasm + gzipped traineddata). Acceptable for a free, offline, CSP-clean feature, but call it out — it inflates clone/deploy size. - **First-use requires network** (to fetch core+lang). "Fully offline" is only true *after* the first successful scan on a device. Be honest in the UX + CHANGELOG. - **CSP `wasm-unsafe-eval`** slightly widens the script policy. It is a standard, well-scoped keyword (not `unsafe-eval`, not a CDN origin) and does not touch the `unsafe-inline` tripwire — but it is a real security-surface change and belongs in the security-review + a Suite 55 guard. --- ## 8. PLAN-AUDIT (Protocol 8 / 26 — entry paths, states, edge cases enumerated) | Path / state | Intended behavior | |---|---| | `visualOcr` on, OCR loads, items detected | Preview → confirm → additive inventory + stat apply. | | `visualOcr` on, OCR loads, **nothing detected** | "NOTHING DETECTED" modal + TRY AI VISION + MANUAL ENTRY. | | OCR **load fails** (network/CSP/wasm) | `_recordFeatureFailure`; auto-offer AI-vision fallback (if allowed) else manual-entry message; app usable. | | `visualOcr` **off** (remote kill) | Skip OCR; AI-vision fallback if allowed, else graceful message. | | `visualAiVision` **off** too / offline / no key | Manual-entry message; **no black screen**, fully usable (Protocol 33). | | Remote config **unreachable/malformed/slow** | Fail-open LKG (Protocol 33) — features default enabled; boot never blocks. | | **First use ever** (cold cache) | Network fetch of core+lang; best-effort cache; works. | | **Reload offline after first use** | core+lang served cache-first; OCR fully offline. | | **Fresh state vs migrated state** | No new state field; writes ride existing setters/inventory (already migrated/sanitized). | | Desktop vs mobile 360/412 | Preview modal verified at all three; no overflow (Protocol 10/17). | | **Unmatched OCR item name** | Shown + flagged; user edits/keeps/discards; DB backfill only when resolved. | | **Clamp abuse** (OCR reads "S: 99") | Native setters clamp (SPECIAL→10) — OCR can never exceed limits (Protocol 24). | | **No clobber** (upload weapons screenshot) | Additive-only; un-pictured armor/junk untouched (enforced in code, not just directive). | | AI-vision fallback invoked | Reuses `transmitMessage` `inlineData` branch unchanged (Protocol 22). | | Cloud sync of applied changes | Serialized-whole via existing save path (Protocol 34) — no per-field sync code. | **Self-audit result:** every load/reload/cache-clear path, each flag state, offline/online, desktop/mobile, fresh/migrated, and matched/unmatched OCR case has a defined behavior. The one irreducible risk that the gate cannot prove — **wasm-in-worker under CSP on both live hosts** — is deliberately front-loaded into Stage 1 and gated on real-host verification (Protocol 8's "actually live on the deployed branch" audit clause). --- ### One-paragraph summary for the report Visual Upload today stashes the picked image in `attachedImageData` and ships it to Gemini vision inside `transmitMessage()` (`api.js:2174`), whose `state` reply merges via `autoImportState()` — an image→state (mostly inventory) parse. The native conversion adds a free, self-hosted, lazy-loaded **Tesseract.js** pipeline (new `js/ocr.js`): pick → preprocess → OCR → a **deterministic, game-agnostic parser** (`lookupItemInDb` for items, `_resolveStatToken` for stats) → a **preview/confirm modal** → **additive apply through the existing native setters** (no clobber, full clamps, no new state field). It is **hybrid + kill-switched**: `visualOcr` primary, `visualAiVision` (the untouched Gemini path) fallback, both fail-open behind `/config/flags` with graceful degradation and offline usability. The **hard parts** are the CSP `wasm-unsafe-eval` + `worker-src 'self'` interaction (must be verified live on both hosts), the ~7–8 MB vendored wasm/traineddata + install-safe caching (heavy files runtime-best-effort, never in the all-or-nothing SW ASSETS), first-use-needs-network offline caveat, and OCR accuracy on green-phosphor game fonts (mitigated by the mandatory player-confirm step). **Recommend staging into three units** — infra proof → parser/apply → hybrid/kill-switch/docs — front-loading the CSP/wasm/offline risk.