# The extension Manifest V3 architecture for a browser extension that **observes** a web application's own network traffic and keeps it, without ever originating a request. Everything here is generic. Where the original project named a specific site or endpoint, this says "the target" and "the API". --- ## 1. The invariants Write these down before any code, because every later decision refers back to them. Ours: > - **Never originates a request.** It wraps `fetch`/`XHR` and reads what came back. > - **Never blocks, delays or alters a response.** The original promise/object is returned > untouched; the body is read from a clone, after the fact. > - **Never throws into the page.** Every hook is wrapped in `try`/`catch`. > - **Never transmits.** Export is a manual download. They are not just ethics — they are what makes the tool safe to leave running in a real session on a real account. A recorder that alters a response can break the page it is watching and you will spend a day thinking the target changed. --- ## 2. Architecture Four pieces, each in a different execution context, which is the part that trips people up. ``` ┌─ MAIN world (the page's own JS context) ─────────────────────────┐ │ page-interceptor.js │ │ wraps window.fetch and XMLHttpRequest │ │ reads bodies from a CLONE, never the original │ │ forwards via window.postMessage │ └──────────────────────────┬───────────────────────────────────────┘ │ postMessage (the only way across the world boundary) ┌──────────────────────────▼───────────────────────────────────────┐ │ ISOLATED world │ │ content-script.js │ │ validates the message, relays it on with chrome.runtime │ │ pushes config (capture mode) the other way │ └──────────────────────────┬───────────────────────────────────────┘ │ chrome.runtime.sendMessage (SIZE-LIMITED — see §7) ┌──────────────────────────▼───────────────────────────────────────┐ │ service worker (background.js, type: module) │ │ classification, storage, export building, health │ │ IndexedDB lives here │ └──────────────────────────┬───────────────────────────────────────┘ │ ┌──────────────────┴──────────────────┐ │ side panel │ toolbar popup │ extension pages: same origin as │ (the real UI) │ (a launcher) │ the worker, so they can open the └────────────────────┴────────────────┘ same IndexedDB directly ``` ### Why MAIN world at all A content script in the ISOLATED world gets its own `window`. Patching `fetch` there patches a `fetch` the page never calls. To see the page's traffic you must run in the page's own context, which manifest V3 supports declaratively: ```json { "matches": ["*://target.example/*"], "js": ["src/detect.js", "src/page-interceptor.js"], "run_at": "document_start", "world": "MAIN", "all_frames": false } ``` `document_start` matters: the app's first API calls happen early, and a hook installed at `document_idle` misses the page load — often the richest payloads in the session. The MAIN-world script has no access to `chrome.*`. It can only shout into `window.postMessage` and hope somebody is listening. That somebody is a second declaration of the same content script in the ISOLATED world, which does have `chrome.runtime`. ### The double-injection trap The worker also injects into tabs that were already open when the extension loaded (otherwise a tab that has not navigated since install is invisible). Without a guard, a tab that gets both the declarative injection and the programmatic one wraps `fetch` twice and reports every response twice. ```js const CHANNEL = '__TOOL_NAME__'; if (window[CHANNEL]) return; // bail before installing anything window[CHANNEL] = counters; // claim the flag BEFORE the hooks go in, // so a racing second injection loses cleanly ``` Claim the flag *first*. If you claim it after installing the hooks, two injections racing can both pass the check. ### Expose your counters on that global ```js window.__TOOL_NAME__.counters ``` The single most useful debugging affordance in the project. The human can open devtools on the real page and read `fetchSeen`, `matched`, `forwarded`, `bodyReadFailures` without any extension UI, and paste the numbers to the agent. It costs three lines. --- ## 3. Reading a body without disturbing it The rule is that the page must be unaffected. For `fetch`: ```js const orig = window.fetch; window.fetch = function (...args) { const p = orig.apply(this, args); p.then((res) => { try { // .clone() gives a second readable stream. Reading the original would consume it // and the page would get an empty body. res.clone().text().then(handleBody, noteFailure); } catch (_) { noteFailure(); } }, () => {}); return p; // the page gets the untouched promise }; ``` For `XMLHttpRequest` you hook `load` and read `responseText` — with one trap that cost three capture runs: **if the app sets `responseType` to `arraybuffer` or `blob`, `responseText` throws.** The symptom is a rising "matched" count and a zero "captured" count, which looks like success from a distance. Handle every `responseType`, and count the failures. Two more: - **`content-type` is not a reliable filter.** A modern app may serve structured data as `application/octet-stream`. Classify on URL shape *and* content type, and let each capture mode decide how much to trust either. - **Cap the body size, truncate rather than drop, and flag it.** A record that says `truncated: true` with 4 MiB of content is a fact you can act on. A dropped record is silence. Store the *original* length alongside so you can see what you lost. **Then revisit the cap when the target changes.** Ours was 1 MiB, tuned when payloads were dense JSON. The target moved to a server-driven-UI format that ships the interface description alongside the data — roughly two orders of magnitude less dense — and 1 MiB started cutting the two most valuable responses in every run. A cap tuned for a dense format silently guts a sparse one. Raising it to 4 MiB then created a *different* problem two layers away (§7), which is the honest shape of this kind of work. --- ## 4. Classification: capture modes, not a filter A single "is this interesting" predicate is a trap, because the answer changes as you learn. Ours is a ladder of named modes, chosen in the UI: | mode | takes | |---|---| | `engagement` | a narrow allow-list of endpoints known to matter | | `wide` | everything on the API paths **except** an explicit noise list ← the default | | `json` | anything that parses as JSON, whatever it claims to be | | `all` | everything | `wide` is the one to default to, and the reasoning generalises: **an allow-list fails silently when the target adds an endpoint; a deny-list fails loudly when it adds noise.** On a real session the narrow list caught 14 responses, `wide` caught 33, and `all` caught 181. The 19 the narrow list missed are exactly the ones you would never have known to ask for. Two lists, and they are not the same kind of thing: - **The noise list** is an optimisation. Badge pollers, tracking pixels, config fetches. Getting it wrong wastes storage. - **The deny list is a privacy floor.** Message content, for instance, is never captured in any mode, including `all`. Getting it wrong is a breach. When the target introduced a second API stack, the deny list had to be re-derived for the new naming convention — the floor is *per transport*, and a rule written for one stack does not protect the other. That is worth an explicit test: iterate every mode and assert the denied surface stays denied in all of them. --- ## 5. Storage: split metadata from bulk, and cap everything IndexedDB, with the object stores split by size: ``` records one row per response: seq, ts, url, pageUrl, status, contentType, bodyLength, truncated ← small, listed constantly bodies seq → the body text ← large, fetched one at a time observations everything seen, matched or not, with the reason it was refused resources performance-timeline entries: what the page fetched at all exports generated file metadata ← small, listed by the UI exportBlobs seq → the file contents ← large ``` **The split is the point.** Listing 500 records to render a table must not load 100 MB of bodies. Every store that grows has a cap and oldest-first eviction. The `observations` store deserves special mention: it records **everything the interceptor saw, including what it refused, and why**. That log is what diagnoses the tool rather than the data. Twice it settled a question nothing else could: once proving the browser was running an old build (the refusal reason string was one the new code cannot emit), and once letting a later version be replayed against an earlier run's traffic to measure what it *would* have captured. Sizing caps: **write down what a "run" costs today, and expect it to move by an order of magnitude.** Our export caps were 20 files / 300 MB, set when a run was 16 MB. Within three weeks a run was 113 MB, and 300 MB held barely two runs of history before silently evicting the rest. Also: an extension storing hundreds of megabytes should declare `unlimitedStorage` in the manifest — without it the browser treats its IndexedDB as best-effort and may discard it wholesale under disk pressure. That is a bigger exposure than any cap you choose. --- ## 6. The circuit breaker A recorder that is failing should say so and stop, not keep running and produce an empty file an hour later. Three capture runs were lost before anyone noticed, because the failure was quiet: the classifier matched 54 responses, the body reader returned `null` for every one, and the UI cheerfully showed a rising observation count. Nothing was on fire. Nothing was captured either. A pure function over the live counters, with generous thresholds: ```js matchedNeverForwarded: 20 // matched this many, none reached storage bodyReadFailures: 20 // unreadable bodies past the point of bad luck relayErrors: 25 // failed sends to the worker (orphaned content script) ``` Each threshold is a shape of failure that actually happened. Keep them generous: **a breaker that trips on noise gets ignored, and an ignored breaker is worse than none.** --- ## 7. The boundary that fails only when the data gets big The most instructive bug in the project, and the one most likely to bite anyone copying this architecture. The download page asked the service worker for an export's bytes: ```js stored = await send({ type: 'export-get', seq }); // over chrome.runtime.sendMessage const blob = new Blob([stored.json], { type: 'application/json' }); ``` This worked for months. Then a run's export reached **113 MB**, which `chrome.runtime.sendMessage` cannot deliver. The callback fired with `undefined`. And then four individually reasonable things combined into a corrupt file: 1. A helper normalised the reply: `resolve(res || {})` — turning *"no answer"* into *"an answer with nothing in it"*. 2. So the guard `if (!stored)` never fired: `{}` is truthy. 3. `stored.json` was `undefined`, and **`new Blob([undefined])` does not throw** — it stringifies, producing a file containing the nine characters `undefined`. 4. `stored.filename` was `undefined` too, so the file was called `undefined.json`. A delivery failure became a file. The smaller exports from the same run (5.3 MB, 4.6 MB) downloaded perfectly, which made it look like a problem with one file rather than with its size. ### The fix is to remove the boundary, not widen it Extension pages run on the extension's own origin, so **they can open the same IndexedDB the worker writes to**. The bytes never need to cross a message channel at all: ```js import { createDb } from '../db.js'; const db = createDb(); const stored = await db.getExport(seq); // no messaging, no size limit ``` The worker is still asked for *metadata* — filenames, sizes, which files belong to which run — because that is small and because it owns that bookkeeping. The boundary that remains is one that cannot fail on size. ### The three rules that fall out of it - **`res || {}` is a lie.** "No reply" and "an empty reply" are different events. Return `null` for the first and check for it. - **Missing contents are `null`, never `''`.** An empty string writes out a valid 0-byte file — the same bug in better clothes. - **Never construct a file from a value you have not checked.** `new Blob([x])` accepts anything and will serialise your bug into a download. Generalise: **any channel with a size limit is a landmine on a path whose data grows.** Message passing, URL parameters, `chrome.storage.local` (10 MB without `unlimitedStorage`), a data URL. If bulk data crosses one, either move the boundary or make the failure loud. --- ## 8. Build provenance Files on disk were updated twice; the browser kept running the build it loaded. Captures came back as if the new code did not exist — indistinguishable from a quiet week on the target. Ten days. **A browser extension does not reload when its files change.** Someone must press Reload on the extension card. Documentation of this is necessary and insufficient; build it into the tool: ```js async function buildInfo() { const version = chrome.runtime.getManifest().version; const st = await getState(); // Not "when did the worker wake" — service workers restart constantly. This is when the // CURRENT version first started running. if (st.buildVersion === version && st.buildLoadedAt) { return { version, loadedAt: st.buildLoadedAt }; } const loadedAt = new Date().toISOString(); await setState({ buildVersion: version, buildLoadedAt: loadedAt }); return { version, loadedAt }; } ``` Then: - **Stamp it into every artefact the tool produces**, and bump the file format version when you do. Diagnosing the original incident meant inferring a version from the wording of an internal reason string. It worked and should never have been necessary. - **Show it in the UI, on every surface**, and colour it when the loaded build is more than a few days old. Put it on the surface the human opens *first* — for us the badge shipped to the side panel and the toolbar popup went on saying nothing for two versions, which is exactly the wrong way round. This one field later prevented a repeat: a run's export said `version: "0.13.0"` when the fix being tested had shipped in `0.13.1`, which is how a wrong conclusion was caught. --- ## 9. UI surfaces Manifest V3 gives you two, and they are for different things: - **Side panel** — docks beside the page, survives navigation, stays open while browsing. This is where the real UI lives: live counts, the debug view, the captured data. - **Toolbar popup** — closes the moment the user clicks the page. A launcher, not an application. Keep it to state, a few numbers, and a button that opens the panel. Anything long-running (building an export, zipping, downloading) belongs in **a full tab**, not the popup: a popup closing revokes its blob URLs and can cancel a download mid-flight. Content scripts only attach on navigation, so open tabs are stale after an install. Inject into them programmatically *and* show a banner when nothing is hooked — injection after the fact misses the initial page load, so a tab reload is still better and the UI should say so. --- ## 10. Exports: three files, one run Every capture run produces three artefacts, and the split is a privacy design as much as a technical one: | file | contains | shareable | |---|---|---| | `full` | verbatim bodies, real identifiers | **no** — private store only | | `skeleton` | structure only: keys, nesting, array shapes, enum values, id *types* | yes | | `debug` | observation log, resource inventory, health counters. No bodies | yes | The **skeleton** is worth building properly. It reduces a payload to its shape, replacing values with placeholders that preserve type and length while keeping everything a parser author needs. Values that are taxonomy rather than data — enum constants, ISO dates, identifier *types* — are kept verbatim, because they are the schema. ``` "7498385662153363456" → "" "Some Person" → "" "REACTED_TO_UPDATE" → "REACTED_TO_UPDATE" (enum: taxonomy, not data) ``` Ship a **redaction verifier** alongside it that takes the real values from the full export and asserts none survive in the skeleton — including as object *keys*, which is the case everyone forgets. Ours counts what it checked, so "0 leaks" means something rather than "we ran zero checks". Two hard-won details: - **A run is the unit people want**, not a file. Offer the whole run as one zip; three separate downloads is three chances to lose one. (We wrote a small zip builder — CRC32 and raw deflate — rather than take a dependency, and tested that entries round-trip byte for byte.) - **Store exports so they survive a reset.** Finishing a run archives it and clears the captured data; the archive stays downloadable. This is what lets a human clear state without a decision about whether they are about to lose something. --- ## 11. What the test suite pins Zero dependencies, `node:assert`, auto-discovered files, 176 tests. It runs the pure modules — classification, skeletonisation, the breaker, the zip builder, the parsers — plus the storage layer against `fake-indexeddb`. Two structural rules: - **Auto-discover test files and fail loudly on zero.** A hand-maintained import list let a new test file sit unimported for two versions, passing vacuously. "No tests found" and "all tests passed" must never look the same. - **The db suite only runs when its optional dependency is installed.** Print `SKIPPED` loudly when it is not — a suite that silently drops 11 tests when a dependency is missing will eventually be the reason a bug ships. For anything with a DOM or a real browser boundary, drive a real browser. The messaging bug above was invisible to unit tests by construction; it took a headless browser, a real IndexedDB, and a genuinely large blob to prove the fix.