chrome-extensions.sgit.ai / guide 02 / architecture

The extension

Manifest V3 architecture for a browser extension that observes a web application's own network traffic and keeps it, without ever originating a request. Everything here is generic. Where the original project named a specific site or endpoint, this says "the target" and "the API".

1. The invariants

Write these down before any code, because every later decision refers back to them. Ours:

They are not just ethics — they are what makes the tool safe to leave running in a real session on a real account. A recorder that alters a response can break the page it is watching and you will spend a day thinking the target changed.

2. Architecture

Four pieces, each in a different execution context, which is the part that trips people up.

┌─ MAIN world (the page's own JS context) ─────────────────────────┐
│  page-interceptor.js                                             │
│    wraps window.fetch and XMLHttpRequest                         │
│    reads bodies from a CLONE, never the original                 │
│    forwards via window.postMessage                               │
└──────────────────────────┬───────────────────────────────────────┘
                           │ postMessage (the only way across the world boundary)
┌──────────────────────────▼───────────────────────────────────────┐
│  ISOLATED world                                                  │
│  content-script.js                                               │
│    validates the message, relays it on with chrome.runtime       │
│    pushes config (capture mode) the other way                    │
└──────────────────────────┬───────────────────────────────────────┘
                           │ chrome.runtime.sendMessage  (SIZE-LIMITED — see §7)
┌──────────────────────────▼───────────────────────────────────────┐
│  service worker (background.js, type: module)                    │
│    classification, storage, export building, health              │
│    IndexedDB lives here                                          │
└──────────────────────────┬───────────────────────────────────────┘
                           │
        ┌──────────────────┴──────────────────┐
        │  side panel        │  toolbar popup │   extension pages: same origin as
        │  (the real UI)     │  (a launcher)  │   the worker, so they can open the
        └────────────────────┴────────────────┘   same IndexedDB directly

Why MAIN world at all

A content script in the ISOLATED world gets its own window. Patching fetch there patches a fetch the page never calls. To see the page's traffic you must run in the page's own context, which manifest V3 supports declaratively:

{
  "matches": ["*://target.example/*"],
  "js": ["src/detect.js", "src/page-interceptor.js"],
  "run_at": "document_start",
  "world": "MAIN",
  "all_frames": false
}

document_start matters: the app's first API calls happen early, and a hook installed at document_idle misses the page load — often the richest payloads in the session.

The MAIN-world script has no access to chrome.*. It can only shout into window.postMessage and hope somebody is listening. That somebody is a second declaration of the same content script in the ISOLATED world, which does have chrome.runtime.

The double-injection trap

The worker also injects into tabs that were already open when the extension loaded (otherwise a tab that has not navigated since install is invisible). Without a guard, a tab that gets both the declarative injection and the programmatic one wraps fetch twice and reports every response twice.

const CHANNEL = '__TOOL_NAME__';
if (window[CHANNEL]) return;          // bail before installing anything
window[CHANNEL] = counters;           // claim the flag BEFORE the hooks go in,
                                      // so a racing second injection loses cleanly

Claim the flag first. If you claim it after installing the hooks, two injections racing can both pass the check.

Expose your counters on that global

window.__TOOL_NAME__.counters

The single most useful debugging affordance in the project. The human can open devtools on the real page and read fetchSeen, matched, forwarded, bodyReadFailures without any extension UI, and paste the numbers to the agent. It costs three lines.

3. Reading a body without disturbing it

The rule is that the page must be unaffected. For fetch:

const orig = window.fetch;
window.fetch = function (...args) {
  const p = orig.apply(this, args);
  p.then((res) => {
    try {
      // .clone() gives a second readable stream. Reading the original would consume it
      // and the page would get an empty body.
      res.clone().text().then(handleBody, noteFailure);
    } catch (_) { noteFailure(); }
  }, () => {});
  return p;                            // the page gets the untouched promise
};

For XMLHttpRequest you hook load and read responseText — with one trap that cost three capture runs: if the app sets responseType to arraybuffer or blob, responseText throws. The symptom is a rising "matched" count and a zero "captured" count, which looks like success from a distance. Handle every responseType, and count the failures.

Two more:

Then revisit the cap when the target changes. Ours was 1 MiB, tuned when payloads were dense JSON. The target moved to a server-driven-UI format that ships the interface description alongside the data — roughly two orders of magnitude less dense — and 1 MiB started cutting the two most valuable responses in every run. A cap tuned for a dense format silently guts a sparse one. Raising it to 4 MiB then created a different problem two layers away (§7), which is the honest shape of this kind of work.

4. Classification: capture modes, not a filter

A single "is this interesting" predicate is a trap, because the answer changes as you learn. Ours is a ladder of named modes, chosen in the UI:

modetakes
engagementa narrow allow-list of endpoints known to matter
wideeverything on the API paths except an explicit noise list ← the default
jsonanything that parses as JSON, whatever it claims to be
alleverything

wide is the one to default to, and the reasoning generalises: an allow-list fails silently when the target adds an endpoint; a deny-list fails loudly when it adds noise. On a real session the narrow list caught 14 responses, wide caught 33, and all caught 181. The 19 the narrow list missed are exactly the ones you would never have known to ask for.

Two lists, and they are not the same kind of thing:

When the target introduced a second API stack, the deny list had to be re-derived for the new naming convention — the floor is per transport, and a rule written for one stack does not protect the other. That is worth an explicit test: iterate every mode and assert the denied surface stays denied in all of them.

5. Storage: split metadata from bulk, and cap everything

IndexedDB, with the object stores split by size:

records       one row per response: seq, ts, url, pageUrl, status, contentType,
              bodyLength, truncated        ← small, listed constantly
bodies        seq → the body text          ← large, fetched one at a time
observations  everything seen, matched or not, with the reason it was refused
resources     performance-timeline entries: what the page fetched at all
exports       generated file metadata      ← small, listed by the UI
exportBlobs   seq → the file contents      ← large

The split is the point. Listing 500 records to render a table must not load 100 MB of bodies. Every store that grows has a cap and oldest-first eviction.

The observations store deserves special mention: it records everything the interceptor saw, including what it refused, and why. That log is what diagnoses the tool rather than the data. Twice it settled a question nothing else could: once proving the browser was running an old build (the refusal reason string was one the new code cannot emit), and once letting a later version be replayed against an earlier run's traffic to measure what it would have captured.

Sizing caps: write down what a "run" costs today, and expect it to move by an order of magnitude. Our export caps were 20 files / 300 MB, set when a run was 16 MB. Within three weeks a run was 113 MB, and 300 MB held barely two runs of history before silently evicting the rest. Also: an extension storing hundreds of megabytes should declare unlimitedStorage in the manifest — without it the browser treats its IndexedDB as best-effort and may discard it wholesale under disk pressure. That is a bigger exposure than any cap you choose.

6. The circuit breaker

A recorder that is failing should say so and stop, not keep running and produce an empty file an hour later.

Three capture runs were lost before anyone noticed, because the failure was quiet: the classifier matched 54 responses, the body reader returned null for every one, and the UI cheerfully showed a rising observation count. Nothing was on fire. Nothing was captured either.

A pure function over the live counters, with generous thresholds:

matchedNeverForwarded: 20   // matched this many, none reached storage
bodyReadFailures:      20   // unreadable bodies past the point of bad luck
relayErrors:           25   // failed sends to the worker (orphaned content script)

Each threshold is a shape of failure that actually happened. Keep them generous: a breaker that trips on noise gets ignored, and an ignored breaker is worse than none.

7. The boundary that fails only when the data gets big

The most instructive bug in the project, and the one most likely to bite anyone copying this architecture.

The download page asked the service worker for an export's bytes:

stored = await send({ type: 'export-get', seq });          // over chrome.runtime.sendMessage
const blob = new Blob([stored.json], { type: 'application/json' });

This worked for months. Then a run's export reached 113 MB, which chrome.runtime.sendMessage cannot deliver. The callback fired with undefined. And then four individually reasonable things combined into a corrupt file:

  1. A helper normalised the reply: resolve(res || {}) — turning "no answer" into "an answer with nothing in it".
  2. So the guard if (!stored) never fired: {} is truthy.
  3. stored.json was undefined, and new Blob([undefined]) does not throw — it stringifies, producing a file containing the nine characters undefined.
  4. stored.filename was undefined too, so the file was called undefined.json.

A delivery failure became a file. The smaller exports from the same run (5.3 MB, 4.6 MB) downloaded perfectly, which made it look like a problem with one file rather than with its size.

The fix is to remove the boundary, not widen it

Extension pages run on the extension's own origin, so they can open the same IndexedDB the worker writes to. The bytes never need to cross a message channel at all:

import { createDb } from '../db.js';
const db = createDb();
const stored = await db.getExport(seq);      // no messaging, no size limit

The worker is still asked for metadata — filenames, sizes, which files belong to which run — because that is small and because it owns that bookkeeping. The boundary that remains is one that cannot fail on size.

The three rules that fall out of it

Generalise: any channel with a size limit is a landmine on a path whose data grows. Message passing, URL parameters, chrome.storage.local (10 MB without unlimitedStorage), a data URL. If bulk data crosses one, either move the boundary or make the failure loud.

8. Build provenance

Files on disk were updated twice; the browser kept running the build it loaded. Captures came back as if the new code did not exist — indistinguishable from a quiet week on the target. Ten days.

A browser extension does not reload when its files change. Someone must press Reload on the extension card. Documentation of this is necessary and insufficient; build it into the tool:

async function buildInfo() {
  const version = chrome.runtime.getManifest().version;
  const st = await getState();
  // Not "when did the worker wake" — service workers restart constantly. This is when the
  // CURRENT version first started running.
  if (st.buildVersion === version && st.buildLoadedAt) {
    return { version, loadedAt: st.buildLoadedAt };
  }
  const loadedAt = new Date().toISOString();
  await setState({ buildVersion: version, buildLoadedAt: loadedAt });
  return { version, loadedAt };
}

Then:

This one field later prevented a repeat: a run's export said version: "0.13.0" when the fix being tested had shipped in 0.13.1, which is how a wrong conclusion was caught.

9. UI surfaces

Manifest V3 gives you two, and they are for different things:

Anything long-running (building an export, zipping, downloading) belongs in a full tab, not the popup: a popup closing revokes its blob URLs and can cancel a download mid-flight.

Content scripts only attach on navigation, so open tabs are stale after an install. Inject into them programmatically and show a banner when nothing is hooked — injection after the fact misses the initial page load, so a tab reload is still better and the UI should say so.

10. Exports: three files, one run

Every capture run produces three artefacts, and the split is a privacy design as much as a technical one:

filecontainsshareable
fullverbatim bodies, real identifiersno — private store only
skeletonstructure only: keys, nesting, array shapes, enum values, id typesyes
debugobservation log, resource inventory, health counters. No bodiesyes

The skeleton is worth building properly. It reduces a payload to its shape, replacing values with placeholders that preserve type and length while keeping everything a parser author needs. Values that are taxonomy rather than data — enum constants, ISO dates, identifier types — are kept verbatim, because they are the schema.

"7498385662153363456"  →  "<id:19>"
"Some Person"          →  "<v>"
"REACTED_TO_UPDATE"    →  "REACTED_TO_UPDATE"     (enum: taxonomy, not data)

Ship a redaction verifier alongside it that takes the real values from the full export and asserts none survive in the skeleton — including as object keys, which is the case everyone forgets. Ours counts what it checked, so "0 leaks" means something rather than "we ran zero checks".

Two hard-won details:

11. What the test suite pins

Zero dependencies, node:assert, auto-discovered files, 176 tests. It runs the pure modules — classification, skeletonisation, the breaker, the zip builder, the parsers — plus the storage layer against fake-indexeddb.

Two structural rules:

For anything with a DOM or a real browser boundary, drive a real browser. The messaging bug above was invisible to unit tests by construction; it took a headless browser, a real IndexedDB, and a genuinely large blob to prove the fix.

← The method Next: reading payloads →