Reading payloads nobody documented

How to go from "here is 100 MB of a web app's network traffic" to a parser you can trust, without inventing facts along the way. Generic throughout: "the target", "the API", urn:<ns>:<type>:<id>.

1. Probe the envelope before the contents

The first question is not "where is the data" but "what shape is this response". Ours had two generations of envelope live at the same time, and an early conclusion drawn from one capture was wrong about the other:

// Form A — a normalised graph
{ "data": { … }, "included": [ { "$type": "…", "entityUrn": "urn:…", … } ] }

// Form B — entities inlined, tagged in place
{ "data": { "elements": [ { "_type": "…", "text": { "text": "…" } } ] } }

Then a third arrived that is not JSON at all (§5).

Write the shape survey before the parser. A short script over every stored body that reports: does it parse as JSON, which top-level keys exist, which $type/_type values appear and how often. Ours found 58 distinct type names in one run, which immediately answered a question that had been open for a week.

Corollary: do not over-generalise from one capture. "There is no included[] array" was true of the first capture and false of the format generally. Probe, don't assume.

2. Routing: let the response name itself

The obvious way to tell which query produced a response is the request URL. Ours had a query id in it — and three completely different result sets shared the same path and the same readable prefix, differing only by a hash that rotates on every deploy. Routing on it works until the next deploy.

The response named itself instead: the root field under data differed per query (…ByAuthorReactions vs …ByAuthorPosts) even when path and prefix did not. Route on something the response says about itself, not on something the request said about it. Request-side identifiers are the caller's opinion; response-side names are the server's.

3. Identifiers carry more than identity

Two things worth checking on any opaque id:

Timestamps. Many systems mint ids with an embedded creation time — the high bits are epoch milliseconds, the low bits a machine/sequence counter. id >> 22 on ours yielded a time. That is a derivation, not a field, so it fails closed:

export function urnTimestamp(urn) {
  const m = String(urn || '').match(/(?:activity|post|share):(\d{15,})/);
  if (!m) return null;
  const ms = Number(BigInt(m[1]) >> 22n);
  // Sanity floor and ceiling: the target did not exist before 2000, and this is not a
  // time machine. An id that dates outside those bounds is not an id of this kind.
  if (!Number.isFinite(ms) || ms < 946684800000 || ms > 4102444800000) return null;
  return ms;
}

This was the single highest-value discovery in the project: it recovers when something happened for events the target's own UI never displays and its official data export omits.

Fixed lengths. Member ids in our target are exactly 39 characters, which mattered because one place glued an id to a card name with no delimiter. Cutting at 39 is only safe because 381 of the 382 distinct ids across every capture were 39 characters — a population, not the one sample that had already burned us once (§7). Check before you cut, and return null rather than a truncated id: an identifier wrong by one character is a different person, or nobody.

Verify the derivation against something the system produced

Never trust a derived timestamp on plausibility. Ours was checked three ways, each against an artefact the target emitted independently:

Two independent clocks agreeing to within a network round trip is strong. One clock agreeing with itself is nothing.

Date the event, not the thing it points at

An early version read the wrong identifier and dated every "user reacted to this" event to the post's birthday rather than the reaction's. The payload had both: an outer identifier for the event and an inner one for the content. They can be months apart.

Whenever a payload wraps one entity in another, ask which one your timestamp is coming from. This recurred later in a different form: an id for a comment dates the comment, not the moment someone reacted to that comment.

Say how good each timestamp is

Where a fact could not be derived, it is dated by observation and labelled:

{ at: 1757000000000, atSource: 'urn' }        // derived from the event's own id, exact
{ at: 1757000000000, atSource: 'captured' }   // when we saw the response — a round trip late

One field, and consumers can filter on evidence quality instead of silently treating a round-trip-delayed observation as an event time. Any pipeline mixing derived and observed values needs this.

4. Read the markup, not the English

A header rendered as "Ada Example and 1 other reacted to your post" also carried an attribute span marking characters 0–11 as the actor's name, with a profile identifier attached.

// Read from the marked range the payload gives us. Parsing the sentence works until the
// string is localised, pluralised, or A/B tested.
export function actorNameFromHeadline(headline) {
  const text = readText(headline);
  const attrs = (headline && headline.attributesV2) || [];
  for (const a of attrs) {
    if (a.start === 0 && a.length > 1) return text.slice(0, a.length).trim() || null;
  }
  return null;                                  // no marked range → no name. Never a guess.
}

The same principle scaled up later: where a payload marks regions rather than characters, use the region names (§5). And the same rule stopped a bad date: a "Connected on September 1, 2026" string is a display string in one locale, so it is kept verbatim in one field and parsed into a plain date in another, only when it matches the English form exactly. A date we cannot read is null, never a guess.

5. When the payload is a document, not a graph

The target migrated to a server-driven-UI stack whose responses are not JSON documents but line-delimited numbered chunks:

1:I["module-ref",[],"default"]
0:["$","div",null,{"children":["$L4","$L9"]}]
4:["$","div",null,{"viewTrackingSpecs":{"viewName":"full-update"},"children":["$L5"]}]
5:["$","div",null,{"viewTrackingSpecs":{"viewName":"commentary"},"children":["$L6"]}]
6:["$","p",null,{"textProps":{"children":["I decommissioned this power strip."]}}]

The first attempt treated this as a bag of JSON objects and fed them to the existing entity walker. It found nothing, and the conclusion drawn — "this format carries no domain data" — was half right and cost three weeks.

They are not a bag. They are one tree, cut into numbered pieces, with "$L<id>" standing in for "the chunk with this id goes here". Rejoin them, walk in render order, and emit a flat event stream:

view  full-update           ← a card starts
urn   7500288780453597184   ← its id, sitting in a UI state key
view  commentary            ← the author's own words
text  "I decommissioned this power strip."
view  comment-body          ← somebody else's words; NOT the author's

Order is the join key. Nothing inside a text node says which item it belongs to. It is simply the text that comes after that item's header. Once you accept that, the same walker reads every list surface the app has.

Four details that each cost an hour:

• Not every chunk hangs off chunk 0. One surface's chunks did, another's did not. Walk every chunk in file order unless something has already reached it as a child.
• <br/> carries meaning. Paragraphs arrive as separate text nodes with no whitespace of their own; without emitting a newline for br the output reads "…frameworks.It is timing."
• Take text from exactly one place. Ours is textProps.children. Everything else in a props object is presentation — font sizes and class names will otherwise turn up in your prose.
• Section headings are structure wearing text's clothes. "About" is the name of a block, not something a person wrote. It was marked tagName: "h2", so headings became their own event type rather than being recognised by their words.

The region names are the schema

The author's own text and a comment on it rendered through the same component with the same typography. Nothing distinguished them except the framework's own analytics region name — commentary vs comment-body. Those names exist because the system needs them for its own telemetry, and they are far more stable than layout.

When two things look identical in a payload, look for the name the system gives the region containing them.

6. Fail closed, and count what you skip

// A card with no identifier yields nothing rather than being attached to the previous one.
// 2 of 14 in the first real run. Losing two is correct; mislabelling two is not.
return out.filter((p) => p.contentId && p.text.length > 0);

Rules that earned their place:

• An unrecognised body is counted in unparsed, keyed by endpoint. That tally is a to-do list: the family that keeps appearing there is the next parser to write.
• A person with no stable identifier is counted as anonymous and never merged with anyone. Names are a weak join key; merging on one invents a relationship, which is worse than a duplicate row.
• Reconstruct deterministically, never inferentially. Building a permalink from an id you were given is fine and should be commented as such: "building a URL from an id is deterministic; it is not the same thing as inferring a fact we were not given." Building one from an id of the wrong type is not fine — it produces a valid-looking URL pointing at nothing.

The gap in fail-closed

Returning null for anything unrecognised is right, but a null is indistinguishable from "this response was not relevant". Sixty percent of the events in one run went missing without a single line in unparsed.

Fail closed keeps bad data out; it does not tell you that you are losing good data. Where a response is on a path you expect to understand, count the misses separately from the skips.

7. Two mistakes to expect

One sample is an example, not a grammar

A pattern written against a single capture matched two of the next five cases:

urn:ns:comment:(urn:ns:event:…,…)   the one shape the pattern knew
urn:ns:comment:(event:…,…)          no namespace prefix — a spelling variant
urn:ns:comment:(post:…,…)           a different TYPE arriving through the same endpoint

Two different mistakes needing opposite fixes. The variant means loosen the pattern; the different type means branch, because loosening would have quietly produced wrong output. When the sample is a structured identifier with a type in it, read the type as data.

A completeness statistic can select against its own evidence

The worst error in the project. A conclusion — "this format carries no content" — rested on a sample of 70 payloads, "59 of which were whole rather than truncated". The 11 truncated ones were the five feed pagers and the three largest other bodies: every payload big enough to contain a feed. They were whole because they were small.

Worse, the run had been captured on a build that still had the old size cap, and its own export said so in a provenance field added two versions earlier for exactly this purpose. It went unread.

"59 of 70 were whole" and "the 11 that were not are the only ones that could have held the answer" are the same sentence about the same run, and only one is worth saying.

Whenever a cap, page size or sample boundary is in play, ask whether it selects for the thing being tested — and when a fix for it has shipped, check the build stamp of the run you are about to call the recapture.

8. Keep the evidence; derive the meaning

Two directories, one direction of travel:

captures/     verbatim exports. Never edited. Never published.
derived/      one small, stable file per run, regenerated from captures at will

The derived contract is deliberately dull: flat arrays of plain objects, no pointers to resolve, no $type, every fact carrying its provenance, null for missing, and a formatVersion at the top so a consumer can refuse a shape it does not know rather than reading it partially.

This is what makes being wrong cheap. Three times a conclusion was overturned and the fix was re-running the deriver over data already in hand. Once a parser added in week five found events sitting in a week-three capture that the week-three code had walked straight past — three post-publication events, recovered for free.

Version the derived format and refuse old ones loudly. A v1 file has no event log; silently merging one understates the dataset without saying so. Bumping the version forces a re-derive, which costs seconds and is always correct.

9. Report coverage, not just results

The single most important habit for a dataset built from observation: a consumer must be able to tell "no edge" from "we never looked".

PROFILE_REACTIONS   122 events   2026-08-09 → 2026-08-26
MEMBER_SHARES        74 events   2026-08-03 → 2026-08-21
CONNECTIONS           0 events   never captured

A surface absent from the output reads as "nothing happened". A surface listed with zero reads as "we never looked" — and for a recorder that only ever sees pages the human actually visited, the second is usually the true one. Say which.

The same applies to every derived collection. Ours ships 20 contact records against a network of 21,720, and the index file says so in its own note field: "only contacts seen while recording. This is a partial view, not an export." A partial dataset presented without that sentence is a misleading one.