Streamability Contract

The rules every block-level plugin must satisfy to be "streamable."

Streaming is Generative DOM's reason to exist. Any plugin whose output diverges between streamed input and a single batch render is, by the project's product definition, broken. This page is the contract that makes streamability a hard, test-enforced invariant rather than a hope.

The four contracts

ID	Name	Verified by	Required of
SC-1	Chunk-boundary invariance	parity test (jsdom)	every plugin
SC-2	Mid-stream render invariance	parity test (jsdom)	every plugin
SC-3	Graceful degrade for unknown input	code review + lint	matchers w/ allow-lists
SC-4	No innerHTML in render	security audit + lint	every plugin

SC-1 — chunk-boundary invariance

For any markdown source S and any splitting S = c₁ ⌢ c₂ ⌢ … ⌢ cₙ, the rendered DOM after push(c₁); push(c₂); …; push(cₙ); flush() must equal the DOM after push(S); flush().

This is the easy contract. The buffer accumulates chunks across boundaries, so plugins that wait for \n before emitting tokens automatically satisfy SC-1. Plugins that commit on partial input — paragraph fallback, indented code, custom-element matchers — must use either a null return (no consumption) or a PartialMatchSignal (ADR-007) to tell the tokenizer "keep buffering, don't try downstream plugins yet."

SC-2 — mid-stream render invariance

SC-1, plus: the rendered DOM must remain identical when the scheduler ticks between every chunk (i.e. each chunk triggers a real render() call with isFinal=false) instead of only at the final flush.

This is the hard contract. Real LLM streams trigger ~60 mid-stream renders per second via the rAF scheduler. After each render the buffer cursor advances past whatever the tokenizer consumed, so a plugin can't extend a previously- emitted token with more bytes from the same buffer slice — those bytes are gone.

Plugins that need to "keep growing" a previously-emitted block (lists adding items, tables adding rows, blockquotes adding lines) satisfy SC-2 via the continuation pattern:

1. Keep closure state across matchBlock calls describing the open block (e.g. OpenList, OpenTable, OpenQuote).
2. When the buffer's next chunk could extend the open block, emit a token whose type ends with -continuation (e.g. list-continuation).
3. The continuation token's render() walks back through the container's children, finds the previously-emitted DOM node owned by this plugin, and appends new children to it in place. Returns an empty Text node as placeholder so the token→DOM index map stays 1:1.
4. When a chunk arrives that's definitively NOT a continuation (different indentation, blank line followed by non-quote content, separator-row without preceding header), clear the closure state.

The differ recognises the -continuation suffix and always appends such tokens — it never replaces a prior pending token with a continuation (ADR-R8-03). This is what makes the pattern composable: list / table / quote continuations stack independently in prevTokens.

Re-entry guard

render() calls ctx.renderBlock(...) for nested content. The recursive call invokes the same plugin's matchBlock again with isFinal=true. Without care, that recursive call mutates the streaming openList / openTable / openQuote state and corrupts the outer DOM. Plugins with state must increment a renderDepth counter on entry to render() and skip both reading and writing the closure state when renderDepth > 0 (ADR-R8-08).

SC-3 — graceful degrade for unknown input

A plugin that opts in to handling some prefix (a tag name, a syntax marker, a fence character) must, on rejection of an instance of that prefix, leave the input structurally unchanged. No half-eaten characters. No cursor- advance side effects.

Today's custom-elements plugin already satisfies SC-3 — it returns null for non-allow-listed tags and the bytes pass to downstream plugins. SC-3 is documented here so plugin authors don't accidentally introduce silent consumption (e.g. "I'll eat just the opening tag and let the close fall through" — a bug that would orphan a closing tag as raw text).

Content-bearing custom elements (<tag>...</tag>) have an additional SC-3 concern during streaming: when the opening tag arrives in one chunk but the closing tag hasn't arrived yet, returning bare null lets markdown-base consume the opening tag as paragraph text. Plugin authors must return PartialMatchSignal ({ status: 'partial' }) in this case to hold the position. The custom-elements plugin implements this pattern — see its matchBlock for the reference implementation.

A separate UX option exists for chat UIs that want non-allow-listed tags hidden rather than leaked as text: customElements({ fallbackBehavior: 'escape' | 'skip-block' }). The default is 'reject' (today's behaviour).

SC-4 — no innerHTML in render

Plugin render() must use ctx.createElement / ctx.createText / DOM API only. Never innerHTML, insertAdjacentHTML, eval, Function(), or setTimeout(string).

Already part of Generative DOM's security model. Restated here because graceful- degrade plugins might be tempted to dump raw HTML when they reject input. Don't.

How a plugin author satisfies the contract

Stateless plugins (paragraph, heading, code fence, link, inline)

Already streamable as long as matchBlock returns null (or PartialMatchSignal) when the buffer's last line is incomplete. The buffer will accumulate the rest. No closure state needed.

Important: If your plugin matches HTML-like content-bearing elements (<tag>...</tag>), a bare null when only the opening tag is present lets markdown-base consume it as paragraph text. You MUST return PartialMatchSignal in this case. See the custom-elements plugin's matchBlock for the reference pattern — it returns { status: 'partial', minCharsNeeded: closeTag.length } when the opening tag is found but the closing tag hasn't arrived.

Stateful plugins (list, table, blockquote, anything that grows)

Follow the canonical template — read it in packages/plugins/markdown-table/src/plugin.ts:

1. Declare a closure-scoped openX: OpenX | null = null.
2. Declare a WeakMap<Node, HTMLElement[]> to track DOM children appended by each continuation (cleanup hook target).
3. Declare a let renderDepth = 0 re-entry counter.
4. In matchBlock:
   • If openX !== null && renderDepth === 0: try matchContinuation. On full match return the continuation token. On partial signal return it. On null, clear openX (the block has ended) and continue.
   • Otherwise try the primary matcher; on success, set openX.
5. In render:
   • Wrap the body in try { ... } finally { renderDepth--; }. Increment renderDepth first.
   • For the primary token type, render normally.
   • For the -continuation token type, walk back to the prior DOM node via findLastX(ctx.container), append new children, track them in the WeakMap, return an empty Text placeholder.
6. In cleanup:
   • Remove tracked children for the placeholder.

The markdown-table, markdown-list, and markdown-quote plugins are reference implementations.

Verifying the contract

Every plugin should ship a parity test:

// packages/plugins/markdown-X/test/streamability.test.ts
import { describe } from "vitest";
import { runParityChecks } from "@generative-dom/tests/streamability";
import { markdownX } from "../src/plugin.js";
import { markdownBase } from "@generative-dom/plugin-markdown-base";
import { markdownInline } from "@generative-dom/plugin-markdown-inline";

describe("markdown-X streamability", () => {
  runParityChecks({
    plugins: () => [markdownX(), markdownInline(), markdownBase()],
    fixtures: {
      basic: "one example of the syntax\n",
      multi_block: "...\n\n...\n",
      nested: "nested case\n",
    },
  });
});

The harness (packages/tests/src/streaming/streamability.ts) registers Vitest describe / it blocks for SC-1 and SC-2 across seven splitters (byChar, byWord, byLine, byChunk(3), byChunk(13), byRandom(42), byRandom(7)) and asserts byte-identical innerHTML. Run with pnpm test.

Debug API for streaming bugs

When a streaming bug shows up in production, capture the input that caused it via the debug log (ADR-R8-07):

const md = new GenerativeDom({ container, plugins, debug: true });

// ... user reports a misrendered DOM ...

console.log("Reproduce with:");
console.log(JSON.stringify(md.getReceivedChunks()));
// or simpler:
console.log(JSON.stringify(md.getReceivedText()));

Replay the chunk array through the parity harness to bisect, then add the captured input to the plugin's fixture set so the regression is locked in. debug: true is off by default — the log grows unbounded for the lifetime of the instance.

Structured pipeline logging

When you need to know why a particular matcher won — or which plugin held the buffer open with a PartialMatchSignal — enable the structured log config:

const md = new GenerativeDom({
  container,
  plugins,
  log: { level: "debug", sink: "buffer", bufferSize: 1000 },
});

md.push("# Heading\n\nBody text.");
md.flush();

for (const event of md.getLogs()) {
  // phase: 'match-block' | 'tokenize' | 'diff' | 'render' | ...
  // type:  'matcher-tried' | 'matcher-matched' | 'token-created' | ...
  console.log(event.phase, event.type, event);
}

The structured log answers questions the chunk-capture debug log cannot: which matchers were tried at every position, which one was fast-rejected by its matchDescriptor, when a token transitioned from pending to complete, and which diff operations the renderer applied. See the Logging guide for the full event taxonomy and recipes.

ADR references

• ADR-R8-01 — Differ append-only merge: continuation tokens always append (packages/core/src/render-pipeline.ts mergeAndBuildOps).
• ADR-R8-02 — Continuation-first matching when stateful block is open (table / list / quote matchBlock order).
• ADR-R8-03 — *-continuation type-suffix convention; differ recognises it and never replaces with it.
• ADR-R8-04 — Continuation placeholders are empty Text nodes (not styled <span>s) so they're invisible in innerHTML.
• ADR-R8-05 — Object pool no longer re-adds class="" on scrub.
• ADR-R8-06 — Pending-replace removed from mergeAndBuildOps; the legacy path conflated buffer-cursor advance with pending-token continuation.
• ADR-R8-07 — GenerativeDomOptions.debug enables getReceivedText() / getReceivedChunks() for capturing exact streamed input.
• ADR-R8-08 — renderDepth re-entry counter on stateful plugins so recursive ctx.renderBlock calls don't corrupt closure state.

Pre-existing ADRs that this contract subsumes

• ADR-007 — PartialMatchSignal: how a plugin says "I might match, give me more bytes."
• ADR-014 — Token status pending / complete. (The streaming meaning of pending was clarified by ADR-R8-06; pending no longer causes cross-block replacement.)
• ADR-016 — Sub-token patching: text-content append for paragraph-style growth. Still active for setext promotion and similar in-place-grow cases where new token's raw is a strict prefix-extension of the old token's raw and both come from the same plugin.
• ADR-024 — finalize() hook for plugins that need to flush an unclosed block at end-of-stream.

Streamability matrix (current state)

Plugin	SC-1	SC-2	SC-3	Notes
markdown-base	✅	✅	n/a	Stateless paragraph + HR fallback.
markdown-inline	✅	✅	n/a	Stateless inline pass.
markdown-heading	✅	✅	n/a	Single-line block.
markdown-code	✅	✅	n/a	Open-fence rejection until close.
markdown-list	✅	✅	n/a	Continuation pattern (ADR-R8-02..-04).
markdown-quote	✅	✅	n/a	Continuation pattern (re-render inner).
markdown-table	✅	✅	n/a	Continuation pattern (reference impl).
markdown-link	✅	✅	n/a	Inline only.
custom-elements	✅	✅	✅	Allow-list rejects without consumption.
highlight, events, interactive	tbd	tbd	tbd	Audit pending — use harness.

Verified by packages/tests/src/streaming/parity.test.ts.