Streaming

Generative DOM is a stream-first markdown parser. This guide explains how streaming works, how chunks are handled, and how the buffer manages partial input.

How It Works

When you call md.push(chunk), the chunk is appended to an internal string buffer. The buffer is marked as dirty, which schedules a parse-and-render cycle on the next animation frame (subject to debounce timing).

The tokenizer starts from where it last left off (the cursor position) and attempts to match tokens from the new content. If it encounters ambiguous trailing content that might be incomplete syntax, it stops and waits for more input.

Chunk Handling

Generative DOM handles chunks that split markdown at any arbitrary point:

Split Point	Example	Behavior
Mid-syntax	**bol then d**	Waits for closing markers, then renders bold
Mid-escape	\ then *	Waits for next character, renders escaped asterisk
Mid-fence	``` at end	Waits for newline to confirm code fence
Mid-link	[text](ur then l)	Waits for closing ), then renders link
Mid-word	hel then lo	Accumulates into current text node

The key principle: when the tokenizer cannot determine whether remaining buffer content is complete, it stops and preserves the unconsumed portion.

Buffer Management

The buffer maintains a cursor that tracks the position of the last successfully parsed token. On each render cycle:

1. Tokenization starts from the cursor position
2. Plugins are tried in priority order at each position
3. When a plugin matches, the cursor advances by consumed characters
4. When no plugin matches and content might be incomplete, tokenization stops
5. The unconsumed tail of the buffer is preserved for the next cycle

Streaming Sources

Generative DOM works with any source that produces text incrementally.

Fetch with ReadableStream

const response = await fetch('/api/stream');
const reader = response.body!.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  md.push(decoder.decode(value, { stream: true }));
}
md.flush();

Server-Sent Events

const source = new EventSource('/api/events');
source.onmessage = (event) => {
  md.push(event.data);
};
source.onerror = () => {
  md.flush();
  source.close();
};

WebSocket

const ws = new WebSocket('wss://example.com/stream');
ws.onmessage = (event) => {
  md.push(event.data);
};
ws.onclose = () => {
  md.flush();
};

Manual Input

textarea.addEventListener('input', () => {
  md.reset();
  md.push(textarea.value);
  md.flush();
});

Debouncing

The debounceMs option controls the minimum interval between renders. The default is 16ms (approximately one frame at 60fps). During rapid push() calls, the scheduler batches updates so the DOM is not thrashed.

const md = new GenerativeDom({
  container: document.getElementById('output'),
  plugins: [markdownBase()],
  debounceMs: 50, // render at most every 50ms
});

Set debounceMs: 0 for immediate rendering on every animation frame. Higher values reduce CPU usage at the cost of perceived latency.

Flush

Call md.flush() to force an immediate render of all buffered content, bypassing the debounce timer. Use this:

• At the end of a stream to ensure the final content is displayed
• When you need the DOM to reflect the latest state synchronously

md.push('final chunk');
md.flush(); // render now, don't wait for rAF