Architecture

This page is a deep dive into Generative DOM's internal architecture. Understanding these internals is helpful for writing advanced plugins and debugging issues.

High-Level Pipeline

push(chunk)
    |
    v
+--------+     +-----------+     +--------+     +----------+     +----------+
| Buffer | --> | Tokenizer | --> | Parser | --> | Diff     | --> | Renderer |
+--------+     +-----------+     +--------+     +----------+     +----------+
    ^                                                                  |
    |                                                                  v
    +---- rAF + debounce scheduling -----------------------------------+
                                                                      DOM

Buffer (buffer.ts)

The buffer is a simple string with a cursor. push() appends to the string; the cursor tracks how far the tokenizer has successfully consumed.

Buffer: "# Hello\n\nThis is **bold**.\n"
                                ^
                              cursor (last successful parse position)

New push("More text"):
Buffer: "# Hello\n\nThis is **bold**.\nMore text"
                                ^
                              cursor (unchanged until next tokenize cycle)

When a token is matched, the cursor advances by consumed characters. Unmatched trailing content remains in the buffer for the next cycle.

Tokenizer (tokenizer.ts)

The tokenizer iterates through registered plugins in priority order at each position in the buffer starting from the cursor.

Position 0: Try plugin[0] (priority 40) -> null
            Try plugin[1] (priority 45) -> null
            Try plugin[2] (priority 50) -> null
            Try plugin[3] (priority 95) -> null
            Try plugin[4] (priority 100) -> MATCH! (heading)
            Advance cursor by match.consumed
            Produce Token { type: "heading", ... }

Position N: Try plugin[0] -> null
            Try plugin[1] -> null
            ...

If no plugin matches at a position and the remaining content looks incomplete, the tokenizer stops. If the content looks complete but nothing matches, markdown-base (the fallback at priority 300) wraps it as a paragraph.

Parser (parser.ts)

The parser takes the flat stream of tokens from the tokenizer and organizes them into an AST. The AST is a flat array of block tokens, where each block token may contain inline tokens as children.

// AST structure
[
  { type: "heading", raw: "# Hello\n", content: "Hello", meta: { level: 1 } },
  { type: "paragraph", raw: "This is **bold**.\n", children: [
    { type: "text", raw: "This is ", content: "This is " },
    { type: "bold", raw: "**bold**", content: "bold" },
    { type: "text", raw: ".\n", content: "." },
  ]},
]

Differ (differ.ts)

The differ compares the new AST against the previous AST to determine the minimal set of DOM operations needed. It compares tokens by their raw value:

• Unchanged token: Same raw at same position -- skip (no DOM update)
• Changed token: Different raw at same position -- re-render in place
• Added token: New position -- create and insert DOM node
• Removed token: Position no longer exists -- remove DOM node and release element to pool

This diffing ensures that unchanged content (like a heading at the top of a document) is never re-rendered even as new content streams in below it.

Renderer (renderer.ts)

The renderer takes diff operations and applies them to the DOM:

1. For each new or changed token, calls plugin.render(token, ctx) to get a DOM node
2. Appends, replaces, or removes nodes from the container
3. For removed tokens, calls plugin.cleanup(element) and releases the element to the pool

All DOM manipulation uses createElement, createTextNode, appendChild, replaceChild, and removeChild. Never innerHTML.

Scheduler (scheduler.ts)

The scheduler prevents excessive rendering:

1. push() sets a dirty flag
2. A requestAnimationFrame callback checks the dirty flag
3. If dirty and enough time has passed since the last render (controlled by debounceMs), it triggers a full tokenize -> parse -> diff -> render cycle
4. flush() triggers the cycle immediately, bypassing both rAF and debounce

push() -> dirty = true
push() -> dirty = true (still true, no-op)
push() -> dirty = true
rAF fires -> dirty? yes. debounce elapsed? yes -> RENDER -> dirty = false
push() -> dirty = true
flush() -> RENDER immediately -> dirty = false

Event System (event-system.ts)

Events are stored in a Map<string, Set<handler>>. When a plugin calls ctx.emit(event, data):

1. Core prepends the plugin name: "pluginName:event"
2. Core iterates all handlers for that namespaced event
3. Each handler is called synchronously with the data
4. If a handler throws, the error is logged and the next handler runs

Object Pool (object-pool.ts)

The pool is a Map<string, HTMLElement[]> keyed by tag name.

Acquire:

1. Check if the pool has elements for the requested tag
2. If yes, pop one, increment reused counter
3. If no, create via document.createElement, increment created counter
4. Increment active counter

Release:

1. Remove all attributes from the element
2. Remove all child nodes
3. Remove event listeners (via cloneNode(false) replacement or explicit tracking)
4. Reset className
5. If pool for this tag is under capacity (100), store the element
6. Decrement active counter, increment pooled counter

Drain:

1. Clear all arrays in the map
2. Reset pooled counter to 0

Source Files

File	Purpose
packages/core/src/generative-dom.ts	Main GenerativeDom class
packages/core/src/buffer.ts	Buffer and cursor management
packages/core/src/tokenizer.ts	Plugin dispatch and token production
packages/core/src/parser.ts	AST construction
packages/core/src/differ.ts	AST diffing
packages/core/src/renderer.ts	DOM manipulation
packages/core/src/scheduler.ts	rAF and debounce scheduling
packages/core/src/event-system.ts	Event pub/sub
packages/core/src/object-pool.ts	DOM element pool
packages/core/src/types.ts	All TypeScript interfaces