In the course of working on kenn-io/agentsview#1679, I came across a gap in the createMarkdownRenderer options.
Right now a consumer can add new markdown syntax but not override marked's built-in tokenizers, so when a consumer (like AgentsView) needs the parser to leave a region alone and render it verbatim, the options offer no way to do so.
The result is mangled output: content that should appear as written, like a diff pasted inside a wrapper in the case of the issue the PR fixes, gets reinterpreted as markdown and -/+ get replaced by bullets. A consumer that hits this has to fork the renderer to work around it, which defeats the point of having a shared renderer.
The AgentsView PR doesn't depend on this change since it forks the renderer. It should land first since it's spec-by-example, showing which overrides must be injected. If we designed this first, we'd be guessing what to expose and missing the this-binding requirement.
Details
Internally the call is:
if (options.extensions?.length) marked.use({ extensions: options.extensions });
marked.use({ renderer: codeRenderer });Nothing forwards marked.use({ tokenizer: {...} }).
agentsview renders agent transcripts that contain unknown XML-style prompt tags with diffs and code inside them:
<current_file_diff>
diff --git a/example.py b/example.py
- # FIXME: replace this
+ new_value
</current_file_diff>
marked tokenizes the content inside those tags as markdown, so the # line becomes an <h1> and the - / + lines become list items. We want to detect a complete unknown block and render it verbatim, while keeping normal markdown before and after it.
A block extension cannot do this. By the time an extension runs, marked's built-in paragraph and html tokenizers have already claimed the surrounding lines, so the block boundary lands in the wrong place. Fixing it requires overriding those built-in tokenizers so they stop at the start of an unknown block:
marked.use({
tokenizer: {
paragraph(src) { /* yield at an unknown-XML boundary */ },
html(src) { /* same */ },
},
});Because the options do not forward a consumer tokenizer, we maintain a forked marked plus sanitize pipeline instead of building on createMarkdownRenderer. The unknown-XML behavior stays in AgentsView; kit-ui stays generic. Sanitization still runs on the output afterward, so the security model does not change. A consumer tokenizer only changes how source text splits into tokens, it cannot get past DOMPurify.
In the course of working on kenn-io/agentsview#1679, I came across a gap in the
createMarkdownRendereroptions.Right now a consumer can add new markdown syntax but not override marked's built-in tokenizers, so when a consumer (like AgentsView) needs the parser to leave a region alone and render it verbatim, the options offer no way to do so.
The result is mangled output: content that should appear as written, like a diff pasted inside a wrapper in the case of the issue the PR fixes, gets reinterpreted as markdown and
-/+get replaced by bullets. A consumer that hits this has to fork the renderer to work around it, which defeats the point of having a shared renderer.The AgentsView PR doesn't depend on this change since it forks the renderer. It should land first since it's spec-by-example, showing which overrides must be injected. If we designed this first, we'd be guessing what to expose and missing the
this-binding requirement.Details
Internally the call is:
Nothing forwards
marked.use({ tokenizer: {...} }).agentsview renders agent transcripts that contain unknown XML-style prompt tags with diffs and code inside them:
marked tokenizes the content inside those tags as markdown, so the
#line becomes an<h1>and the-/+lines become list items. We want to detect a complete unknown block and render it verbatim, while keeping normal markdown before and after it.A block extension cannot do this. By the time an extension runs, marked's built-in
paragraphandhtmltokenizers have already claimed the surrounding lines, so the block boundary lands in the wrong place. Fixing it requires overriding those built-in tokenizers so they stop at the start of an unknown block:Because the options do not forward a consumer tokenizer, we maintain a forked marked plus sanitize pipeline instead of building on
createMarkdownRenderer. The unknown-XML behavior stays in AgentsView; kit-ui stays generic. Sanitization still runs on the output afterward, so the security model does not change. A consumer tokenizer only changes how source text splits into tokens, it cannot get past DOMPurify.