walkTokens takes quadratic time because it copies the results array for every token #4120

Marked version: 18.0.14 (also on master)

Describe the bug
When any extension registers walkTokens, parse time grows quadratically with document size. Marked.walkTokens in src/Instance.ts collects the callback results with values = values.concat(...) for every token, and concat copies the whole array each time. On large documents, walking the tokens takes far longer than lexing and parsing combined.

To Reproduce

import { Marked } from 'marked';

const doc = (n) =>
  Array.from({ length: n }, (_, i) => `## Heading ${i}\n\nParagraph ${i} with **bold** and \`code\`.\n\n- a\n- b\n`).join('\n');

function time(marked, src) {
  for (let i = 0; i < 3; i++) marked.parse(src);
  const start = performance.now();
  for (let i = 0; i < 10; i++) marked.parse(src);
  return (performance.now() - start) / 10;
}

const plain = new Marked();
const withHook = new Marked({ walkTokens() {} });
for (const n of [500, 1000, 2000, 4000]) {
  const src = doc(n);
  console.log(n, time(plain, src).toFixed(1), time(withHook, src).toFixed(1));
}

Node 22.21.1, marked 18.0.14:

Doubling the document multiplies the time by about 4–5 when the hook is present.

We hit this in a live-preview editor that re-parses on every keystroke. Common extensions (marked-highlight, marked-alert, marked-footnote) register walkTokens. In the browser, for a 164 KB document, walkTokens took about 135 ms per parse, and it was the largest single cost per keystroke.

Expected behavior
Walking the tokens should take linear time. Appending in place fixes it: we swapped every values = values.concat(x) for a helper that pushes x (or each item of x when it's an array) onto values. For a 328 KB document, walkTokens dropped from about 607 ms to 6 ms per parse, and the returned values were the same (concat flattens one level, as the helper does). The values.concat in the combined walkTokens function built in use() could get the same change, although its arrays stay small.