Yahya/token healing

[yahya010]

Adaptive token healing (@timvieira's idea) to handle any dead ends whete a tokenization makes the next byte unreachable. The boundary is moved earlier in the current partial bytes, commit an EOT, rematerialize and generate.

It first does a precheck O(len(P)) trie lookups and 0 LM calls, to find a valid earlier boundary where EOT + replay + target byte are available, then a materialize so only <=1 LM call needed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yahya010]

making some changes for multi splits...

[yahya010]

O(|P|) per k (greedy scan), worst‑case O(|P|^2) across k — but |P| is small (bytes since last boundary), so this is negligible.
LM calls: 1 materialization for the initial P[:k] commit + 1 per internal split planned inside S, no extra calls during planning

[benlebrun]

@yahya010 Are you planning on fixing the code coverage?

[yahya010]

@benlebrun yeah, got distracted

[yahya010]

alright, is that enough? rest are verbose and such

[avyavkumar]

Hi, we are using your library in our work -- and it's amazing, many thanks! If possible, can you please let me know by when this feature can be merged? It's something that would be extremely useful for our work.

[yahya010]

Hey, thanks a lot! I need to clean and add tests for code coverage and should be good to be reviewed. It's ready to use as is from the branch if needed, but I'll get it finalized soon