Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After skimming through https://tc39.es/ecma262/#sec-unicode-format-control-characters, I realized tokenizing JavaScript is not that difficult. This PR doodles on such a tokenizer. It's largely AI-produced and probably wrong on at least few counts, but it seems to produce useful results for a few normative scripts I've fed it. I don't have a good use-case for it yet, other than maybe rewriting URLs in JavaScript snippets, so I'm posting this PR and forgetting about it for now.
cc @dmsnell @sirreal @brandonpayton in case it makes you think of some problem you've seen somewhere.