Repository navigation
Conversation
`delimited_query_char` sliced the input at `pos + 1` before checking whether the character at `pos` opens a literal. When it is a multi-byte character, `pos + 1` is not a char boundary and the slice panics. Such a character is reached when a comment directly follows another comment in a block: the comment branch returned the `#` of the next comment as an ordinary character, so the rest of that comment was scanned character by character (and a quote in it was lexed as the start of a string literal). Only build the literal lexer in the branches that need it, and after skipping a comment, scan the following character with the same rules so consecutive comments and literals are skipped as a whole. Fixes vectordotdev#1989 Generated with Claude Opus 5.5. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac5c6cfe83
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Avoid recursing once per comment line in `delimited_query_char`, which could overflow the stack on long runs of comments in a block. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm
|
@pront Thanks for the review, the follow-up commit, and the merge! |
2 participants
Summary
Since #1946, the lexer panics with
start byte index N is not a char boundaryon a valid program where a comment directly follows another comment inside a block and the second one has a non-ASCII character (#1989).Root cause, in
delimited_query_char:self.input[pos + 1..]before checking whether the character atposopens a literal. When that character is multi-byte,pos + 1is not a char boundary. Before chore(lints): remove parser pedantic exceptions #1946 the slice happened only inside the literal branches.#of an immediately following comment as an ordinary character. The rest of that comment was then scanned one character at a time. This is also why a"in that second comment fails with an unterminated-string error.Fix:
",s',r'andt'branches.How did you test this PR?
src/parser/lex.rs. One covers the multi-byte character in a second consecutive comment, which panicked before this change. The other covers a quote in a second consecutive comment, which returned a lex error before.lib/tests/tests/issues/1989_multibyte_char_in_consecutive_comments.vrl, the program from the issue. It panics onmainand passes with this change.cargo test --lib --no-default-features --features compiler,stdlibpasses (1954 tests).cargo run -p vrl-tests --bin vrl-testspasses (960 passed).cargo fmtandcargo clippy --lib --tests -- -D warnings(compiler,stdlib features) are clean.This PR was mostly generated with Claude Opus 5.5. I reviewed and tested it.
Does this PR include user-facing changes?
our guidelines.
Checklist
run
dd-rust-license-tool writeand commit the changes. More details here.References
🤖 Generated with Claude Code
https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm