perf(utf8): ASCII wrapping via strict printable-only invariant #506
Conversation
simonklee
force-pushed
the
perf-utf8-ascii-invariant
branch
from
9c432af to
6d32b43
Compare
|
I am not happy with how tab width is handled currently. Drove me mad a while ago so I suppressed it. |
I wasn't sure it was the right approach in the PR either, it but I noticed the inconsistency between naming and implementation and aligned things. As far as I can see is if you have ascii with tabs you'll end in that unicode path anyway atm. Started thinking about how to make ascii+tabs fast as well, but didn't get too far on it yet. |
|
It currently uses manual loops to scan for tabs, could use the same approach as "find newlines" does? If this improves the performance significantly for now it might be good. |
The is ascii only checks isn't exactly what the name implies. It strictly enforces printable ASCII (32-126), explicitly excluding control characters like tabs (`\t`) and newlines. This provides a stronger guarantee than typical 7-bit ASCII checks: if `isAsciiOnly` is true, every byte is exactly 1 column wide. However, we ignored this and still running O(N) width loops even on the fast path. This patch deletes those loops entirely: 1. If it's guaranteed printable ASCII, the display width is identical to `text.len`. We don't need to iterate N bytes just to add 1 N times. 2. Since width maps 1:1 to byte index, the wrap position is simply `min(text.len, max_width)`. We don't need to scan the string to find where it overflows. The obvious risk here is tabs (byte 9), which are strictly ASCII but variable width. But since `isAsciiOnly` checks `val >= 32`, so it returns false for `\t`. This forces tabbed content into the slow Unicode path where `tab_width` is handled properly. I also considered if it was possible for FFI consumers to pass `isAsciiOnly=true` for strings with tabs or newlines. But since the public API doesn't expose `isAsciiOnly` directly, and instead derives it via `utf8.isAsciiOnly()`, which returns false for empty strings and control characters, this optimization is safe and transparent to external users.
simonklee
force-pushed
the
perf-utf8-ascii-invariant
branch
from
6d32b43 to
715d195
Compare
|
The ASCII-only fast paths now just return min(text.len, max_columns) and skip byte scans in findWrapPosByWidth*/findPosByWidth*, plus calculateTextWidth* returns text.len. Tabs already force the Unicode path since isASCIIOnly excludes control chars, so there isn't a tab scan left to optimize on the ASCII fast path. This change makes the ASCII fast path as fast as possible given the current design, so there is an improvement on the inner loops for that case. It also shows up as a "regression" in some of the tabbed benchmarks since I've now correctly re-wired them to always go through the Unicode path by setting ascii=false, like the actual logic dictates. I agree, if you want a faster ASCII+tabs path, we could add a follow-up that uses the same SIMD/memchr-style scan as findLineBreaks (vector compare for \t, scalar only on matching chunks) to avoid the full O(N) loop. There already is the findTabStops function, but it's not used anywhere currently. |
2 participants
The is ascii only checks isn't exactly what the name implies. It strictly enforces printable ASCII (32-126), explicitly excluding control characters like tabs (
\t) and newlines.This provides a stronger guarantee than typical 7-bit ASCII checks: if
isAsciiOnlyis true, every byte is exactly 1 column wide.However, we ignored this and still running O(N)
width loops even on the fast path. This patch deletes those loops entirely:
If it's guaranteed printable ASCII, the display width is identical to
text.len. We don't need to iterate N bytes just to add 1 N times.Since width maps 1:1 to byte index, the wrap position is simply
min(text.len, max_width). We don't need to scan the string to find where it overflows.The obvious risk here is tabs (byte 9), which are strictly ASCII but variable width.
But since
isAsciiOnlychecksval >= 32, so it returns false for\t. This forces tabbed content into the slow Unicode path wheretab_widthis handled properly.I also considered if it was possible for FFI consumers to pass
isAsciiOnly=truefor strings with tabs or newlines. But since the public API doesn't exposeisAsciiOnlydirectly, and instead derives it viautf8.isAsciiOnly(), which returns false for empty strings and control characters, this optimization is safe and transparent to external users.