fix: preserve non-ASCII (CJK) path segments in auto-generated project name#624
fix: preserve non-ASCII (CJK) path segments in auto-generated project name#624mvanhorn wants to merge 1 commit into
Conversation
… name Percent-encode non-ASCII bytes in cbm_project_name_from_path so distinct CJK paths keep recoverable, collision-free slugs; accept '%' in cbm_validate_project_name. Bound the slug length with a hash suffix so deep multibyte paths stay within the OS filename-component limit. Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
Huge thanks for opening this PR and for the work you put into it. The maintainer shop is currently full, so this may sit for a bit before it gets a proper review. We will come back to this as soon as possible with real feedback; I wanted to make sure it did not sit unacknowledged in the meantime. |
|
Thank you @mvanhorn — closing this in favor of a distilled follow-up. The core of #571 (preserving CJK path segments by hex-encoding non-ASCII bytes) already landed on I've distilled exactly that improvement — a name-length cap with a hash suffix, added on top of the existing hex scheme so there's no project-name drift for already-indexed projects — into #745 ( Really appreciate the careful catch on the filename length limit. 🙏 |
cbm_project_name_from_path hex-encodes non-ASCII path bytes (DeusData#571), but had no length cap -- a deep CJK / non-ASCII path triples in length past the filesystem's 255-byte filename-component limit, so <cache>/<name>.db becomes un-openable (ENAMETOOLONG). Cap the derived name at FQN_MAX_NAME_LEN (200): names within the cap are returned UNCHANGED (no drift for already-indexed projects), while a longer name keeps its first (CAP-9) bytes plus an FNV-1a hash of the full name as a "-%08x" suffix, so distinct long paths stay distinct. The hex encoding scheme is unchanged. Distilled from DeusData#624 by Matt Van Horn (the ENAMETOOLONG catch); the rest of that PR was redundant with the DeusData#571 fix already on main and would have swapped the encoding scheme, causing project-name drift. Reproduce-first: project_name_length_capped_issue624 builds a deep CJK path (~376-byte raw name) and asserts the result is <= 200, validator-safe, and that two long paths differing only in the trailing segment map to different names; a short CJK path stays byte-identical (no drift). RED (376 > 200) -> GREEN. Full suite 5738/0. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> Co-Authored-By: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
What does this PR do?
Fixes #571:
cbm_project_name_from_path()derived an auto project slug by mappingevery byte outside
[A-Za-z0-9._-]to-and then collapsing dashes. For a pathsuch as
/Users/yunxin/Desktop/开发/后端, every UTF-8 byte of the CJK segments isunsafe, so each segment collapsed to a single
-that was then trimmed away,yielding
Users-yunxin-Desktop. Two different non-ASCII directories under the sameASCII parent collapsed to the same name, so the identifying information was
silently lost and distinct repos collided. Users with non-Latin directory names
saw indexing create a DB under a truncated, unrecognizable name.
This change percent-encodes non-ASCII bytes instead of discarding them, so
non-ASCII paths keep a distinct, recoverable name. It also keeps the slug
generator in agreement with
cbm_validate_project_name()-- the same invariantthat motivated the
#349space fix (a name that indexes butresolve_storelater rejects reports project-not-found).
Changes:
src/pipeline/fqn.c--cbm_project_name_from_path()now percent-encodes anybyte
>= 0x80as uppercase%HHinto a new, larger buffer, while ASCIIseparators / unsafe bytes still map to
-. ASCII slugs are byte-identical tobefore (e.g.
/home/u/my project->home-u-my-project). The existingdash/dot collapse and leading/trailing trim are preserved. Because the slug is
later used as
<cache>/<name>.db, long slugs are bounded to 200 bytes with adeterministic FNV-1a hash suffix so distinct long paths still produce distinct
names and stay within the OS filename-component limit. Includes overflow guards
and correct memory management.
src/foundation/str_util.c--cbm_validate_project_name()additionallyaccepts
%. It remains filesystem-safe; the existing..,/,\, andleading-
.rejections are unchanged.tests/test_fqn.c-- new coverage for CJK percent-encoding, distinctness, thelong-path bound, the
%-accepting validator, and an unchanged-ASCIIregression.
Example:
/Users/yunxin/Desktop/开发/后端now yieldsUsers-yunxin-Desktop-%E5%BC%80%E5%8F%91-%E5%90%8E%E7%AB%AF(distinct, valid)instead of
Users-yunxin-Desktop.Checklist
git commit -s) -- required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test) -- 5689 passed, 0 failedmake -f Makefile.cbm lint-ci) -- clang-format clean ontouched files
Fixes #571