Skip to content

fix: preserve non-ASCII (CJK) path segments in auto-generated project name#624

Closed
mvanhorn wants to merge 1 commit into
DeusData:mainfrom
mvanhorn:fix/571-cjk-project-name-slug
Closed

fix: preserve non-ASCII (CJK) path segments in auto-generated project name#624
mvanhorn wants to merge 1 commit into
DeusData:mainfrom
mvanhorn:fix/571-cjk-project-name-slug

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #571: cbm_project_name_from_path() derived an auto project slug by mapping
every byte outside [A-Za-z0-9._-] to - and then collapsing dashes. For a path
such as /Users/yunxin/Desktop/开发/后端, every UTF-8 byte of the CJK segments is
unsafe, so each segment collapsed to a single - that was then trimmed away,
yielding Users-yunxin-Desktop. Two different non-ASCII directories under the same
ASCII parent collapsed to the same name, so the identifying information was
silently lost and distinct repos collided. Users with non-Latin directory names
saw indexing create a DB under a truncated, unrecognizable name.

This change percent-encodes non-ASCII bytes instead of discarding them, so
non-ASCII paths keep a distinct, recoverable name. It also keeps the slug
generator in agreement with cbm_validate_project_name() -- the same invariant
that motivated the #349 space fix (a name that indexes but resolve_store
later rejects reports project-not-found).

Changes:

  • src/pipeline/fqn.c -- cbm_project_name_from_path() now percent-encodes any
    byte >= 0x80 as uppercase %HH into a new, larger buffer, while ASCII
    separators / unsafe bytes still map to -. ASCII slugs are byte-identical to
    before (e.g. /home/u/my project -> home-u-my-project). The existing
    dash/dot collapse and leading/trailing trim are preserved. Because the slug is
    later used as <cache>/<name>.db, long slugs are bounded to 200 bytes with a
    deterministic FNV-1a hash suffix so distinct long paths still produce distinct
    names and stay within the OS filename-component limit. Includes overflow guards
    and correct memory management.
  • src/foundation/str_util.c -- cbm_validate_project_name() additionally
    accepts %. It remains filesystem-safe; the existing .., /, \, and
    leading-. rejections are unchanged.
  • tests/test_fqn.c -- new coverage for CJK percent-encoding, distinctness, the
    long-path bound, the %-accepting validator, and an unchanged-ASCII
    regression.

Example: /Users/yunxin/Desktop/开发/后端 now yields
Users-yunxin-Desktop-%E5%BC%80%E5%8F%91-%E5%90%8E%E7%AB%AF (distinct, valid)
instead of Users-yunxin-Desktop.

Checklist

  • Every commit is signed off (git commit -s) -- required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test) -- 5689 passed, 0 failed
  • Lint passes (make -f Makefile.cbm lint-ci) -- clang-format clean on
    touched files
  • New behavior is covered by a test (reproduce-first for bug fixes)

Fixes #571

… name

Percent-encode non-ASCII bytes in cbm_project_name_from_path so distinct
CJK paths keep recoverable, collision-free slugs; accept '%' in
cbm_validate_project_name. Bound the slug length with a hash suffix so
deep multibyte paths stay within the OS filename-component limit.

Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
@DeusData

Copy link
Copy Markdown
Owner

Huge thanks for opening this PR and for the work you put into it.

The maintainer shop is currently full, so this may sit for a bit before it gets a proper review. We will come back to this as soon as possible with real feedback; I wanted to make sure it did not sit unacknowledged in the meantime.

@DeusData DeusData added bug Something isn't working ux/behavior Display bugs, docs, adoption UX priority/normal Standard review queue; useful PR with ordinary maintainer urgency. labels Jun 29, 2026
@DeusData

DeusData commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Thank you @mvanhorn — closing this in favor of a distilled follow-up.

The core of #571 (preserving CJK path segments by hex-encoding non-ASCII bytes) already landed on main in 3f754cf, which is why this PR now conflicts. But your PR caught a genuinely-missing piece that fix lacked: without a length cap, a deep CJK / non-ASCII path hex-encodes past the filesystem's 255-byte filename-component limit, making the .db un-openable (ENAMETOOLONG).

I've distilled exactly that improvement — a name-length cap with a hash suffix, added on top of the existing hex scheme so there's no project-name drift for already-indexed projects — into #745 (db20e53), co-authored with you. It's guarded by a reproduce-first test (project_name_length_capped_issue624), and the full suite is green.

Really appreciate the careful catch on the filename length limit. 🙏

@DeusData DeusData closed this Jul 1, 2026
timothybrush pushed a commit to timothybrush/codebase-memory-mcp that referenced this pull request Jul 1, 2026
cbm_project_name_from_path hex-encodes non-ASCII path bytes (DeusData#571), but had no
length cap -- a deep CJK / non-ASCII path triples in length past the
filesystem's 255-byte filename-component limit, so <cache>/<name>.db becomes
un-openable (ENAMETOOLONG).

Cap the derived name at FQN_MAX_NAME_LEN (200): names within the cap are
returned UNCHANGED (no drift for already-indexed projects), while a longer name
keeps its first (CAP-9) bytes plus an FNV-1a hash of the full name as a "-%08x"
suffix, so distinct long paths stay distinct. The hex encoding scheme is
unchanged.

Distilled from DeusData#624 by Matt Van Horn (the ENAMETOOLONG catch); the rest of that
PR was redundant with the DeusData#571 fix already on main and would have swapped the
encoding scheme, causing project-name drift.

Reproduce-first: project_name_length_capped_issue624 builds a deep CJK path
(~376-byte raw name) and asserts the result is <= 200, validator-safe, and that
two long paths differing only in the trailing segment map to different names; a
short CJK path stays byte-identical (no drift). RED (376 > 200) -> GREEN. Full
suite 5738/0.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-Authored-By: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/normal Standard review queue; useful PR with ordinary maintainer urgency. ux/behavior Display bugs, docs, adoption UX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Project name strips non-ASCII (CJK) characters from path, resulting in truncated/unrecognizable names

2 participants