fix(tounicode): decode Type0 GBK-EUC-H fonts without ToUnicode via predefined CMaps

[yzxcj797]

Summary

A Type0 font whose /Encoding names a predefined CMap such as GBK-EUC-H (anything other than Identity-H/Identity-V) was skipped during ToUnicode CMap discovery, so its text decoded to U+FFFD garble and could misroute otherwise text-based pages to OCR (#400).

Three coordinated changes:

1. Predefined-CMap recognition (src/extractor/fonts.rs): Type0 fonts with any predefined encoding CMap shipped in external/bcmaps now participate in CMap collection. Untrusted /Encoding values are validated against a whitelisted character set and resolved only to files that actually ship — never interpreted as filesystem paths.
2. Binary CMap differential parsing (src/tounicode.rs): correct consumption of codespace/notdef operand pairs, cidchar/cidrange with hex code increments and signed CID deltas (read_signed), 16-bit range checks on both code and CID, and a bounded-expansion guard on the resulting map.
3. Adobe-GB1 CID→Unicode table (src/adobe_gb1.rs, 28,660 entries): generated from adobe-type-tools/cmap-resources Adobe-GB1-6/cid2code.txt (UniGB-UCS2 column, last mapping per CID — plain ideographs preferred over Kangxi radicals/compatibility forms), mirroring the existing adobe_korea1 module convention. GB1-ordered CIDSystemInfo now resolves through this table instead of failing.

Testing

• New regression test_type0_gbk_euc_h_without_tounicode_decodes_predefined_cmap: builds a synthetic Type0/GBK-EUC-H PDF without ToUnicode in memory and asserts both the composed CMap (0xB6AB 0xB7BD → "东方") and full extract_text_with_positions_mem output.
• New unit tests: gbk_euc_h_binary_cmap_maps_issue_code_points (>20k mappings, spot-checks the issue's code points) and adobe_gb1_prefers_plain_ideographs_for_reverse_mapping.
• cargo test --lib — 955/955 pass
• cargo test --test integration_tests — 163/163 pass
• cargo fmt --check clean; cargo clippy --lib --tests — 0 errors

Fixes #400

---

Summary by cubic

Decode Type0 fonts whose /Encoding names a predefined CMap (e.g., GBK‑EUC‑H) even without ToUnicode, avoiding U+FFFD output and OCR fallback. Previously these were skipped; now we compose the predefined encoding code→CID map with the collection CID→Unicode map.

• Compose non‑identity predefined encodings from bundled external/bcmaps with a new Adobe‑GB1 CID→Unicode table (adobe_gb1.rs); GB1 fonts now resolve >28k mappings.
• Preserve mixed‑width codespaces and decode 1‑ and 2‑byte codes; keep ASCII for printable unmapped single‑byte codes instead of emitting U+FFFD.
• Binary CMap parsing: consume codespace/notdef operands, handle hex increments and signed CID deltas, reject differential code overflow, enforce 16‑bit bounds, reject oversized hexadecimal operands before packing, and guard bounded expansion.
• Discovery in FontCMaps: Identity‑H/V still try embedded TrueType/OpenType then CIDSystemInfo; for other predefined encodings (e.g., GBK‑EUC‑H), build a composed ToUnicode directly.
• Security: validate /Encoding against a safe charset, only load bundled CMaps by name, never interpret names as filesystem paths, and check bundled availability without reading full files.
• Tests: synthetic Type0 GBK‑EUC‑H without ToUnicode end‑to‑end, bcmap parser mixed‑width/overflow/oversized‑operand cases, and GB1 mapping preferences; all pass.

^{Written for commit 7116f19. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 5 files

Not reviewed (too large): src/adobe_gb1.rs (~28,678 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/extractor/fonts.rs">

<violation number="1" location="src/extractor/fonts.rs:940">
P2: When a predefined Type0 font is used across many pages, this predicate rereads the entire `.bcmap` from disk during every page’s font setup. Cache the shipped-name validation or reuse the loaded CMap so extraction does not perform page-count-scaled filesystem I/O.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a predefined Type0 font is used across many pages, this predicate rereads the entire .bcmap from disk during every page’s font setup. Cache the shipped-name validation or reuse the loaded CMap so extraction does not perform page-count-scaled filesystem I/O.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/extractor/fonts.rs, line 940:

<comment>When a predefined Type0 font is used across many pages, this predicate rereads the entire `.bcmap` from disk during every page’s font setup. Cache the shipped-name validation or reuse the loaded CMap so extraction does not perform page-count-scaled filesystem I/O.</comment>

<file context>
@@ -936,7 +936,8 @@ pub(crate) fn get_font_file2_obj_num(doc: &Document, font_dict: &lopdf::Dictiona
         let encoding = font_dict.get(b"Encoding").ok()?.as_name().ok()?;
-        if encoding != b"Identity-H" && encoding != b"Identity-V" {
+        let is_identity = encoding == b"Identity-H" || encoding == b"Identity-V";
+        if !is_identity && !crate::tounicode::is_predefined_encoding_cmap(encoding) {
             return None;
         }
</file context>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic</sub>