Preserve multibyte code points across fixed byte-chunk boundaries

[MicroMilo]

Summary

Opening an ordinary non-.void UTF-8 text file just over 512 KiB corrupts a multibyte character crossing the fixed chunk boundary. The minimized routed input changes éz to ��z.

Validation source commit: 5f109383e0b03f6db46ab30938d1328943fb19d1.

Code path

• apps/electron/src/main/ipc/files.ts:225-246: The main-process producer claims arbitrary byte slices preserve UTF-8 code-point boundaries, then decodes every slice independently.
• apps/electron/src/main/state.ts:787-804: Files larger than 512 KiB are routed into the streamed document path.
• apps/ui/src/core/editors/code/CodeEditor.tsx:788-820: The renderer requests fixed 512-KiB chunks, inserts each returned string independently, and advances using byte counts.
• apps/electron/src/preload/api/files.ts:15-16: The preload API exposes each chunk as decoded string content plus byte offsets.

Steps to reproduce

This report is based on a source-control-flow validation against the commit above.

1. Check out the source commit listed in this report.
2. Inspect the code path and contract anchors listed above.
3. Exercise the described boundary/state path: Preserve multibyte code points across fixed byte-chunk boundaries.
4. Compare the observed behavior with the expected contract below.

Validation note: The smallest payload exceeding the product's stream threshold and splitting a two-byte character at byte 524288 produces expectedBoundaryText: "éz", observedBoundaryText: "��z", replacementCount: 2, and preserved: false. The validation first verifies the exact producer, routing, and consumer operations remain present in the frozen sources.

Expected behavior

The implementation should preserve the repository contract described by the documentation, public type, parser, serializer, or sibling implementation anchors above.

Actual behavior

Opening an ordinary non-.void UTF-8 text file just over 512 KiB corrupts a multibyte character crossing the fixed chunk boundary. The minimized routed input changes éz to ��z.

Existing coverage

I did not find an existing issue or PR title that appears to cover this exact root cause in the pre-submission check.

Suggested fix

UTF-8-safe streamed file decoding

Suggested tests

• Add a regression test for: Preserve multibyte code points across fixed byte-chunk boundaries.
• Include the boundary value or state transition described above so the old behavior fails before the fix.

---

Submitted with Codex.

[davinder-sudo]

Hi @MicroMilo,

Thank you for taking the time to report this issue. We appreciate your feedback and will work on addressing it in an upcoming release.

[A-S-Manoj]

I'd like to take this one.

Plan: in apps/electron/src/main/ipc/files.ts (around 225-246), track any trailing undecoded bytes at the end of a 512 KiB chunk and prepend them to the next chunk before decoding, so multibyte UTF-8 sequences aren't split across chunk boundaries. Will add a regression test based on the boundary case in this report (character split at byte 524288) to confirm it fails before the fix and passes after.

Will open a PR shortly.

[davinder-sudo]

Hi @A-S-Manoj,
Thanks for taking this up! Your proposed approach looks good to me. I've assigned the issue to you—feel free to proceed. Looking forward to your PR. Good luck!