`unicode_data` refactors #147622

Kmeakin · 2025-10-13T00:58:14Z

Minor refactors to unicode_data that occured to me while trying to reduce the size of the tables. Splitting into a separate PR. NFC

rustbot · 2025-10-13T00:58:17Z

library/core/src/unicode/unicode_data.rs is generated by the src/tools/unicode-table-generator tool.

If you want to modify unicode_data.rs, please modify the tool then regenerate the library source file via ./x run src/tools/unicode-table-generator instead of editing unicode_data.rs manually.

rustbot · 2025-10-13T00:58:19Z

r? @joboet

rustbot has assigned @joboet.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

library/core/src/unicode/rt.rs

src/tools/unicode-table-generator/src/main.rs

joboet · 2025-10-14T17:23:33Z

src/tools/unicode-table-generator/src/main.rs

+}
+
+fn rustfmt(path: &str) {
+    std::process::Command::new("rustfmt").arg(path).status().expect("rustfmt failed");


Is rustfmt really always in PATH when this command is run? Otherwise, I think it'd be easier to slap a big #[rustfmt::skip] on the mod unicode_data.

The intention is to keep the generated unicode_data.rs readable without having to carefully construct well-formatted code in the metaprogram. For that, we need to run rustfmt

I personally would rather just add #![rustfmt::skip] to the top of the file than run it through rustfmt, personally. I don't really think that the file is readable, rustfmt or not, due to the sheer amount of data included.

src/tools/unicode-table-generator/src/case_mapping.rs

src/tools/unicode-table-generator/src/main.rs

rustbot · 2025-10-19T23:42:39Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Kmeakin · 2025-10-27T22:32:00Z

@joboet ping?

joboet

Sorry for the delay. This looks great!
r=me after addressing the nit (or just ignore it, it's fine)

View changes since this review

library/core/src/unicode/rt.rs

joboet · 2025-10-31T13:20:51Z

@bors delegate+

bors · 2025-10-31T13:20:55Z

✌️ @Kmeakin, you can now approve this pull request!

If @joboet told you to "r=me" after making some further change, please make that change, then do @bors r=@joboet

Instead of `include_str!()`ing `range_search.rs`, just make it a normal module under `core::unicode`. This means the same source code doesn't have to be checked in twice, and it plays nicer with IDEs. Also rename it to `rt` since it includes functions for searching the bitsets and case conversion tables as well as the range represesentation.

Remove `#[rustfmt::skip]` from all the generated modules in `unicode_data.rs`. This means we won't have to worry so much about getting indetation and formatting right when generating code. Exempted for now some tables which would be too big when formatted by `rustfmt`.

This check was made redundant (it will always be true) when we removed all ASCII characters from the tables (rust-lang@a8c6694).

To make the final output code easier to see: * Get rid of the unnecessary line-noise of `.unwrap()`ing calls to `write!()` by moving the `.unwrap()` into a macro. * Join consecutive `write!()` calls using a single multiline format string. * Replace `.push()` and `.push_str(format!())` with `write!()`. * If after doing all of the above, there is only a single `write!()` call in the function, just construct the string directly with `format!()`.

Instead of generating a standalone executable to test `unicode_data`, generate normal tests in `coretests`. This ensures tests are always generated, and will be run as part of the normal testsuite. Also change the generated tests to loop over lookup tables, rather than generating a separate `assert_eq!()` statement for every codepoint. The old approach produced a massive (20,000 lines plus) file which took minutes to compile!

Kmeakin · 2025-10-31T16:23:36Z

@bors r+

bors · 2025-10-31T16:23:38Z

📌 Commit 0e6131c has been approved by Kmeakin

It is now in the queue for this repository.

joboet · 2025-10-31T16:33:35Z

@bors r-

joboet · 2025-10-31T16:33:48Z

@bors r=joboet

bors · 2025-10-31T16:33:51Z

📌 Commit 0e6131c has been approved by joboet

It is now in the queue for this repository.

… r=joboet `unicode_data` refactors Minor refactors to `unicode_data` that occured to me while trying to reduce the size of the tables. Splitting into a separate PR. NFC

Rollup of 9 pull requests Successful merges: - #139310 (add first HelenOS compilation targets) - #147161 (implement VecDeque extend_from_within and prepend_from_within) - #147622 (`unicode_data` refactors) - #147780 (Implement VecDeque::extract_if) - #147942 (Enable regression labeling aliases) - #147986 (Use fstatat() in DirEntry::metadata on Apple platforms) - #148103 (cg_llvm: Pass `debuginfo_compression` through FFI as an enum) - #148319 (docs: Fix argument names for `carrying_mul_add`) - #148322 (Enable file locking support in illumos) r? `@ghost` `@rustbot` modify labels: rollup

… r=joboet `unicode_data` refactors Minor refactors to `unicode_data` that occured to me while trying to reduce the size of the tables. Splitting into a separate PR. NFC

Rollup of 9 pull requests Successful merges: - #139310 (add first HelenOS compilation targets) - #147161 (implement VecDeque extend_from_within and prepend_from_within) - #147622 (`unicode_data` refactors) - #147780 (Implement VecDeque::extract_if) - #147942 (Enable regression labeling aliases) - #147986 (Use fstatat() in DirEntry::metadata on Apple platforms) - #148103 (cg_llvm: Pass `debuginfo_compression` through FFI as an enum) - #148319 (docs: Fix argument names for `carrying_mul_add`) - #148322 (Enable file locking support in illumos) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of #147622 - Kmeakin:km/unicode-data/refactors, r=joboet `unicode_data` refactors Minor refactors to `unicode_data` that occured to me while trying to reduce the size of the tables. Splitting into a separate PR. NFC

Zalathar · 2025-11-03T10:31:17Z

I think this PR might have caused the x86_64-gnu-aux CI job to go from ~2 hours to ~3.5 hours, making that job the slowest CI job.

https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/x86_64-gnu-aux.20job.20went.20from.20~2.20to.20~3.2E5.20hours/with/553349514

jieyouxu · 2025-11-03T10:58:50Z

src/tools/unicode-table-generator/src/main.rs

-    let mut out = String::new();
-    out.push_str("pub const UNICODE_VERSION: (u8, u8, u8) = ");
+fn rustfmt(path: &str) {
+    std::process::Command::new("rustfmt").arg(path).status().expect("rustfmt failed");


This is surprising to me, this will pick up whichever rustfmt happens to be in PATH? That we run rustfmt or not on the generated code is not the problem, the problem is that we need to run the same rustfmt for this generated code and also for the rest of standard library.

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Oct 13, 2025

rustbot assigned joboet Oct 13, 2025

Kmeakin force-pushed the km/unicode-data/refactors branch from 2c5244e to 90adbe2 Compare October 13, 2025 01:07

bjorn3 reviewed Oct 13, 2025

View reviewed changes

library/core/src/unicode/rt.rs Outdated Show resolved Hide resolved

Kmeakin force-pushed the km/unicode-data/refactors branch from 90adbe2 to 1a646cf Compare October 13, 2025 20:31

This comment has been minimized.

Sign in to view

joboet reviewed Oct 14, 2025

View reviewed changes

Kmeakin force-pushed the km/unicode-data/refactors branch 3 times, most recently from 1b56c98 to dc0dcf5 Compare October 19, 2025 23:42

rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Oct 19, 2025

This comment has been minimized.

Sign in to view

Kmeakin force-pushed the km/unicode-data/refactors branch from dc0dcf5 to d2c9773 Compare October 20, 2025 01:02

This comment has been minimized.

Sign in to view

Kmeakin force-pushed the km/unicode-data/refactors branch 2 times, most recently from 41d988f to ba38f37 Compare October 21, 2025 23:32

joboet approved these changes Oct 31, 2025

View reviewed changes

library/core/src/unicode/rt.rs Outdated Show resolved Hide resolved

Kmeakin added 5 commits October 31, 2025 14:11

refactor: remove check that first_code_point is non-ascii

6d75cd2

This check was made redundant (it will always be true) when we removed all ASCII characters from the tables (rust-lang@a8c6694).

Kmeakin force-pushed the km/unicode-data/refactors branch from ba38f37 to 0e6131c Compare October 31, 2025 14:20

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 31, 2025

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 31, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 31, 2025

matthiaskrgr mentioned this pull request Oct 31, 2025

Rollup of 9 pull requests #148337

Merged

Zalathar mentioned this pull request Nov 1, 2025

Rollup of 19 pull requests #148351

Closed

Zalathar mentioned this pull request Nov 1, 2025

Rollup of 19 pull requests #148354

Closed

bors merged commit 75fbbd3 into rust-lang:master Nov 1, 2025
11 checks passed

rustbot added this to the 1.93.0 milestone Nov 1, 2025

Kmeakin deleted the km/unicode-data/refactors branch November 1, 2025 11:26

clarfonthey mentioned this pull request Nov 1, 2025

Move more code out of unicode-table-generator into core #148365

Closed

dtolnay mentioned this pull request Nov 2, 2025

Standard library internal modules are treated as potential name colliders #148387

Open

jieyouxu reviewed Nov 3, 2025

View reviewed changes

joboet mentioned this pull request Nov 3, 2025

reland and fix RUST-147622 #148438

Open

unicode_data refactors #147622

unicode_data refactors #147622

Uh oh!

Conversation

Kmeakin commented Oct 13, 2025

Uh oh!

rustbot commented Oct 13, 2025

Uh oh!

rustbot commented Oct 13, 2025

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

joboet Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Kmeakin Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clarfonthey Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rustbot commented Oct 19, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Kmeakin commented Oct 27, 2025

Uh oh!

joboet left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joboet commented Oct 31, 2025

Uh oh!

bors commented Oct 31, 2025

Uh oh!

Kmeakin commented Oct 31, 2025

Uh oh!

bors commented Oct 31, 2025

Uh oh!

joboet commented Oct 31, 2025

Uh oh!

joboet commented Oct 31, 2025

Uh oh!

bors commented Oct 31, 2025

Uh oh!

Uh oh!

Zalathar commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

`unicode_data` refactors #147622

`unicode_data` refactors #147622

Kmeakin Oct 14, 2025 •

edited

Loading

joboet left a comment •

edited by rustbot

Loading

Zalathar commented Nov 3, 2025 •

edited

Loading

jieyouxu Nov 3, 2025 •

edited

Loading