Skip to content

Commit 8f490a9

Browse files
Speed up unpacking .tar.xz files in the hermetic toolchain by using multithreaded xz execution.
Note that the multithreaded feature should be available in non-hermetic `xz` tool. The next step will be to make `tar` and `xz` tools available in hermetic toolchains. Linux x86 platform operations | repository creation time, s :-----------: | :-----------: download clang.tar.xz and extract with bazel | 137 download clang.tar and extract with bazel | 48 download clang.tar.xz and extract with non-hermetic tar (with multithreaded xz) | 19 download clang.tar and extract with non-hermetic tar | 38 Parameters of the Linux machine that was used to collect the data: ``` Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 ``` PiperOrigin-RevId: 814354233
1 parent 0606f9b commit 8f490a9

File tree

2 files changed

+3
-6
lines changed

2 files changed

+3
-6
lines changed

.bazelrc

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -413,9 +413,6 @@ common:rbe_linux_x86_64_base --platforms="@ml_build_config_platform//:platform"
413413
common:rbe_linux_x86_64 --config=rbe_linux_x86_64_base
414414
common:rbe_linux_x86_64 --config=ci_linux_x86_64
415415

416-
common:use_tar_archive_files --repo_env=USE_CUDA_TAR_ARCHIVE_FILES=1
417-
common:use_tar_archive_files --repo_env=USE_NVSHMEM_TAR_ARCHIVE_FILES=1
418-
419416
common:rbe_linux_x86_64_cuda_common --config=rbe_linux_x86_64_base
420417
common:rbe_linux_x86_64_cuda_common --repo_env=REMOTE_GPU_TESTING=1
421418
# Speed up CUDA repos creation by downloading ".tar" dists from the mirror.

WORKSPACE

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@ load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
1717
# Details: https://github.com/google-ml-infra/rules_ml_toolchain
1818
http_archive(
1919
name = "rules_ml_toolchain",
20-
sha256 = "bbe18d4948cd4845255ee8d400d2f29854b64d1434523052e304c1d41e90f2ec",
21-
strip_prefix = "rules_ml_toolchain-3a77adf9473cf958967bc42eb9c1a663cd7ea878",
20+
sha256 = "d5aef14096f367bae6f2deb5946a31892b06badd660f8c30f6565a0f9a99185f",
21+
strip_prefix = "rules_ml_toolchain-18ba88facf7f0d7203adbaccb982315936b1ac7a",
2222
urls = [
23-
"https://github.com/google-ml-infra/rules_ml_toolchain/archive/3a77adf9473cf958967bc42eb9c1a663cd7ea878.tar.gz",
23+
"https://github.com/google-ml-infra/rules_ml_toolchain/archive/18ba88facf7f0d7203adbaccb982315936b1ac7a.tar.gz",
2424
],
2525
)
2626

0 commit comments

Comments
 (0)