perf: Add a new tool for calculating file checksums faster

[frankdavid]

When producing disk images during the IC-OS build, the Bazel built-in checksum calculator is quite slow when those images contain a lot of holes (long sections of zero bytes). These holes are stored in an optimized form using sparse files in the file system but Bazel's checksum algorithm is not optimized for them.

This wasn't an issue previously because we processed the artifacts between build steps (tarring, compressing, decompressing, untarring), but this processing between build steps adds additional slowness and code/debugging complexity.

As a preliminary step to removing the artifact processing, this PR introduces a much faster way (up to 100x faster on our biggest artifacts) to calculate checksums.

The new icsum tool implements a fast checksum algorithm optimized specifically for sparse files. The calculated checksum is stored in the user.icsum extended attribute of IC-OS artifacts and the Bazel config has been changed to try to use this checksum when available.

[basvandijk]

@frankdavid should we review after CI is green or do you expect things are mostly done?

[github-actions]

Run URL: https://github.com/dfinity/ic/actions/runs/12991989539

New container image: sha256:5926f073e5d8c1b37d5905697107f13dd3876a44863225910e146f5e2d26de2c
New container tag: 9afdf4979b00656c1ad15198488721a1055170c1b8b45323c60a434574327670

[mbjorkqvist]

Impressive performance improvements - thanks for looking into this, @frankdavid!

I'm wondering if "the Bazel built-in checksum calculator" uses sha256, or some other cryptographic hash function? xxHash seems to be a non-cryptographic hash algorithm, so changing could have some security implications.

[frankdavid]

should we review after CI is green

It's all green now.

I'm wondering if "the Bazel built-in checksum calculator" uses sha256, or some other cryptographic hash function?

It uses blake hash which is a cryptographic hash function. I wonder how important it is to have a cryptographic hash for a cache checksum. I thought that since the hash is only used to calculate if there has been a change in the file, it doesn't need to be safe against adversaries. Even if someone could create a version of a file which produces the same checksum, I don't see what they would gain from tricking the build system into not recompiling dependent targets.

The more problematic thing would be if we hit collisions by accident, meaning two subsequent builds would produce the same hash even though some files have changed. With 128-bit hash values the probability of this is very low. Furthermore, xxhash is widely used in the industry.

Please let me know what you think.

[andrewbattat]

LGTM! I'll leave it to Eero to take a look and approve though. Nice job!

Out of curiosity, do you have numbers on the overall build time improvements?

[nmattia]

hi, can you clarify why this need to be a standalone workspace?

[frankdavid]

The choice was motivated by how the tool is compiled. It's compiled in the Dockerfile (also included in this PR). I didn't want to copy the entire repo just to compile this one tool, so it seemed a better choice to make it into its own workspace. Let me know if you can recommend a better approach.

[mbjorkqvist]

I'm wondering if "the Bazel built-in checksum calculator" uses sha256, or some other cryptographic hash function?

It uses blake hash which is a cryptographic hash function. I wonder how important it is to have a cryptographic hash for a cache checksum. I thought that since the hash is only used to calculate if there has been a change in the file, it doesn't need to be safe against adversaries. Even if someone could create a version of a file which produces the same checksum, I don't see what they would gain from tricking the build system into not recompiling dependent targets.

The more problematic thing would be if we hit collisions by accident, meaning two subsequent builds would produce the same hash even though some files have changed. With 128-bit hash values the probability of this is very low. Furthermore, xxhash is widely used in the industry.

Please let me know what you think.

Thanks for the insight! I pinged @dfinity/product-security on Slack to have a look. The changes where FI approval is needed are trivial, so if the approach otherwise is approved then I have no problem approving.

[Bownairo]

Just a few small comments and nits

[frankdavid]

I've updated the algorithm to be cryptographically secure. It runs a bit slower now (~half the speed of the previous implementation) but it's still fast enough for now (0.8s on our largest artifacts).