Large file (blob) hash values e.g. 5GB of zeros

[PhilipOakley]

I'm looking at an initial small change for the long -> size_t conversion, but I need some reference hash values for testing.

The current test code I'm using is:

        test-tool genzeros $((5*1024*1024*1024)) >big &&
        echo 0be2be10a4c8764f32c4bf372a98edc731a4b204 >expect &&
        # sha1: of 5GiB zeros, via WSL: dd if=/dev/zero bs=$((1024*1024)) count=$((5*1024)) >big &&
        git hash-object --stdin --literally <big >actual &&
        test_cmp expect actual

but the value determined using WSL doesn't match my updated code.

-0be2be10a4c8764f32c4bf372a98edc731a4b204
+4fce05a4e4ed8cefef2d99f32c519b2fd7841b74
not ok 13 - files over 4GB hash literally

Would someone be able to check/confirm what the expected hash should be on a linux box?
If a sha256 was also available that would be even better.

Aside: does the GitHub CI allow these EXPENSIVE tests to be performed, and any pointers as to how to initiate a single test?

[derrickstolee]

I checked the test on Linux and it passed. Seems that WSL is also affected by the unsigned long problem.

       test-tool genzeros $((5*1024*1024*1024)) >big &&
        echo 0be2be10a4c8764f32c4bf372a98edc731a4b204 >expect &&
        # sha1: of 5GiB zeros, via WSL: dd if=/dev/zero bs=$((1024*1024)) count=$((5*1024)) >big &&
        git hash-object --stdin --literally <big >actual &&
        test_cmp expect actual

+ test-tool genzeros 5368709120
+ echo 0be2be10a4c8764f32c4bf372a98edc731a4b204
+ git hash-object --stdin --literally
+ test_cmp expect actual
+ test 2 -ne 2
+ eval diff -u "$@"
+ diff -u expect actual
ok 1

[dscho]

FWIW my WSL Git agrees:

$ dd if=/dev/zero bs=$((1024*1024)) count=$((5*1024)) | time git hash-object --stdin --literally
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 35.8639 s, 150 MB/s
0be2be10a4c8764f32c4bf372a98edc731a4b204
76.25user 28.10system 1:48.27elapsed 96%CPU (0avgtext+0avgdata 5247324maxresident)k
0inputs+0outputs (0major+237423minor)pagefaults 0swaps

(And I think you meant to say that WSL is not affected by the unsigned long problem, @derrickstolee, because of Linux' LP64 data model, which is different than Windows' LLP64 data model, note the number of Ls.)

[PhilipOakley]

Thanks.

IIUC, the 0be2be10a4c8764f32c4bf372a98edc731a4b204 is linux correct. (which I generated on WSL)

That actually means that my simple code fix doesn't actually fix the problem (test result from my reinstalled SDK & my git version 2.34.0.rc0.windows.1.1.g53c1076ceb.dirty), and I must have missed out some size conversion factor.

[dscho]

Aside: does the GitHub CI allow these EXPENSIVE tests to be performed, and any pointers as to how to initiate a single test?

You could set GIT_TEST_LONG to force those to be performed, via the env: attribute in the windows-test job matrix.

[derrickstolee]

Would be worth breaking it out as its own CI job, and only running a specific set of long tests, not all of them in the test suite.

[PhilipOakley]

yes, I was looking (hopefully) for way of directly requesting just my updated t/t1051 tests

[dscho]

Right, we could do this in main.yml as a conditional step that only runs when targeting the main branch. But then, we do rely on CI builds in our rebase PRs, so I'd rather have those few expensive tests run as part of the regular CI/PR builds, even for Pull Requests.

[PhilipOakley]

Extra query: how to set the hash to sha256 for the dd if=/dev/zero bs=$((1024*1024)) count=$((5*1024)) | git hash-object --stdin --literally command? It's not something I've tried before.

Looking to use the test_oid and add the sha256 value.

[dscho]

how to set the hash to sha256

https://github.com/git-for-windows/git/blob/v2.33.1.windows.1/ci/run-build-and-tests.sh#L40: GIT_TEST_DEFAULT_HASH=sha256

[PhilipOakley]

Thanks I'll try that. It's shorter than the route that SO gave ;-)