Test failure of 'sgn_1' with musl libc

[hhoffstaette]

Gentoo CI noticed a test suite failure on musl

Out of curiosity I tried to reproduce it and indeed can do so (with 4.2.2 and also the latest 4.2.4), which is odd and probably a musl problem since the test suite of 4.2.2 passes on glibc 🎉

The test output summary says:

TEST SUMMARY
ERROR: number of missing images: 29
skipped images: ct.pnm cr.pnm sar2.pnm sar1.pnm mat.pnm mat.pnm mat.pnm sar1.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm
WARNING: skipped 38 tests
lossless_5 lossless_6 lossless_7 lossless_8 lossless_9 lossy_7 lossy_8 lossy_9 prg_2 prg_3 prg_4 prg_7 prg_8 prg_9 csty_0 csty_1 csty_2 csty_3 csty_4 csty_5 csty_6 csty_7 csty_8 csty_9 csty_10 csty_11 csty_12 csty_13 csty_14 ilyr_600 ilyr_605 ilyr_606 ilyr_607 ilyr_608 ilyr_609 roi_0 roi_1 roi_2
Number of tests: 91
Number of errors: 1
Failed tests:
    sgn_1.encode
STATUS: FAIL

and the specific test is:

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-171/test.jpc fmt=jpc mode=int verbose
Running JasPer encoder
/var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 -O prg=lrcp < /var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4/data/images/signed_12.pgx > /tmp/jasper-@localhost/jpcod-171/test.jpc
error: too few guard bits (2 < 20)
cannot encode main body
jpc_encode failed
error: cannot encode image
ERROR: ENCODER FAILURE
ERROR: ENCODER FAILURE

Looking at the test arguments, it does pass numgbits=2 and then complains about that, so it's not clear what is going here or why it succeeds on glibc.

Any idea what the problem might be? I can provide the full log if necessary.

[mdadams]

@hhoffstaette As best I can tell from looking at the source code, the "2 < 20" is referring to the fact that the value of 2 guard bits specified on the command line is too small for the data to be encoded, and the encoder believes that the number of guard bits should be at least 20. Although I cannot say definitively that this is bug in musl, I am very suspicious that this might be the case, given that none of the code in JasPer that calculates the number of necessary guard bits has been changed in a long time as far as I can recall. One thing that would be potentially helpful to know: If you move the print statement that outputs the "too few guard bits" message prior to the "if" body in which it is located for the case where things work (i.e., no using musl), what values does it print? In other words, in a situation where things work, what does it think that the two printed integers (related to numbers of guard bits) are equal to?

[hhoffstaette]

I enabled this line by removing the guard.

on glibc (success):

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-168/test.jpc fmt=jpc m>
Running JasPer encoder
/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 -O p>
numgbits 2 mingbits 0
Running JasPer decoder
/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T pgx < /tmp/jasper-@localhost/jpcod-168/test.jpc > /tmp/jasper-@localhost/jpcod-16>
STATUS: LOSSLESS (ENCFILESIZE=196)

on musl (failure):

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/var/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-2234/test.jpc fmt=>
Running JasPer encoder
/var/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 >
numgbits 2 mingbits 20
error: too few guard bits (2 < 20)
cannot encode main body
jpc_encode failed
error: cannot encode image
ERROR: ENCODER FAILURE
ERROR: ENCODER FAILURE

So the difference is the "numgbits 2 mingbits 0" vs. "numgbits 2 mingbits 20",
and the problem is inconsistent detection/configuration of mingbits.

[mdadams]

@hhoffstaette Thanks for checking this. So, it would appear that the use of musl somehow results in the number of necessary guard bits being computed incorrectly. Unfortunately, I don't recall how the number of guard bits is computed, since I wrote the code for that about 20 years ago ☹️. I guess that I'll have to trace through the computation and try to figure out how musl is causing an incorrect calculation. Of course, in the unlikely event that you have some spare time on your hands, you are certainly welcome to try this yourself and let me know what you find. 😉

[hhoffstaette]

Just looking at the possible origins of the value - if I had to make a guess where things go wrong:

jasper/src/libjasper/jpc/jpc_enc.c

Line 554 in 8766848

cp->tccp.numgbits = atoi(jas_tvparser_getval(tvp));

looks like a good candidate.

Update: that's not it, sorry. Somehow the expected value of 2 gets turned into a 20 somewhere else.

[hhoffstaette]

Someone also reproduced this bug on arm (32-bit) with glibc, so apparently it's not just musl itself.

[mdadams]

@hhoffstaette: I was looking at this issue again today. I am very suspicious of an overflow problem. You mentioned that the sgn_1 test fails for 32-bit ARM with glibc. Was the initial ARM+Musl failure also with a 32-bit system? If so, there is a high likelihood that the problem is caused by overflow in fixed-point numerical computation. Most users tend to be using 64-bit systems these days, and the sgn_1 test uses larger dynamic range data (i.e., 12/bits sample instead of the more common 8 bits/sample). It is not possible to guarantee that all of the tests will pass on a 32-bit machine, since some test cases use larger dynamic range data.

[hhoffstaette]

Thanks for taking a look again. I do not know any further details since I only forwarded the bug(s) from our tracker and do not have any 32-bit hardware. But the problem also reproduced for some people on amd64, so.. 🤷
Edit: sorry for not answering your actual question (Sunday morning 😴): the original reproducer was on 64-bit in a musl chroot, as sam explained.

[thesamesam]

• Holger reproduced it on amd64 (64-bit) musl in Test failure of 'sgn_1' with musl libc #384 (comment)
• The original bug report downstream in Gentoo was on 32-bit arm + musl
• In the downstream Gentoo bug, someone later managed to hit it on 32-bit arm + glibc

So maybe there's two different bugs (whatever makes it fail on 64-bit musl and whatever makes it fail on 32-bit {glibc,musl}).

I do not know any further details since I only forwarded the bug(s) from our tracker and do not have any 32-bit hardware.

In general, you don't actually need a 32-bit kernel running for these things, and all x86_64 CPUs support running in 32-bit mode in a chroot or container as long as the support wasn't disabled in your kernel.

Anyway, I'm confident that for the downstream cases mentioned, they were on 64-bit hardware but running in a 32-bit chroot (done for the purposes of testing for 32-bit only hardware where doing many package builds for testing is infeasible). It shouldn't make a difference though.

[hhoffstaette]

Fwiw since this has been a while I just re-ran things with my current toolchain (gcc-15 etc.) and can still reproduce with 4.2.4 in my musl chroot, but it passes natively on glibc. Identical toolchain options in both cases, plain -O2 (no crazy -frice). I also tried with -O1 to rule out any gcc vectorizer bitsmashing shenanigans, but that failed as well. So it does look like whatever is going wrong is going wrong consistently, but only in musl.

[hhoffstaette]

Hmm.. looking at the configure logs between glibc vs. musl:

--- config-glibc
+++ config-musl
@@ -19,7 +19,7 @@ JAS_SIZEOF_SIZE_T: 8
 JAS_HAVE_SSIZE_T: 
 -- Check size of int_fast32_t
 -- Check size of int_fast32_t - done
-JAS_SIZEOF_INT_FAST32_T: 8
+JAS_SIZEOF_INT_FAST32_T: 4
 -- Check size of int_least32_t
 -- Check size of int_least32_t - done
 JAS_SIZEOF_INT_LEAST32_T: 4
@@ -41,6 +41,6 @@ JAS_SIZEOF_INT128_T: 16
 -- Looking for mkostemp
 -- Looking for mkostemp - found
 -- Looking for snprintf
--- Looking for snprintf - found
+-- Looking for snprintf - not found
 JAS_ENABLE_MIF_CODEC: OFF
 Support for all native codecs has been included.

The smaller JAS_SIZEOF_INT_FAST32_T maybe?

[mdadams]

@thesamesam : You may well be right in that there may be more than one bug involved. It's difficult to say. It is strange, however, that only the sgn_1 test case seems to be impacted (if I have understood the issue correctly).

@hhoffstaette :
Incidentally, I noticed yesterday that GitHub has some 64-bit ARM-based Ubuntu runners in public preview. I am not sure how long they have been available, but I wasn't aware of them until now. In any case, yesterday, I added an ARM-based Ubuntu runner to the CI workflow for JasPer to help with catching bugs that are ARM-specific. As an additional data point for this issue, the CI tests all pass in the 24.04 Ubuntu ARM runner (with both GCC and Clang). So, at least the sgn_1 test case is passing for (64-bit) ARM in some cases.

@hhoffstaette: The smaller JAS_SIZEOF_INT_FAST32_T value could definitely be the cause of a fixed-point overflow problem. Of course, it is impossible to know for certain if overflow is the cause of the failed test case. I suspect that it may be at least part of the problem.

[SlavekB]

I can add that I observe the same problem with glibc not only on armhf – test performed on the future Ubuntu 26.04 (Resolute), but also on i386 – test performed on very old Debian 10 (Buster). This means that the problem is larger in scope, and there is a common denominator: 32-bit architecture.

My test builds were successful on amd64, arm64 and ppc64el.