Test failure of 'sgn_1' with musl libc

[hhoffstaette]

Gentoo CI noticed a test suite failure on musl

Out of curiosity I tried to reproduce it and indeed can do so (with 4.2.2 and also the latest 4.2.4), which is odd and probably a musl problem since the test suite of 4.2.2 passes on glibc 🎉

The test output summary says:

TEST SUMMARY
ERROR: number of missing images: 29
skipped images: ct.pnm cr.pnm sar2.pnm sar1.pnm mat.pnm mat.pnm mat.pnm sar1.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm target.pnm
WARNING: skipped 38 tests
lossless_5 lossless_6 lossless_7 lossless_8 lossless_9 lossy_7 lossy_8 lossy_9 prg_2 prg_3 prg_4 prg_7 prg_8 prg_9 csty_0 csty_1 csty_2 csty_3 csty_4 csty_5 csty_6 csty_7 csty_8 csty_9 csty_10 csty_11 csty_12 csty_13 csty_14 ilyr_600 ilyr_605 ilyr_606 ilyr_607 ilyr_608 ilyr_609 roi_0 roi_1 roi_2
Number of tests: 91
Number of errors: 1
Failed tests:
    sgn_1.encode
STATUS: FAIL

and the specific test is:

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-171/test.jpc fmt=jpc mode=int verbose
Running JasPer encoder
/var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 -O prg=lrcp < /var/tmp/portage/media-libs/jasper-4.2.4/work/jasper-version-4.2.4/data/images/signed_12.pgx > /tmp/jasper-@localhost/jpcod-171/test.jpc
error: too few guard bits (2 < 20)
cannot encode main body
jpc_encode failed
error: cannot encode image
ERROR: ENCODER FAILURE
ERROR: ENCODER FAILURE

Looking at the test arguments, it does pass numgbits=2 and then complains about that, so it's not clear what is going here or why it succeeds on glibc.

Any idea what the problem might be? I can provide the full log if necessary.

[mdadams]

@hhoffstaette As best I can tell from looking at the source code, the "2 < 20" is referring to the fact that the value of 2 guard bits specified on the command line is too small for the data to be encoded, and the encoder believes that the number of guard bits should be at least 20. Although I cannot say definitively that this is bug in musl, I am very suspicious that this might be the case, given that none of the code in JasPer that calculates the number of necessary guard bits has been changed in a long time as far as I can recall. One thing that would be potentially helpful to know: If you move the print statement that outputs the "too few guard bits" message prior to the "if" body in which it is located for the case where things work (i.e., no using musl), what values does it print? In other words, in a situation where things work, what does it think that the two printed integers (related to numbers of guard bits) are equal to?

[hhoffstaette]

I enabled this line by removing the guard.

on glibc (success):

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-168/test.jpc fmt=jpc m>
Running JasPer encoder
/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 -O p>
numgbits 2 mingbits 0
Running JasPer decoder
/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T pgx < /tmp/jasper-@localhost/jpcod-168/test.jpc > /tmp/jasper-@localhost/jpcod-16>
STATUS: LOSSLESS (ENCFILESIZE=196)

on musl (failure):

TESTID=sgn_1 ENC=jasper DEC=jasper
FMT=pgx WIDTH=8 HEIGHT=8 PREC=12 NUMCMPTS=1 RAWSIZE=96
test information:
    image            signed_12.pgx   
    id               sgn_1           
    pae              0               
    bug              oj_dec          
    fmt              jpc             
    mode             int             
software=jasper input=/var/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2/data/images/signed_12.pgx output=/tmp/jasper-@localhost/jpcod-2234/test.jpc fmt=>
Running JasPer encoder
/var/tmp/portage/media-libs/jasper-4.2.2/work/jasper-version-4.2.2_build/src/app/jasper  -T jpc -O cblkwidth=64 -O cblkheight=64 -O mode=int -O numrlvls=6 -O numgbits=2 >
numgbits 2 mingbits 20
error: too few guard bits (2 < 20)
cannot encode main body
jpc_encode failed
error: cannot encode image
ERROR: ENCODER FAILURE
ERROR: ENCODER FAILURE

So the difference is the "numgbits 2 mingbits 0" vs. "numgbits 2 mingbits 20",
and the problem is inconsistent detection/configuration of mingbits.

[mdadams]

@hhoffstaette Thanks for checking this. So, it would appear that the use of musl somehow results in the number of necessary guard bits being computed incorrectly. Unfortunately, I don't recall how the number of guard bits is computed, since I wrote the code for that about 20 years ago ☹️. I guess that I'll have to trace through the computation and try to figure out how musl is causing an incorrect calculation. Of course, in the unlikely event that you have some spare time on your hands, you are certainly welcome to try this yourself and let me know what you find. 😉

[hhoffstaette]

Just looking at the possible origins of the value - if I had to make a guess where things go wrong:

jasper/src/libjasper/jpc/jpc_enc.c

Line 554 in 8766848

cp->tccp.numgbits = atoi(jas_tvparser_getval(tvp));

looks like a good candidate.

Update: that's not it, sorry. Somehow the expected value of 2 gets turned into a 20 somewhere else.

[hhoffstaette]

Someone also reproduced this bug on arm (32-bit) with glibc, so apparently it's not just musl itself.