Unit tests failing since Jan 6th

[jonwzheng]

Description

Unit tests have been failing since January 6th. Notably, neither RMG-Py and RMG-database have had recent new commits - the last ones were in December.
Last successful CI: https://github.com/ReactionMechanismGenerator/RMG-Py/actions/runs/12617937457

First failing CI: https://github.com/ReactionMechanismGenerator/RMG-Py/actions/runs/12629122576

Edit: Chemprop is also having failures since Jan 6th (chemprop/chemprop#1145) so more likely than not it is related to the runners or some meta-aspect of how we are conducting the CI. turns out this was something completely unrelated

3 tests are failing:

FAILED test/rmgpy/molecule/groupTest.py::TestGroup::test_repr_png - AssertionError: 1
FAILED test/rmgpy/tools/fluxdiagramTest.py::FluxDiagramTest::test_avi_simple - AssertionError: 1
FAILED test/rmgpy/tools/fluxdiagramTest.py::FluxDiagramTest::test_avi_liquid - AssertionError: 1

apparently all due to a graphviz failure, traceback:

Warning: Could not load "/home/runner/miniconda3/envs/rmg_env/lib/graphviz/libgvplugin_pango.so.6" - file not found
Warning: Could not load "/home/runner/miniconda3/envs/rmg_env/lib/graphviz/libgvplugin_pango.so.6" - file not found
Format: "png" not recognized. Use one of: bmp canon cmap cmapx cmapx_np dot eps fig gtk gv ico imap imap_np ismap jpe jpeg jpg pdf pic plain plain-ext png pov ps ps2 svg svgz tif tiff tk vml vmlz x11 xdot xdot1.2 xdot1.4 xlib

Troubleshooting steps so far

• I verified that the mamba environments for both of the fail and success runs were the same. The runners were slightly different:
  Success run:

__glibc=2.35=0
__linux=6.5.0=0
Linux/6.5.0-1025-azure ubuntu/22.04.5 glibc/2.35

Fail run:

__glibc=2.39=0
__linux=6.8.0=0
Linux/6.8.0-1017-azure ubuntu/24.04.1 glibc/2.39

Could these subtle variations in runner be making a difference?

• I started a new branch, fix_failing_ci, and tried modifying the env file in case we're missing any hidden dependencies: https://stackoverflow.com/questions/60058398/graphviz-warning-with-libgvplugin-pango-so-6-dot-fails-to-produce-pdf
  Interestingly, we had this problem in the past ("graphviz/libgvplugin_pango.so.6" - file not found" missing dependency on WSL installations #2141) for WSL and we solved it by just apt-get install-ing graphviz.
  So far, adding these dependencies hasn't solved the problem.

What to do next?

if anyone has any better ideas, please comment below.

• Test manual run of unit-tests
• investigate whether different runners could be causing this issue? maybe force runner type in CI?

[JacksonBurns]

I ran into something similar to the 'pnbg' unit test failure while working on #2741 which I resolved by specifying - conda-forge::graphviz >=12 in the environment file. Notably the CI is still working in that PR. I will try adding this to that branch.

[JacksonBurns]

Note that the version of graphviz we are using is from the rmg channel - it is 10+ major versions out of date, and presumably likely to have issues such as this one.

[JacksonBurns]

This is the CI run for that change: https://github.com/ReactionMechanismGenerator/RMG-Py/actions/runs/12793707669/job/35667150523

[jonwzheng]

I see, thanks Jackson this is helpful!
So the interesting thing was that the mamba environments in the past successful and failing runs are the exact same. Meaning something else about the CI changed.

I think the runners were updated on Jan 6th to use slightly different defaults and that might be causing the issues. For example the CI run you just linked has issues with libglib, and the Jan 6th++ runners use glibc/2.39 while the old ones use glibc/2.35 which might be the source of the issue. I guess we can either try to enforce the runner defaults as a stopgap, or try to use a newer graphviz version (hopefully doable...)

[JacksonBurns]

Thanks for pointing that out - a likely culprit.

I have loosened the version requirement for graphviz in the hope that we can find a version which is compatible with our gblibc limits and still supports the PNG-related features: https://github.com/ReactionMechanismGenerator/RMG-Py/actions/runs/12793885598/job/35667738025

[JacksonBurns]

Thanks for pointing that out - a likely culprit.

I have loosened the version requirement for graphviz in the hope that we can find a version which is compatible with our gblibc limits and still supports the PNG-related features: https://github.com/ReactionMechanismGenerator/RMG-Py/actions/runs/12793885598/job/35667738025

@jonwzheng this run hasn't completed yet, but passed the unit tests, so it seems like this fixes the issue 🤞

Can you clean up the commits and open a PR? Or maybe just tack it on to #2748 ?

[jonwzheng]

@JacksonBurns nice work! I'll tack this onto #2748 and delete the fix_failing_ci branch. Thanks for the patch.