set_variant(): try next requested variant if one fails to import

[merlinND]

Description

If the user specifies several variants in order, e.g.:

mi.set_variant("cuda_ad_rgb", "llvm_ad_rgb")

and they are all compiled (available) but some fail to import, then keep trying with the next requested variant instead of throwing an exception.

This could happen e.g. when requested a CUDA variant with the NVIDIA driver installed / CUDA available but no GPU available, or when LLVM is not installed.

Testing

On a machine with the NVIDIA driver installed,

CUDA_VISIBLE_DEVICES= python -c 'import mitsuba as mi; mi.set_variant("cuda_ad_rgb", "llvm_ad_rgb"); print(mi.variant())'

Old behavior: raises ImportError and exits.
New behavior: fails to load cuda_ad_rgb, but correctly goes to the second choice llvm_ad_rgb.

The old behavior can still be obtained by requesting a single variant at a time (no fallback): mi.set_variant("cuda_ad_rgb")

Checklist

• My code follows the style guidelines of this project
• My changes generate no new warnings
• My code also compiles for cuda_* and llvm_* variants. If you can't test this, please leave below
• I have commented my code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I cleaned the commit history and removed any "Merge" commits
• I give permission that the Mitsuba 3 project may redistribute my contributions under the terms of its license

[njroussel]

I like this! I've had it on my backlog to have mi.variants() only report valid variants, but there's one nasty problem: if that logic is ever wrong/broken, we'll be skipping hundreds of tests on the CI. Keeping this fallback logic in set_variant() shouldn't ever mess with how we've setup the test suite 👍

[merlinND]

Thanks for the review @njroussel and @wjakob!
I've rebased and incorporated the feedback.

Currently, both on master and with this PR, failing to set a variant results in a lot of leaks when exiting, even if the exception is caught and we then successfully set another variant:

# CUDA_VISIBLE_DEVICES= python ./test_failed_variant.py

import mitsuba as mi

if __name__ == "__main__":
    mi.set_log_level(mi.LogLevel.Debug)
    try:
        mi.set_variant("cuda_ad_rgb")
    except ImportError as e:
        print("Failed to set variant")

    print("Current variant:", mi.variant())
    # Still leaks if we then set a valid variant
    mi.set_variant("llvm_ad_rgb")
    print("Current variant:", mi.variant())

nanobind: leaked 23 instances!
 - leaked instance 0x769448183008 of type "mitsuba.Color0f"
 - leaked instance 0x769443f3bac8 of type "drjit.cuda.ad.TensorXf"
 - leaked instance 0x769443f1ebd0 of type "drjit.scalar.Array3f"
 - ... skipped remainder
nanobind: leaked 273 types!
 - leaked type "drjit.cuda.ad.Int"
 - leaked type "mitsuba.Ray3d"
 - ... skipped remainder
nanobind: leaked 1486 functions!
- ...

it's unrelated to this PR, though.

Edit: hopefully fixed with 4306f55

[merlinND]

Pinging this PR, would it be okay to merge?

The latest commit (4306f55) avoids partial initialization of variants when a backend turns out to be unavailable, which used to result in leaks (even before this PR).

[njroussel]

LGTM!

Thanks for looking at the leaks too :)