Getting full error stack trace during multirun

[claysmyth]

Hello,

Hydra is a game-changer, but I've been running into this issue lately that's been difficult to debug. In short, I use hydra to sweep through various configurations for ML experiments, and log each experiments results to Weights and Biases. When using the hydra multirun, I consistently run into the following issue:

  File "/home/claysmyth/miniconda3/envs/myenv/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
           ^^^^^^
  File "/home/claysmyth/miniconda3/envs/myenv/lib/python3.12/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
            ^^^^^^^^^^^^^^^
  File "/home/claysmyth/miniconda3/envs/myenv/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/claysmyth/miniconda3/envs/myenv/lib/python3.12/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 181, in sweep
    _ = r.return_value
        ^^^^^^^^^^^^^^
  File "/home/claysmyth/miniconda3/envs/myenv/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
wandb.sdk.wandb_manager.ManagerConnectionRefusedError: Connection to wandb service failed.

While I don't believe this is an issue with Hydra, it'd be useful to go deeper into the stack trace for debugging. Can anyone please provide some suggestions or insight? Thanks for the consideration!

[claysmyth]

Alternatively, what would the the correct location to catch this exception and turn off multirun to see if that fixes the issue?

[omry]

Are you calling the wandb logging from your code?