The AIFS generates different predictions with the same inputs. How to fix it?

[LQscience]

What happened?

I'm running AIFS with the same inputs and random seed, but I'm getting inconsistent results. Could you please guide me on how to properly set the random seed so that I can obtain the same predictions under the same conditions?

Here's the random seed setting I'm using:

import torch
import numpy as np
import random

seed = 42
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(seed)
random.seed(seed)

What are the steps to reproduce the bug?

Here are the inconsistent results I have gotten:

Version

v2.1.0

Platform (OS and architecture)

Linux I1ed537bc0000401ffb 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

No response

Accompanying data

No response

Organisation

No response

[anaprietonem]

Hello @LQscience !

Thanks for opening this issue and for the interest in Anemoi. The behaviour that your are seeing is not a "bug" but the expected behaviour unless as you correctly suggested, one limits some of the Pytorch settings to enforce a deterministic behaviour. Regarding those settings, could you try with the following:

#First in your terminal
export CUBLAS_WORKSPACE_CONFIG=:4096:8

#And then before running inference:
import torch
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.use_deterministic_algorithms(True)

Note that enforcing a deterministic behavior might come at the cost of increased runtime.
We will aim to update our anemoi-inference docs to have a section that illustrates this better!