SoftplusBackward0 nan values

[GrMrWb]

Function 'SoftplusBackward0' returned nan values in its 0th output. File "", line 189, in time_for_magic fit_gpytorch_model(mll) RuntimeError: Function 'SoftplusBackward0' returned nan values in its 0th output.

Hello everyone,

I currently have the above problem which I can't find the root cause, I have changed all the values to be torch.double and on the same device cpu. This problem occurred after I have upgraded my PC's RAM.

Do you have any suggestions? How can I collect more info about this warning?

[saitcakmak]

Hi @GrMrWb. Do you have a minimal code snipped that can reproduce the issue? A full stack trace would also be helpful in understanding where the error is coming from.

[GrMrWb]

def model_iniatilisation(self, x, y):
        
        # Single Task for Gaussian Process
        model = SingleTaskGP(x, y.view(-1,1))
        mll = ExactMarginalLogLikelihood(model.likelihood, model) 
        
        return model, mll
    
def acquistion_function(self, model: object, best_y: Tensor) -> object:
    
    return qExpectedImprovement(
        model = model,
        best_f = best_y
    )

def optimise_acquistion_function(self, EI: object, device):
    bounds = torch.tensor([[0]*96, [2.0]*96])
    
    return optimize_acqf(
        acq_function=EI,
        bounds = bounds,
        q = 1,
        num_restarts=100,
        raw_samples=1024,
        options = {"batch_limit":96, "maxiter":100}
    )

lambdas = torch.Tensor(
                [
                    np.ones((1, 96)),
                    self.high*np.ones((1, 96)), 
                    np.zeros((1, 96))
                ],
            ).view(3, 96)
scores = [0.5, 1, 2] # random scores
best_y = 0.0001
n_runs = 100

for i in range(n_runs):
            
          # Initiliase the model
          model_gp, mll = self.model_iniatilisation(lambdas, scores)
          
          fit_gpytorch_model(mll)
          
          # Acquistion Function Expected Improvement            
          EI = self.acquistion_function(model_gp, best_y)
          
          sample_next_lambda, _ = self.optimise_acquistion_function(EI, device)
      
          sample_next_score = self.generate_new_sample(sample_next_lambda, model, loss_function, device) # set a random score from 0 - 2
          
          scores = torch.cat([scores, sample_next_score])
          lambdas = torch.cat([lambdas, sample_next_lambda])

Hi @saitcakmak, thank you for your help

This might help you to reproduce the problem

[GrMrWb]

This error started just after the upgrade of my RAM and I am not sure whether is related or not but previously was running without any errors. I hope you will able to help me

[GrMrWb]

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\fit.py", line 164, in fit_gpytorch_model
    mll = fit_gpytorch_mll(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\fit.py", line 105, in fit_gpytorch_mll
    return FitGPyTorchMLL(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\utils\dispatcher.py", line 93, in __call__
    return func(*args, **kwargs)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\fit.py", line 252, in _fit_fallback
    optimizer(mll, closure=closure, **optimizer_kwargs)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\fit.py", line 120, in fit_gpytorch_mll_scipy
    result = scipy_minimize(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\core.py", line 109, in scipy_minimize
    raw = minimize_with_timeout(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\utils\timeout.py", line 82, in minimize_with_timeout
    return optimize.minimize(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_minimize.py", line 696, in minimize
    res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 359, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 285, in fun_and_grad
    self._update_fun()
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_optimize.py", line 76, in __call__
    self._compute_if_needed(x, *args)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\scipy\optimize\_optimize.py", line 70, in _compute_if_needed
    fg = self.fun(x, *args)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\closures\core.py", line 160, in __call__
    value, grads = _handle_numerical_errors(e, x=self.state, dtype=np_float64)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\utils\common.py", line 48, in _handle_numerical_errors
    raise error  # pragma: nocover
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\closures\core.py", line 150, in __call__
    value_tensor, grad_tensors = self.closure(**kwargs)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\botorch\optim\closures\core.py", line 66, in __call__
    self.backward(value)
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\torch\_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "c:\Users\wsma14\Documents\PhD Enviroment\venv2023\lib\site-packages\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'SoftplusBackward0' returned nan values in its 0th output.

[saitcakmak]

I cannot reproduce the error. Does this happen all the time for you (i.e. does the fit_gpytorch_model(mll) call always error out) or only after some number of iterations? If it is the first option, I'd recommend re-installing everything in a new virtual environment to see if it makes any difference (in case it is somehow related to RAM change). If it happens after some number of iterations, I'd recommend using dtype=torch.double (see #1444) to help with numerical precision. This would imply that you're running into some numerical errors (leading to NaNs) that are data dependent. Having too many identical or very close observations is a known cause of numerical errors, so that'd be another thing to look at.

[Balandat]

I would be quite surprised if this were related to your RAM change, it's probably a red herring. More likely that this is due to some numerical issues as Sait mentioned above.