Different results with and without library(future)

[ahaeusser]

When I load package future before running my modelling code, the models (and forecasts) produced with the tidy fabletools workflow differ from the models (and forecasts) produced by the underlying base functions in echos. Without future the two approaches return identical numbers.

Hint: The functions ESN() and forecast.ESN() are just wrapper for train_esn() and forecast_esn() and should produce the same results. There is randomization within the ESN model handled internally by set.seed().

Minimal reproducible example

# Case (1) WITHOUT library(future) ============================================

library(echos)
library(fabletools)
library(tidyverse)

# Case (1.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point

# Case (1.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
  model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean

# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)

# Case (2) WITH library(future) ===============================================

library(echos)
library(fabletools)
library(tidyverse)
library(future)

# Case (2.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point

# Case (2.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
  model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean

# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)

Question
I think the issue begins within model()... Is model() and/or forecast() (or other fabletools internals) expected to react differently when future is loaded? If not, could there be an interaction with future’s RNG handling that slips through the tidy interface but not the base one?

Any tips or recommended work-arounds would be much appreciated.

Thank you for your time and for the fantastic package!

Best regards,
Alex

[mitchelloharawild]

Thanks for bring this to my attention, I do suspect that this is due to the RNG handling in parallel - however if you set.seed inside your functions that should still be respected.

I plan to switch parallelisation from {future} to {mirai} this year to be more minimal (#418). I'll investigate this while I make the transition to mirai.