Do we actually need standalone Kalman filtering?

[GordStephen]

I've been looking at Durbin and Koopman's smoothing algorithm (page 17 of this PDF) as a way to avoid repeatedly inverting the state covariance matrices - the method actually doesn't even require the filtered state distributions, only the prediction and observation innovation distributions and the Kalman gains.

Given that this package is only designed for offline analysis, are there any real-world situations where it would be necessary to explicitly filter - but not smooth - a time series? Could we get away with removing the filtering function and type, and just consolidating its relevant elements into the smoothing function? It would save a fair bit of computation and memory allocation during smoothing, and simplify the code base as well.

This might also lead back into the discussions in JuliaStats/Roadmap.jl#16 about how TimeModels.jl relates to Kalman filtering in Julia more broadly... Ideally I'd say Julia would have a fully-featured Kalman package that did filtering + smoothing, online + offline analysis, etc, and we could just use the relevant pieces here, but I definitely don't think TimeModels.jl should be that package - we shouldn't be implementing more than is necessary (in the context of the current package ecosystem) for fitting statistical time-series models.

As I see it, our filtering right now is just a means to an end, and not an end itself, but there may be other use cases I'm not aware of. @codles, your last PR hinted that you might be doing standalone filtering, is that the case?

[rob-luke]

Sorry to be a pain but I do use standalone filtering. Could we provide multiple smoothing algorithms?

I imagine you could have a common kalman_smooth(y, model; kwargs...) that defaulted to a fast method. But you could also pass in a requested method for specific use cases using kalman_smooth(method, y, model; kwargs...). Could this be feasible?

[rob-luke]

Also a clean stand alone filter implementation would be useful for undergrad students learning about filtering.

[GordStephen]

Definitely feasible, whether TimeModels.jl is the right place for it, I'm less sure... Sounds like the type of thing that could be in a standalone Kalman package. TimeModels could import it and use the relevant parts, but focus more on model specification and parameter estimation.

@wkearn and @ElOceanografo both have dedicated Kalman packages right now. It's been brought up before, but maybe there's some synergy potential there?

[rob-luke]

Is it the plan for TimeModels to provide an interface to the more complete packages such as StateSpace? If so, which other package would you suggest? Currently this is the only one that allows time varying matrices, but I would happily contribute to whichever package TimeModels would link to.

[GordStephen]

Yes, I think eventually that would be ideal, although there's still some work to be done to ensure that whatever package ends up being used supports everything that's needed - I could imagine moving parts of the existing Kalman code here out to a different repository to support that goal.

https://github.com/ElOceanografo/StateSpace.jl and https://github.com/wkearn/Kalman.jl would seem to be the two main candidates at this point, although I'd be happy to switch to any package that can provide feature parity and equal or better performance.

Edit: I should also reiterate that more generally I'd love to see a canonical, full-featured Julia Kalman package and would be happy to contribute to it where I could!

[ElOceanografo]

My original idea for StateSpace was to be a more general, full-featured Kalman filtering package, with the possibility that it might eventually be a back end for filtering/smoothing in TimeModels (if that ends up making sense/being worth it). StateSpace can actually now accept control input, thanks to a contribution from @JonnyCBB. Other contributions are welcome!

[GordStephen]

Ok, cool, then an easy first step is probably to recreate (where nessecary) TimeModel's general-purpose Kalman functionality (i.e the kalman_filter and RTS version of kalman_smooth) over in StateSpace and remove that from TimeModels - with EM estimation we won't actually use it any more.

I have some sparse matrix optimizations to push that will be easier to keep in the DK version of kalman_smooth for now, but eventually we can look at moving those parts over as well, then import StateSpace for smoothing, and keep the code here focused on specifying particular statistical models and estimating parameters.