Support exact Hessians of nonlinear objective and constraints in NonLinMPC and MovingHorizonEstimator

[franckgaga]

@baggepinnen I think for Hessians, it would make sense to always default to sparse computations (when activated, I would keep the LBFGS approximation as the default)? Should it be pure ForwardDiff, or mixed mode?

[franckgaga]

Also maybe @gdalle has tips on that subject ? Thanks!

[gdalle]

If your problems are reaasonably small, pure-ForwardDiff sparse Hessians are not a bad idea.
Otherwise you'd have to combine backends, and that's where it gets tricky:

• ReverseDiff only remains alloc-free in the absence of constants
• Zygote is of course not alloc-free, nor does it support mutation
• Mooncake and Enzyme don't work as inner backends for ForwardDiff

Pure-Enzyme sparse Hessians would also be an option but at the moment they're not so optimized, and there are still bugs I don't understand with respect to DI.Cache in second-order Enzyme