Checkpoint Streaming?

[blester125]

One thing that will always slow down Git Theta is the fact we have to wait for the native framework to be loaded in it's entirety before we can start processing parameters.

Will it be possible to start processing parameters as they are loaded? It would require support from the downstream framework, it would require custom reader code that understands a framework's format and does custom reading.

A similar approach would be to do lazy loading for parameters where the structure of the model is pre-populated but the values are only loaded when needed, this would allow things to start quicker and for the readings to be interleaved with the rest of execution via async.

This is a very ambitious idea and should probably be on of the last speedups we try

[blester125]

Some checkpoint plug-ins, like safetensors, support lazy loading which would probably be part of an implementation like this.