Skip to content

The node migration system to end all node migration systems #63

@xujustinj

Description

@xujustinj

This issue describes how node migration should ultimately be implemented.

The current NodeRegistry simply maps names (strings) to Node subclasses. We can do way better by registering each class with a minimum and maximum version defined as ClassVars. That way our registry could look like

FooNodeV1: >=1.0.0 <2.0.0
FooNodeV2: >=2.0.0 <3.0.0
BarNode: None None

So when we load a node with data {"type": "Foo", "version": "1.2.3"}, it'll get properly dispatched to the old FooNodeV1 class. If the engine operator wants, they can continue supporting the old version and the new version simultaneously.

Once the engine operator is sick of having FooNodeV1 nodes in their workflows, they can deprecate them by simply modifying the FooNodeV1 class. The operator still keeps both classes around, but they replace FooNodeV1.run method with a NotImplementedError. Then, they add a .to_next_version(self, edges: Sequence[Edge]) -> tuple[Self, Sequence[Edge]] method on the class. Some magic part of our pipeline will check for this method, and if it exists it will automatically bump the FooNodeV1 instance to FooNodeV2 during deserialization while also following through with those edge replacements.

Finally, once all the FooNodeV1 instances are gone (from the database, etc.), the engine operator can finally delete that class.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions