Add unset field semantics for structs ("strict structs")

[AdamGlustein]

Cleanup and a few fixes to #574, thanks @sim15 for the contribution!

[ptomecek]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True?
I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

[AdamGlustein]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

[robambalu]

is there a central place we can do this check on push rather than consume? Will be much harder to track down on consumption

[AdamGlustein]

I did not see such a place - we could add it to each adapter type (push, pull, pushpull and managers) separately in their pushTick impls, but I think it's cleaner to have the validation logic in one spot.

[robambalu]

I agree, but my point stands that it would be a lot harder to tell where the error is coming from
Also unfortunate to vall validate on all input adapters, even python based ones which were already validated due to PyStruct creation in python

[ptomecek]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it), but if there isn't, then it's not. I guess I was wrong about pydantic inferring it. However, the behavior for non-defaulted fields will be different for the two types of Struct.