Fix ARG_NOT_SET deserialization for _date fields in OperatorSerialization#66564
Open
hwang-cadent wants to merge 1 commit intoapache:mainfrom
Open
Fix ARG_NOT_SET deserialization for _date fields in OperatorSerialization#66564hwang-cadent wants to merge 1 commit intoapache:mainfrom
hwang-cadent wants to merge 1 commit intoapache:mainfrom
Conversation
3 tasks
hwang-cadent
force-pushed
the
fix-arg-not-set-deserialization-date-fields
branch
from
7005b5f to
e1d1f5a
Compare
``OperatorSerialization._deserialize_field_value`` previously sent every
``*_date`` field to ``_deserialize_datetime`` unconditionally. When an
operator stores a date-suffixed field as ``NOTSET`` (an ``ArgNotSet``
sentinel meaning "use the default at runtime") the value is encoded as
``{__type: ARG_NOT_SET}``. Passing that encoded value to
``_deserialize_datetime`` fails because it is not a datetime payload, so
deserializing such an operator (e.g. ``TriggerDagRunOperator`` whose
``logical_date`` defaults to ``NOTSET``) raised an error.
This change short-circuits ARG_NOT_SET on date fields and restores the
``NOTSET`` singleton, leaving ``None`` and real datetimes unchanged.
Adds a direct unit test in ``test_serialized_objects.py`` covering
``logical_date``, ``start_date`` and ``end_date``.
hwang-cadent
force-pushed
the
fix-arg-not-set-deserialization-date-fields
branch
from
e1d1f5a to
f2b465e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OperatorSerialization._deserialize_field_valuepreviously routed every*_datefield directly to_deserialize_datetime. When an operator stores a date-suffixed field asNOTSET(anArgNotSetsentinel meaning "use the default at runtime"), the value is encoded as{__type: ARG_NOT_SET}. Passing that encoding to_deserialize_datetimeraises because it is not a datetime payload, so deserializing such an operator (for exampleTriggerDagRunOperatorwhoselogical_datedefaults toNOTSET) fails.This PR short-circuits the ARG_NOT_SET case on date-suffixed fields and restores the
NOTSETsingleton.Nonevalues and real datetimes keep their existing behavior.Why a separate PR
This was originally bundled inside #56973 (dynamic
dag_idresolution inTriggerDagRunOperatorlinks). Per reviewer feedback from @potiuk, splitting it out keeps each concern independently reviewable, bisectable and backportable. #56973 is being rebased to drop the serialization change once this PR lands.Changes
airflow-core/src/airflow/serialization/serialized_objects.py: handle ARG_NOT_SET in_deserialize_field_valuefor date-suffixed fields.airflow-core/tests/unit/serialization/test_serialized_objects.py: addtest_deserialize_field_value_with_arg_not_set_for_date_fieldscoveringlogical_date,start_date,end_date.Type of change
Testing
_deserialize_field_valuereturnsNOTSETforlogical_date,start_dateandend_datewhen given an ARG_NOT_SET encoding.TriggerDagRunOperatorround-trip tests on Fix dynamic dag_id resolution inTriggerDagRunOperatorlinks #56973.