Conversation
…age_datetime_format field
Previously, `_check_image_item` raised a ValueError when the stored `image-datetime` string did not match `image-datetime-format`, breaking existing iFDO files where these fields were inconsistent — a common case when files were produced by tools that set `image-datetime-format` to describe the intended serialization format while Pydantic serialized the datetime as ISO 8601 regardless. The validator now catches parse failures and leaves the string unchanged, allowing Pydantic's own ISO 8601 handling to take over; tests are updated to assert this fallback behaviour.
Renamed the misspelled `check_datatime_format` ("data" + "time") to `check_datetime_format` ("date" + "time") across all call sites and exports, and corrected the companion validator method `_validate_image_datatime` to `_validate_image_datetime`.
Datetime format-aware serialization and deserialization constitutes new behaviour rather than a pure bug fix, as datetimes are now always loaded as UTC-aware objects and written in the spec-defined space-separated format instead of ISO 8601, warranting a minor version bump.
Hi @GermanHydrogen, Thanks for this PR. I've added a couple of commits on top: one to make the datetime format parsing lenient so that existing iFDO files with mismatched datetime strings continue to load without error, and one to fix a typo in the exported function name (`check_datatime_format` -> `check_datetime_format`). I've also bumped the version to 1.4.0 given the behaviour change. I'll look to package this up and get it into the Marimba v1.1.0 release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR merges the
devbranch intomainto release v1.4.0. It makesimage-datetimeserialization and deserialization aware of theimage-datetime-formatfield, so that datetime strings are written using the format declared in the iFDO file rather than Pydantic's default ISO 8601 output. The default format is%Y-%m-%d %H:%M:%S.%fwith a space separator, as specified by the iFDO v2.1.0 schema. Loaded datetimes are parsed using the declared format and stored as UTC-aware objects. The format is inherited from the image-set-header to image-set-items, and from the first frame to subsequent frames for video items. A backwards-compatibility fallback ensures existing files with mismatched datetime strings continue to load without error.Problem
The
image-datetimefield was being serialized by Pydantic's default datetime handler, producing ISO 8601 strings with aTseparator (e.g.2019-03-04T08:37:24Z) regardless of the declaredimage-datetime-format. This violated the iFDO v2.1.0 spec, which defines the default format as%Y-%m-%d %H:%M:%S.%f. Additionally, theimage-datetime-formatfield was not used during deserialization, meaning datetime strings were not validated or parsed according to their declared format.Solution
A new
ifdo/_datetime/module handles format-aware parsing and serialization. Amodel_validatorpre-processes raw dict data on load, converting datetime strings to UTC-aware datetime objects using the declared format with a lenient fallback for mismatched strings. Amodel_serializeroniFDOpropagates the correct format to eachImageDatainstance before serialization, and afield_serializeronimage_datetimewrites the string using that format. Theimage_datetime_formatfield has been moved fromImageCaptureFieldstoImageCoreFieldsto reflect its role in both capture and core metadata.Design
The format inheritance logic mirrors the iFDO spec: the image-set-header format is the default for all items, overridden per-item by a locally declared
image-datetime-format, with video frames inheriting from the first frame. The serializer uses a private_image_datetime_formatattribute set on a deepcopy of theiFDOobject to avoid mutating state during serialization. The backwards-compatibility fallback in_check_image_itemsilently passes onValueError, leaving non-matching strings for Pydantic's own ISO 8601 handling.Impact
Datetime strings in newly written iFDO files will now use the spec-defined space-separated format, improving compliance. Existing files that contain ISO 8601 datetime strings load cleanly via the fallback. Consumers that parse
image-datetimestrings directly rather than using the Python library may see a format change in output files.Testing
Tests for the new
_datetimemodule cover format-matching (asserting parsed datetime objects), format fallback (asserting strings are left unchanged), format inheritance for images and videos, and round-trip load/save behaviour. All 8 tests pass.Breaking Changes
The serialized format of
image-datetimechanges from ISO 8601 (e.g.2019-03-04T08:37:24Z) to the spec-defined default (e.g.2019-03-04 08:37:24.000000) for files that do not declare a customimage-datetime-format. Consumers that parse datetime strings directly from iFDO files should update accordingly.Added Files
ifdo/_datetime/__init__.py: Public exports for the datetime moduleifdo/_datetime/_check_datetime.py: Format-aware datetime parsing with backwards-compatible fallbackifdo/_datetime/_format.py: Default datetime format constantifdo/_datetime/_serialize_datetime.py: Format propagation logic for serializationtests/test_check_datetime.py: Tests for datetime parsing and fallback behaviourtests/test_serialize_datetime.py: Tests for datetime serializationModified Files
ifdo/models/ifdo.py: Addedmodel_validatorandmodel_serializerfor format-aware datetime handlingifdo/models/ifdo_core.py: Movedimage_datetime_formathere fromImageCaptureFields; addedfield_serializerforimage_datetimeifdo/models/ifdo_capture.py: Removedimage_datetime_format(moved toImageCoreFields)pyproject.toml: Bumped version to 1.4.0config/mypy.ini: Added pydantic mypy plugintests/test_load.py: Updated datetime assertions to expect UTC-aware objectstests/test_save.py: Updated for new serialization format