This repository was archived by the owner on Sep 3, 2024. It is now read-only.
This repository was archived by the owner on Sep 3, 2024. It is now read-only.
Implement 'recovery mode' #17
Description
This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the dialect-map-core models, or the dialect-map-data-diff DiffMessage model) mismatch the adapters and models defined within this repository.
For those cases, it would be great to adapt the current pubsub-job execution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️
Then, create a recovery-job execution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.