Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avro time converter respects timezone #24

Conversation

johnny-schmidt
Copy link

Summary

Addresses this bug (internally reported by me during S3 testing)

When a time with a timezone is converted to time of day, the timezone is ignored:

Given some input data like

"time_wo_tz":"03:20:48",
"time_w_tz":"07:53:25.917663+05:00

The output is

time_wo_tz:
12048000000 => / 1000000 / 3600 => 3.34... => 3:20:48...
time_w_tz:
28405917663 => / 1000000 / 3600 => 7.89.. => 7:53:43...

When we should expect

time_wo_tz:
12048000000 => / 1000000 / 3600 => 3.34... => 3:20:48...
time_w_tz:
10405917663 => / 1000000 / 3600 => 2.89.. => 2:53:43...

**NOTE: I'm doing this here and not in the PR that implements new style field conversions, because there is currently no way to pass logicalType="time-micros" through after converting a timestamp to a long in json-schema. I decided to hold off until refactoring it for the CDK (which will entail changing the schema mapper from using json-schema to using a bespoke airbyte type + migrating avro schema and record processing to the schema/record mappers).

Checklist

  • [X ] Write unit tests
  • Make sure there is no logging in the Json / Avro conversion code
  • [n/a] Update documentation in README.md

@johnny-schmidt johnny-schmidt merged commit 2c4d8da into master Aug 5, 2024
1 check passed
@johnny-schmidt johnny-schmidt deleted the jschmidt/issue-43019-avro-time-conversions-do-not-respect-timezone branch August 5, 2024 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants