Skip to content

Commit

Permalink
Support timestamp-nanos logical type
Browse files Browse the repository at this point in the history
  • Loading branch information
dstelljes committed Jan 7, 2025
1 parent dcea553 commit 504e0de
Show file tree
Hide file tree
Showing 25 changed files with 465 additions and 247 deletions.
2 changes: 1 addition & 1 deletion docs/cli/create.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ dotnet avro create --assembly ./out/Example.Models.dll --type Example.Models.Exa
: Whether reference types selected for nullable record fields should be annotated as nullable.

`--temporal-behavior`
: Whether timestamps should be represented with `"string"` schemas (ISO 8601) or `"long"` schemas (timestamp logical types). Options are `iso8601`, `epochmilliseconds`, and `epochmicroseconds`.
: Whether timestamps should be represented with `"string"` schemas (ISO 8601) or `"long"` schemas (timestamp logical types). Options are `iso8601`, `epochmilliseconds`, `epochmicroseconds`, and `epochnanoseconds`.
17 changes: 9 additions & 8 deletions docs/internals/mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,12 +106,12 @@ In addition to <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system

## Dates and times

The Avro spec defines six logical types for temporal data:
The Avro spec defines ten logical types for temporal data:

* calendar dates with no time or time zone (`"date"`)
* duration comprised of months, days, and milliseconds (`"duration"`)
* times of day with no date or time zone (`"time-millis"` and `"time-micros"`)
* instants in time (`"timestamp-millis"` and `"timestamp-micros"`)
* calendar dates (`"date"`)
* duration in months, days, and milliseconds (`"duration"`)
* times of day (`"time-millis"` and `"time-micros"`)
* instants in time (`"timestamp-millis"`, `"timestamp-micros"`, `"timestamp-nanos"`, `"local-timestamp-millis"`, `"local-timestamp-micros"`, and `"local-timestamp-nanos"`)

In addition to the conversions described later in this section, these logical types can be treated as their underlying primitive types:

Expand Down Expand Up @@ -142,15 +142,16 @@ Serializing and deserializing `"duration"` values still work, though there are s

.NET’s <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.timeonly">TimeOnly</a></code> struct represents a time of day with no time zone. To match other temporal types, Chr.Avro prefers to map <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.timeonly">TimeOnly</a></code>s to [ISO 8601 strings](https://en.wikipedia.org/wiki/ISO_8601#Times), avoiding `"time-millis"` and `"time-micros"` by default when building schemas. <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.formatexception">FormatException</a></code> is thrown when deserializing a value that cannot be parsed.

Serializing and deserializing `"time-millis"` and `"time-micros"` values is supported. However, .NET date types are tick-precision, so serializing to `"time-millis"` or deserializing from `"time-micros"` may result in a loss of precision.
Serializing and deserializing `"time-millis"` and `"time-micros"` values is supported. However, .NET date types are tick-precision, so serializing to `"time-millis"` or `"time-micros"` may incur a loss of precision.

### Timestamps

Both <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.datetime">DateTime</a></code> and <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.datetimeoffset">DateTimeOffset</a></code> can be used to represent timestamps. Chr.Avro prefers to map those types to [ISO 8601 strings](https://en.wikipedia.org/wiki/ISO_8601#Combined_date_and_time_representations), avoiding `"timestamp-millis"` and `"timestamp-micros"` by default when building schemas. This behavior is consistent with how durations are handled, and it also means that <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.datetime">DateTime</a></code> kind and timezone are retained—the [round-trip (“O”, “o”) format specifier](https://docs.microsoft.com/en-us/dotnet/standard/base-types/standard-date-and-time-format-strings#Roundtrip) is used for serialization. <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.formatexception">FormatException</a></code> is thrown when a string cannot be parsed.
Serializing and deserializing `"timestamp-millis"` and `"timestamp-micros"` values are supported as well, with a few caveats:

Serializing and deserializing `"timestamp-millis"`, `"timestamp-micros"`, and `"timestamp-nanos"` values are supported as well, with a few caveats:

* All <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.datetime">DateTime</a></code>s are converted to UTC. Don’t use <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.datetime">DateTime</a></code>s with kind unspecified.
* .NET date types are tick-precision, so serializing to `"timestamp-millis"` or deserializing from `"timestamp-micros"` may result in a loss of precision.
* .NET date types are tick-precision, so serializing to `"timestamp-millis"` or `"timestamp-micros"` or deserializing from `"timestamp-nanos"` may result in a loss of precision.

## Enums

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="NodaTime" Version="3.1.9" />
<PackageReference Include="NodaTime" Version="3.2.0" />
</ItemGroup>

<ItemGroup>
Expand Down

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,39 @@ namespace Chr.Avro.NodaTimeExample.Infrastructure
using Chr.Avro.Abstract;
using Chr.Avro.Serialization;
using NodaTime;
using NodaTime.Text;

public class NodaTimeDeserializerBuilderCase : BinaryTimestampDeserializerBuilderCase
{
private readonly BinaryStringDeserializerBuilderCase stringDeserializer = new();

public override BinaryDeserializerBuilderCaseResult BuildExpression(Type type, Schema schema, BinaryDeserializerBuilderContext context)
{
if (type != typeof(Instant))
{
return BinaryDeserializerBuilderCaseResult.FromException(
new UnsupportedTypeException(type, $"{nameof(NodaTimeDeserializerBuilderCase)} can only be applied to {nameof(Instant)}."));
}

if (schema is StringSchema)
{
// Fallback to default implementation if not NodaTime
if (!(type == typeof(NodaTime.Instant) || type == typeof(NodaTime.Instant?)))
{
return base.BuildExpression(type, schema, context);
}
var readString = typeof(BinaryReader)
.GetMethod(nameof(BinaryReader.ReadString), Type.EmptyTypes)!;

// Use default conversion from string to DateTimeOffset
var dateTimeOffset = stringDeserializer.BuildExpression(typeof(DateTimeOffset), schema, context).Expression;
var parse = typeof(IPattern<Instant>)
.GetMethod(nameof(IPattern<Instant>.Parse), new[] { typeof(string) })!;

var instantFromDateTimeOffset = typeof(Instant).GetMethod(nameof(Instant.FromDateTimeOffset), System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.Public, new[] { typeof(DateTimeOffset) });
var getValueOrThrow = typeof(ParseResult<Instant>)
.GetMethod(nameof(ParseResult<Instant>.GetValueOrThrow), Type.EmptyTypes)!;

try
{
// Code: NodaTime.Instant.FromDateTimeOffset(dateTimeOffset);
return BinaryDeserializerBuilderCaseResult.FromExpression(
BuildConversion(
Expression.Call(instantFromDateTimeOffset!, dateTimeOffset!),
Expression.Call(
Expression.Call(
Expression.Constant(InstantPattern.ExtendedIso),
parse,
Expression.Call(context.Reader, readString)),
getValueOrThrow),
type));
}
catch (InvalidOperationException exception)
Expand All @@ -45,41 +52,31 @@ public override BinaryDeserializerBuilderCaseResult BuildExpression(Type type, S
throw new UnsupportedSchemaException(schema, $"{nameof(TimestampLogicalType)} deserializers can only be built for {nameof(LongSchema)}s.");
}

// Fallback to default implementation if not NodaTime
if (!(type == typeof(Instant) || type == typeof(Instant?)))
{
return base.BuildExpression(type, schema, context);
}

var factor = schema.LogicalType switch
{
MicrosecondTimestampLogicalType => TimeSpan.TicksPerMillisecond / 1000,
MillisecondTimestampLogicalType => TimeSpan.TicksPerMillisecond,
_ => throw new UnsupportedSchemaException(schema, $"{schema.LogicalType} is not a supported {nameof(TimestampLogicalType)}."),
};

var readInteger = typeof(BinaryReader)
.GetMethod(nameof(BinaryReader.ReadInteger), Type.EmptyTypes);
.GetMethod(nameof(BinaryReader.ReadInteger), Type.EmptyTypes)!;

Expression expression = Expression.Call(context.Reader, readInteger!);
var plusNanoseconds = typeof(Instant)
.GetMethod(nameof(Instant.PlusNanoseconds), new[] { typeof(long) })!;

var addTicks = typeof(DateTime)
.GetMethod(nameof(DateTime.AddTicks), new[] { typeof(long) });
var plusTicks = typeof(Instant)
.GetMethod(nameof(Instant.PlusTicks), new[] { typeof(long) })!;

var fromDateTimeUtc = typeof(NodaTime.Instant).GetMethod(nameof(NodaTime.Instant.FromDateTimeUtc), new[] { typeof(DateTime) });
Expression epoch = Expression.Constant(Instant.FromDateTimeOffset(Epoch));
Expression expression = Expression.Call(context.Reader, readInteger);

// avoid losing nanosecond precision
expression = schema.LogicalType switch
{
NanosecondTimestampLogicalType =>
Expression.Call(epoch, plusNanoseconds, expression),
_ =>
Expression.Call(epoch, plusTicks, BuildTimestampToTicks(expression, schema)),
};

try
{
// Code: return NodaTime.Instant.FromDateTimeUtc( Epoch.AddTicks(value * factor) );
return BinaryDeserializerBuilderCaseResult.FromExpression(
BuildConversion(
Expression.Call(
fromDateTimeUtc!,
Expression.Call(
Expression.Constant(Epoch),
addTicks!,
Expression.Multiply(expression, Expression.Constant(factor)))),
type));
BuildConversion(expression, type));
}
catch (InvalidOperationException exception)
{
Expand All @@ -88,7 +85,8 @@ public override BinaryDeserializerBuilderCaseResult BuildExpression(Type type, S
}
else
{
return BinaryDeserializerBuilderCaseResult.FromException(new UnsupportedSchemaException(schema, $"{nameof(NodaTimeDeserializerBuilderCase)} can only be applied to schemas with a {nameof(TimestampLogicalType)} and NodaTime properties."));
return BinaryDeserializerBuilderCaseResult.FromException(
new UnsupportedSchemaException(schema, $"{nameof(NodaTimeDeserializerBuilderCase)} can only be applied to {nameof(StringSchema)}s or schemas with a {nameof(TimestampLogicalType)}."));
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ namespace Chr.Avro.NodaTimeExample.Infrastructure
{
using System;
using Chr.Avro.Abstract;
using NodaTime;

public class NodaTimeSchemaBuilderCase : SchemaBuilderCase, ISchemaBuilderCase
{
Expand All @@ -15,7 +16,7 @@ public NodaTimeSchemaBuilderCase(TemporalBehavior temporalBehavior)
public SchemaBuilderCaseResult BuildSchema(Type type, SchemaBuilderContext context)
{
// Handle NodaTime.Instant like the TimestampSchemaBuilderCase
if (type == typeof(NodaTime.Instant))
if (type == typeof(Instant))
{
Schema timestampSchema = TemporalBehavior switch
{
Expand All @@ -27,6 +28,10 @@ public SchemaBuilderCaseResult BuildSchema(Type type, SchemaBuilderContext conte
{
LogicalType = new MillisecondTimestampLogicalType(),
},
TemporalBehavior.EpochNanoseconds => new LongSchema()
{
LogicalType = new NanosecondTimestampLogicalType(),
},
TemporalBehavior.Iso8601 => new StringSchema(),
_ => throw new ArgumentOutOfRangeException(nameof(TemporalBehavior)),
};
Expand All @@ -45,7 +50,7 @@ public SchemaBuilderCaseResult BuildSchema(Type type, SchemaBuilderContext conte
}
else
{
return SchemaBuilderCaseResult.FromException(new UnsupportedTypeException(type, $"{nameof(TimestampSchemaBuilderCase)} can only be applied to the {nameof(NodaTime.Instant)} type."));
return SchemaBuilderCaseResult.FromException(new UnsupportedTypeException(type, $"{nameof(TimestampSchemaBuilderCase)} can only be applied to the {nameof(Instant)} type."));
}
}
}
Expand Down
Loading

0 comments on commit 504e0de

Please sign in to comment.