Native Trino Ion Format Integration #24511

rmarrowstone · 2024-12-18T01:14:40Z

Description

This PR is the implementation of a PageSourceFactory and FileWriterFactory for the Ion data format. It replaces the legacy Ion Hive SerDe. It enables trino users to read and write Ion datasets. Ion is a richly-typed superset of JSON with a well-defined binary encoding. It is widely used within Amazon, and we see some external usage as well.

Today the Ion Hive SerDe is used by Athena to provide access to Ion datasets. That Ion/Athena integration is used both by internal (to Amazon) teams and external customers. Newer releases of trino do not support the actual Hive SerDes. This implementation provides an Ion integration that will work in trino going forward.

The behavior is mostly backwards compatible with the Hive SerDe, with the exceptions listed below. There are a host of SerDe Properties supported by the Ion Hive SerDe. For this PR we are supporting a limited set of SerDe Properties: ones for which we have identified actual usage or assume are otherwise needed.

The new integration is disabled by default, behind a Hive Config flag: hive.ion.nativetrino our intention is to make that enabled by default at some point.

It has not been benchmarked. However, the performance of the existing Ion Hive SerDe is quite poor and this implementation intentionally avoids two of the primary causes of inefficiencies there.

We believe it is ready to be used for most tables.

Preserved Behaviors of Note

For the most part, mistyped values result in errors; e.g. an Ion String encountered where an Int is expected is an error.
Mistyped Top-Level-Values result in nulls, not errors. A new SerDe property ion.path_extractor.strict was added to address this, it defaults to false.
Duplicate keys in Structs/Rows: last one wins
There are some built-in and well-defined coercions, e.g. Ion Int to Trino Decimal, truncate an Ion Float (64 bits) to Trino REAL iff doing so is lossless.

Exceptions to Backwards Compatibility

Previously Timestamps were simply truncated. Now they are rounded. Athena users will not see a change as Timestamps will still be truncated there.
Previously Chars and Varchars longer than a fixed length were errors. Now they are truncated as others are.
Now Decimals with oversize whole digits (e.g. 1222.34 in a DECIMAL(3, 2)) are actually an error. Before the behavior was not defined.
Previously the ion.path_extractor.case_sensitive property affected nested row field names as well. That adds some complexity and seems borderline non-sensical to me: all trino schema is lower-cased. I'm also fairly sure that it was never used in conjunction with nested schema, so I'm comfortable making the saner choice going forward.

SerDe Properties

For reads this PR includes support for column-wise path_extractor properties and a table-wise case_sensitive property. In my searching that seems to be the most used of the SerDe Properties.

We added a ion.path_extractor.strict property. Setting this to true will ensure that mistyped top-level-values as well as any path extractions (looking for a field by name is implicitly expecting an Ion Struct) are hard errors (they are not in the Hive SerDe).

For writes this includes support for writing Ion as Binary or Text.

No other SerDe properties are supported, and Optional.empty... <todo: the null encoding behavior>.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X) Release notes are required, with the following suggested text:

## Section
* Ion File Format Integration

cla-bot · 2024-12-18T01:14:43Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-12-18T19:22:02Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-12-18T19:30:36Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-12-18T19:34:10Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-12-19T01:12:10Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-12-19T16:49:54Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

rmarrowstone · 2024-12-19T17:01:35Z

Aware of the CLA verification. I believe all 3 authors have at least submitted them at this point.

rmarrowstone · 2024-12-19T17:50:36Z

The suite-hive-transactional test failure appears related to ORC and transactions. I have not seen it when running locally or in the fork. I assume it is transient but I don't seem to have permissions to retry it.

findinpath · 2024-12-20T13:34:47Z

@rmarrowstone could you pls share in the description of the PR the business motivation for this contribution?

What kind of use-cases will the users of trinodb/trino be able to cover with this contribution?

findinpath · 2024-12-20T13:36:48Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java

+                // backwards incompatible. It requires explicitly opting to use this feature.
+                // TODO: In future this property should change to `true` as default and then the following statement can
+                //  be removed.
+                .addHiveProperty("hive.ion.nativetrino", "true")


Is the apparent low test coverage for the new functionality intentional?

I don't believe the test coverage for the new functionality actually is low.

We have added the TestIonFormat in lib/trino-hive-formats and the IonPageSourceSmokeTest as well the addition of testIonWithBinaryEncoding to TestHiveFileFormats. And that flag is enabled for them. I see that those tests do run when I build locally.

Can you point me to the artifacts that indicate there is low test coverage?

I did check that the TestHiveConnectorTest runs against the ION format and checked the coverage for our tests. There are a few gaps that we can and will address, but on the whole the coverage seems pretty robust.

With the latest commit in my IntelliJ I ran the unit tests in this PR and the BaseHiveConnectorTest and I see coverage of:
hive.formats.ion: 98% Method, 98% Line, and 95% Branch
plug.hive.ion: 96% Method, 94% Line, and 94% Branch

rmarrowstone · 2024-12-20T15:26:29Z

@rmarrowstone could you pls share in the description of the PR the business motivation for this contribution?

What kind of use-cases will the users of trinodb/trino be able to cover with this contribution?

Sure! I mentioned that this replaces the Ion Hive SerDe, which is used in Athena today. But using actual Hive Formats is no longer supported in newer trino releases. I will make that clearer.

zhaner08 · 2024-12-31T15:57:33Z

lib/trino-hive-formats/pom.xml

+        <dependency>
+            <groupId>com.amazon.ion</groupId>
+            <artifactId>ion-java</artifactId>
+            <version>1.11.9</version>


probably move the version to the main pom

zhaner08 · 2024-12-31T16:03:13Z

lib/trino-hive-formats/src/main/java/io/trino/hive/formats/HiveClassNames.java

+    public static final String ION_SERDE_CLASS = "com.amazon.ionhiveserde.IonHiveSerDe";
+    public static final String ION_INPUT_FORMAT = "com.amazon.ionhiveserde.formats.IonInputFormat";
+    public static final String ION_OUTPUT_FORMAT = "com.amazon.ionhiveserde.formats.IonOutputFormat";
+


Nit. Remove the extra line

zhaner08 · 2024-12-31T16:19:00Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ion/IonPageSourceFactory.java

+            boolean originalFile,
+            AcidTransaction transaction)
+    {
+        if (!this.nativeTrinoEnabled) {


This probably doesn’t make sense in oss trino as ion was never supported

Right... the goal was to facilitate roll-out to deployments using non-OSS trino. I will ping you offline to discuss.

zhaner08 · 2024-12-31T16:21:03Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ion/IonPageSourceFactory.java

+        }
+
+        if (schema.serdeProperties().entrySet().stream().filter(entry -> entry.getKey().startsWith("ion.")).anyMatch(this::isUnsupportedProperty)) {
+            return Optional.empty();


Would the better option to just ignore them?

We don't want user's to think a property is respected but is actually ignored. Returning an Optional.empty seemed a good way to indicate that we can't produce the result sets the user is expecting for the Table.

The key idea being that this implementation and the legacy hive impl could be used in tandem in some deployments.

zhaner08 · 2024-12-31T16:33:12Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ion/IonPageSource.java

+    @Override
+    public long getReadTimeNanos()
+    {
+        return 0;


Is this intentional?

Somewhat? I see that returning 0 is valid when there is no known read time. We can look at getting an actual estimate here.

I found that I could use the TrinoDataInputStream and get this in a way that's consistent with other connectors.

zhaner08 · 2025-01-02T07:15:35Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ion/IonFileWriter.java

+            writer.close();
+        }
+        catch (IOException e) {
+            throw new TrinoException(HIVE_WRITER_CLOSE_ERROR, "Error rolling back write to Hive", e);


Yea that's confusing. FWIW it's what the LineFileWriter does today as well.

zhaner08 · 2025-01-02T07:18:37Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveStorageFormat.java

@@ -135,6 +142,7 @@ public boolean isSplittable(String path)
        return switch (this) {
            case ORC, PARQUET, AVRO, RCBINARY, RCTEXT, SEQUENCEFILE -> true;
            case JSON, OPENX_JSON, TEXTFILE, CSV, REGEX -> CompressionKind.forFile(path).isEmpty();
+            case ION -> false;


Is this true?

It is true for the implementation in this PR. It is also true that you can't merely seek to some point in an Ion file and start reading. I do not know enough about how the PageSources communicate where and how to split to trino to answer whether we could or could not.

For example, the existing Hive SerDe does "split" in a sense, but how it does that is to effectively de-then-re-serialize each value in the stream. I'm pretty sure the cost of that isn't worth it.

Confirmed that splitting as Trino does for the Hive Formats won't work for Ion. FWIW some of the biggest users of storing Ion in S3 are gzipping anyway.

zhaner08 · 2025-01-02T07:26:36Z

lib/trino-hive-formats/src/main/java/io/trino/hive/formats/ion/IonDecoderFactory.java

+                rowDecoder.appendNulls(blockSelector);
+            }
+            else {
+                throw new TrinoException(StandardErrorCode.GENERIC_USER_ERROR,


Feel there is better error code for this

I didn't see a clearly better one in the existing error code and the other hive formats are using a mish-mash of runtime exceptions for errors. We attempted to align the errors we threw on TrinoExceptions, but I'm open to suggestions here!

This is the initial implementation of a Native Trino Ion Hive format. It does not handle the full range of Trino or Ion values, nor the set of coercions that we will likely need to implement. But I think it's a pretty good start. I implemented enough of the scalar types to make the tests pass but there are definitely some edges that need cleaning. I also added the ion-hive-serde as a test dependency so that we can add Ion to TestHiveFileFormats. I chose to avoid building off of the "Line" abstractions. While that could be done, and more closely mimics how the Ion Hive SerDe works, doing so is suboptimal, and for reading is actually more complicated. This is because while Ion is unschema'ed, each value stream has an evolving "encoding context." In Ion 1.0 that is the "Symbol Table" which is a form of dictionary encoding for the field names and other common text values. The other high-level choice was _not_ to use the ion-java-path-extraction library that is used in the Hive SerDe. The main value in that was to allow users to do path navigation in the SerDe config. But using it to go into the Block abstractions without a middle layer (as it uses in Hive SerDe) is complex, and I'm concerned that some data quality issues are effectively silenced. Given Trino's current capability set with nested and complex structures I doubt the value proposition. In fact, using SerDe config for what can be expressed in SQL is actually an anti-pattern.

Converts the field name into lower case for both Decoder and Encoder to ensure case-insensitivity for field names/keys

* Adds `ion.nativetrino` hvie config feature flag * Adds changes for Ion page source to use the feature flag * Adds test changes for feature flag

This change factors all of the setup involved in creating PageSources and FileWriters into a single TestFixture inner class. This reduces some existing duplication and should make it simpler to add/change table properties and hive config in the test.

There are a few changes related to how mistyped or oversize Ion Values are Handled: * Enable Strict or Lax Path Typing This change adds a SerDe Property to support non-strict path typing. It defaults to false to mimic the behavior of the ion-hive-serde which used path extraction for the top-level-values, whether the user had defined any extractions or not. Without support for pathing this is effectively a type-check (or not) for TLVs. With support for extraction the behavior is a little more subtle than that, so I named the property for how it will behave. The name is also consistent with other properties. I also added some tests for nested mistypings and changed the code to consistently throw TrinoExceptions. * IonInts are coerced to Decimals (as Ion Hive does) * Decimals with more whole digits than fit is now an error (oddly this is not an error in Ion Hive) Fairly exhaustive test cases were added for each of the changes. * Ensure Timestamps are Rounded Consistently This change makes it so that Ion Timestamps are rounded using Timestamps.round(). This makes it so that Timestamp down casting works consistently with the other formats. I considered using DecodedTimestamp and the TrinoTimestampEncoders but those didn't cover picos. So this code uses the same pattern they do and the Timestamps.round(). This code simply ignores anything after picos. * Truncate Char and Varchar Columns With this change we truncate text that is longer than the length of the Char or Varchar column. Before this change that caused an error. Note that this is an error with the Hive Serde, but per the public Athena docs and the behavior of the other Hive formats, truncation is preferred.

This commit fixes up a few things I noticed when previewing the PR to Trino. * Column/Field name casing should be preserved when writing * Some missing operational/metrics calls in IonPageSource * Throw clearer Exception for errors in IonFileWriter * Move some tests from TestHiveFileFormats to IonPageSourceSmokeTest * Add test for Timestamp Encoding

This change implements support for path extraction SerDe Properties. It uses the same ion-java-path-extraction library the Ion Hive SerDe does. Unlike the Hive SerDe, this ensures that the "strict" and more performant path extraction implementation is used. I chose to use the path-extraction in the absence of any defined path extractors. When a path extractor is defined, you have to define all columns as extractions. With the strict implementation, the field lookup is effectively the same as the Decoder here. So given that I would rather cut modality unless there's a really compelling reason.

This commit gives us complete coverage for the IonDecoderFactory, except for the default branch we need to make the compiler happy. In the process of covering both Char and Varchar Types for Map Keys I discovered we weren't truncating the keys so I fixed that.

This commit addresses comments on the PR to the main trino repo.

…ted Props Before this change the FileWriterFactory was not checking if hive.ion.nativetrino is True, nor if there are non-default values for unimplemented SerDe properties. This change fixes that. I combined IonReaderOptions and IonWriterOptions into a single IonSerDeProperties to avoid code duplication around the unimplemented check.

github-actions bot added the hive Hive connector label Dec 18, 2024

rmarrowstone force-pushed the master branch from 30cdf9e to 2e67934 Compare December 18, 2024 19:30

rmarrowstone force-pushed the master branch from 2e67934 to 0d593c2 Compare December 18, 2024 19:34

rmarrowstone force-pushed the master branch from 56bf620 to f18f95d Compare December 19, 2024 16:49

rmarrowstone marked this pull request as ready for review December 19, 2024 16:50

findinpath reviewed Dec 20, 2024

View reviewed changes

zhaner08 reviewed Jan 2, 2025

View reviewed changes

cla-bot bot added the cla-signed label Jan 14, 2025

rmarrowstone force-pushed the master branch 2 times, most recently from bba2adc to 6241d83 Compare January 14, 2025 22:08

rmarrowstone and others added 10 commits January 15, 2025 11:17

Adds support for case-insensitivity for field names

f8d6184

Converts the field name into lower case for both Decoder and Encoder to ensure case-insensitivity for field names/keys

Adds HiveConfig fetaure flag for Ion native trino integration

98f9cc9

* Adds `ion.nativetrino` hvie config feature flag * Adds changes for Ion page source to use the feature flag * Adds test changes for feature flag

Add support of configurable Ion encoding format

4ee0ba3

Refactor IonPageSoureSmokeTest

02813f0

This change factors all of the setup involved in creating PageSources and FileWriters into a single TestFixture inner class. This reduces some existing duplication and should make it simpler to add/change table properties and hive config in the test.

Omit null fields instead of encoding them as null

66d6b17

Adds support for map type with string key

4c02014

Return empty PageSource when there's unsupported Serde property

d09a6e9

desaikd and others added 6 commits January 15, 2025 11:17

Adds support to string coercions

135714e

Adds encoding tests

f10e06a

Trino PR Fixups

5af2d2e

This commit addresses comments on the PR to the main trino repo.

rmarrowstone force-pushed the master branch from f069d41 to 105e9ae Compare January 15, 2025 19:18

rmarrowstone mentioned this pull request Jan 15, 2025

Provide Access to Reader's Current Memory Usage amazon-ion/ion-java#1031

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native Trino Ion Format Integration #24511

Native Trino Ion Format Integration #24511

rmarrowstone commented Dec 18, 2024 •

edited

Loading

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 19, 2024

cla-bot bot commented Dec 19, 2024

rmarrowstone commented Dec 19, 2024

rmarrowstone commented Dec 19, 2024

findinpath commented Dec 20, 2024

findinpath Dec 20, 2024

rmarrowstone Dec 20, 2024

rmarrowstone Dec 20, 2024

rmarrowstone Jan 15, 2025

rmarrowstone commented Dec 20, 2024

zhaner08 Dec 31, 2024

zhaner08 Dec 31, 2024

zhaner08 Dec 31, 2024

rmarrowstone Jan 3, 2025

zhaner08 Dec 31, 2024

rmarrowstone Jan 3, 2025

zhaner08 Dec 31, 2024

rmarrowstone Jan 3, 2025

rmarrowstone Jan 15, 2025

zhaner08 Jan 2, 2025

rmarrowstone Jan 3, 2025

zhaner08 Jan 2, 2025

rmarrowstone Jan 3, 2025

rmarrowstone Jan 9, 2025

zhaner08 Jan 2, 2025

rmarrowstone Jan 3, 2025

Native Trino Ion Format Integration #24511

Are you sure you want to change the base?

Native Trino Ion Format Integration #24511

Conversation

rmarrowstone commented Dec 18, 2024 • edited Loading

Description

Preserved Behaviors of Note

Exceptions to Backwards Compatibility

SerDe Properties

Release notes

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 18, 2024

cla-bot bot commented Dec 19, 2024

cla-bot bot commented Dec 19, 2024

rmarrowstone commented Dec 19, 2024

rmarrowstone commented Dec 19, 2024

findinpath commented Dec 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmarrowstone commented Dec 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmarrowstone commented Dec 18, 2024 •

edited

Loading