HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 #5846

kasakrisz · 2025-06-04T13:37:08Z

What changes were proposed in this pull request?

Backport patches from iceberg:

mr
+ ec2c2e978e40a4bf7bf6a3a0bccbe7e3a68a0cc1 mr：Fix ugi not correct in WORKER_POOL (#10661)
+ a5c8f9cd4557639d39eed716d499e9837d13b88e Build: Remove unused variables, fields and parameters (#11101)
+ 168a9839425f9d3e2518ed345b7847ecb7cabb89 ThreadPools introduce newExitingWorkerPool and newFixedThreadPool for clearer semantics (#11073)
+ 5e279c868f8e2087f88ae779d8cc8474768bc5e2 Build, Spark, Flink: Bump junit from 5.10.1 to 5.11.1 (#11262)
+ 0280885ac95bdf763556a84bb9d7c6fd9c8c5e2a Pig: Remove iceberg-pig (#11380)
+ b38951db6a7061a595605229c21c1a1912a3a4c1 Data, Flink, MR, Spark: Test deletes with format-version=3 (#11538)
+ da53495bc1bb52db37cdd1ced5c2377001c9d482 Core, Flink, Spark, KafkaConnect: Remove usage of deprecated path API (#11744)
+ b9b61b1d72ebb192d5e90453ff7030ece73d2603 Avro: Support default values for generic data (#11786)

hive-metastore
+ d17a7f189afa25c6be37df1415f4e2f8594effbe Core: Remove deprecated APIs for 1.7.0 (#10818)
+ cf02ffac4329141b30bca265cafb9987f64f6cc4 AWS, Core, Hive: Extract FileIO closing into separate FileIOTracker class (#10893)
+ e449d3405cfdb304c94835845bd8f34a73b4a517 Hive: Add View support for HIVE catalog (#9852)
+ 6a5ae1ae6a01f1395ee70e046537cd87b990c4ae Core: Switch usage to DataFileSet / DeleteFileSet (#11158)
+ 6e9e07aa0f35197bb23b218d23716e928d1c2814 Hive: Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError (#11576)
+ a95943e5561c78c18062852e7f8027a191562e08 Core: Propagate custom metrics reporter when table is created/replaced through Transaction (#11671)
+ da53495bc1bb52db37cdd1ced5c2377001c9d482 Core, Flink, Spark, KafkaConnect: Remove usage of deprecated path API (#11744)
+ a3dcfd19fd1b2a709f7bdf013b83836953d49c6f Hive: Optimize tableExists API in hive catalog (#11597)
+ e1d2271ad911d4224ad53ac2e0142b28984e5f0b Hive: Optimize viewExists API in hive catalog (#11813)
+ f129588461ad02c2fa2021af30b4e9bca70eee93 Core: Add support for view-default property in catalog (#11064)
- a05b2b53b792a6215732aa8b1118adfba0cf3317 Hive: Use correct classloader to load SQL script (#12140)
+ c02ebe4740b22d6f5a78b636aea2d918037b2751 Core: Set missing table-default property in RESTSessionCatalog (#11646)
+ d35cf23eb63cc88d4fdb6ae1db8e63630b073d13 Core: Add missing table-override property to REST catalog (#12548)
+ 8f6ebb5b36a0263edfcb04e0c104b26225f95b07 Core: Add `view-override` catalog property (#12534)
+ cbf34a6ab980ab9b7bb01e45c9e2bcfad8e2986b Build: Enforce error message check on Exception assertions (#12624)
+ c661a71091e496393c743ddd879d9e1a0f2747b2 Core, Hive: Double check commit status in case of commit conflict for NoLock (#12637)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Run existing and backported tests

deniskuzZ · 2025-06-04T16:25:39Z

@kasakrisz, please drop the following

from iceberg-handler
org.apache.iceberg.data.PartitionStatsHandler
org.apache.iceberg.data.TestPartitionStatsHandler
from patched-iceberg-core
org.apache.iceberg.avro.*
org.apache.iceberg.BaseScan.java
org.apache.iceberg.Partitioning.java
org.apache.iceberg.PartitionStats.java
org.apache.iceberg.PartitionStatsUtil.java
cleanup excludes in patched-iceberg-core pom (remove the dropped classes)

<excludes>
    **/HadoopInputFile.class
    **/HadoopTableOperations.class
    **/StructLikeMap.class
    **/StructLikeWrapper.class
    org.apache.iceberg.PartitionsTable.class
</excludes>

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java

deniskuzZ

LGTM, pending tests

okumin · 2025-06-05T12:48:36Z

iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveCatalog.java

 import org.apache.thrift.TException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

-public class HiveCatalog extends BaseMetastoreCatalog implements SupportsNamespaces, Configurable {
+public class HiveCatalog extends BaseMetastoreViewCatalog


This is so nice. The lack of view support is one of the bottlenecks when I try HIVE-28059 with other query engines.

@okumin, are you using RestCatalog locally? if yes, please suggest the default configs for the docker image: https://github.com/apache/hive/pull/5834/files#diff-cfa8481579367e1e9127939f845eba4d2ee8c796d6858bcd0ca0a4b5fbfb8019R35-R42

Yes, but I'm still halfway and I don't have very concrete suggestions. I have not successfully integrated it with other query engines.

Aggarwal-Raghav · 2025-06-09T16:08:50Z

Should we upgrade parquet to 1.15.2 just to be in sync with iceberg 1.9.1? Let me know I can raise the PR for the same.

kasakrisz · 2025-06-10T13:11:10Z

Should we upgrade parquet to 1.15.2 just to be in sync with iceberg 1.9.1? Let me know I can raise the PR for the same.

@Aggarwal-Raghav
Yes please go ahead.

…ted, only test was added

… clearer semantics (#11073)

…MemoryError (#11576)

…d through Transaction (#11671)

… (#11744)

… NoLock (#12637)

… execution TestHiveIcebergRestrictDataFiles.testRestrictDataFiles

…eaders/writers so we fall back to AVRO

sonarqubecloud · 2025-06-10T18:33:20Z

Quality Gate passed

Issues
88 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kasakrisz marked this pull request as draft June 4, 2025 13:37

asf-ci-hive added the tests pending label Jun 4, 2025

deniskuzZ reviewed Jun 4, 2025

View reviewed changes

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java Show resolved Hide resolved

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 4, 2025

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from fa9c656 to 16da407 Compare June 5, 2025 10:18

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 5, 2025

deniskuzZ approved these changes Jun 5, 2025

View reviewed changes

kasakrisz marked this pull request as ready for review June 5, 2025 11:44

okumin reviewed Jun 5, 2025

View reviewed changes

asf-ci-hive added tests failed and removed tests pending labels Jun 6, 2025

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from 16da407 to a7a1cbf Compare June 6, 2025 09:16

asf-ci-hive added tests pending tests failed tests unstable and removed tests failed tests pending labels Jun 6, 2025

mr：Fix ugi not correct in WORKER_POOL (#10661) - code already backpor…

dfdae41

…ted, only test was added

kasakrisz added 25 commits June 10, 2025 15:29

Build: Remove unused variables, fields and parameters (#11101)

dfbc90d

ThreadPools introduce newExitingWorkerPool and newFixedThreadPool for…

6dbf5d5

… clearer semantics (#11073)

Build, Spark, Flink: Bump junit from 5.10.1 to 5.11.1 (#11262)

19aadbb

Pig: Remove iceberg-pig (#11380)

486b662

Data, Flink, MR, Spark: Test deletes with format-version=3 (#11538)

7ca5723

bump iceberg version to 1.9.1

e8e73dd

Core: Remove deprecated APIs for 1.7.0 (#10818)

43e17d1

Hive: Add View support for HIVE catalog (#9852)

39cd1cd

Core: Switch usage to DataFileSet / DeleteFileSet (#11158)

b853bdf

Hive: Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOf…

1a57940

…MemoryError (#11576)

Core: Propagate custom metrics reporter when table is created/replace…

e762bbd

…d through Transaction (#11671)

Core, Flink, Spark, KafkaConnect: Remove usage of deprecated path API…

1339f52

… (#11744)

checkstyle

27e2b9b

Hive: Optimize tableExists API in hive catalog (#11597)

15ba7de

Core: Add support for view-default property in catalog (#11064)

01fff36

Core: Set missing table-default property in RESTSessionCatalog (#11646)

26b6b1d

Core: Add missing table-override property to REST catalog (#12548)

74dbe0b

Core: Add view-override catalog property (#12534)

73212c0

Build: Enforce error message check on Exception assertions (#12624)

3db9ba2

Core, Hive: Double check commit status in case of commit conflict for…

8b6509a

… NoLock (#12637)

remove copied classes in favor the originals

09c3b88

revert checkstyle module RegexpMultiline because it prevents the test…

1113454

… execution TestHiveIcebergRestrictDataFiles.testRestrictDataFiles

fix DeleteReadTests

fccaec4

fix TestHiveViewCommits.testInvalidObjectException

b31892c

copy PartitionStatsHandler because ORC is not supported in internal r…

931bdec

…eaders/writers so we fall back to AVRO

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from a7a1cbf to 931bdec Compare June 10, 2025 13:40

asf-ci-hive added tests pending and removed tests unstable labels Jun 10, 2025

Aggarwal-Raghav mentioned this pull request Jun 10, 2025

HIVE-29002: Upgrade parquet to 1.15.2 to be in sync with iceberg 1.9.1 #5858

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 #5846

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 #5846

Uh oh!

kasakrisz commented Jun 4, 2025

Uh oh!

deniskuzZ commented Jun 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

deniskuzZ left a comment

Uh oh!

okumin Jun 5, 2025

Uh oh!

deniskuzZ Jun 5, 2025 •

edited

Loading

Uh oh!

okumin Jun 5, 2025

Uh oh!

Aggarwal-Raghav commented Jun 9, 2025

Uh oh!

kasakrisz commented Jun 10, 2025 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Jun 10, 2025

Uh oh!

Uh oh!

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 #5846

Are you sure you want to change the base?

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 #5846

Uh oh!

Conversation

kasakrisz commented Jun 4, 2025

What changes were proposed in this pull request?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

deniskuzZ commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

deniskuzZ left a comment

Choose a reason for hiding this comment

Uh oh!

okumin Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jun 9, 2025

Uh oh!

kasakrisz commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 10, 2025

Quality Gate passed

Uh oh!

Uh oh!

deniskuzZ commented Jun 4, 2025 •

edited

Loading

deniskuzZ Jun 5, 2025 •

edited

Loading

kasakrisz commented Jun 10, 2025 •

edited

Loading