HIVE-28735: Propagated the exception properly from hash table-loader threads to main thread #5845

Paramvir109 · 2025-06-04T06:40:47Z

What changes were proposed in this pull request?

Need to properly propagate the exception which can occur in drainAndLoadForPartition method while parallel loading of fast Hash tables by the threads.
In the current scenario the thread is silently dying causing incorrect(lesser number)of HT entries to be reported which is giving incorrect results

Why are the changes needed?

It fixes bug caused by https://issues.apache.org/jira/browse/HIVE-25149 that hides actual exceptions and gives incorrect results. Example query is added in the jira : https://issues.apache.org/jira/browse/HIVE-28735

Is the change a dependency upgrade?

No

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tested using the sample query.

Hive 3

Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.expandAndRehash(VectorMapJoinFastLongHashTable.java:166)
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.add(VectorMapJoinFastLongHashTable.java:100)
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.adaptPutRow(VectorMapJoinFastLongHashTable.java:91)
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashMap.putRow(VectorMapJoinFastLongHashMap.java:147)
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:184)
    at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:130)
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:388)
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:457)
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:241)
    ... 3 more

Hive 4

Before the fix : Wrong results on each run
After the fix : Getting same OOM exception as in hive 3

Also added a unit test which verifies the same

…threads to main thread

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java

…r threads

…otected

deniskuzZ · 2025-06-11T05:00:54Z

...org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/TestVectorMapJoinFastHashTableLoader.java

+        vectorMapJoinFastHashTableLoader.initHTLoadingService(1048577);
+        List<CompletableFuture<Void>> loaderTasks = vectorMapJoinFastHashTableLoader.submitQueueDrainThreads(mockTableContainer);
+        assertEquals(2, loaderTasks.size());
+        vectorMapJoinFastHashTableLoader.getLoadExecService().shutdown();


i don't think that is a good way to repro the issue:

why prepare mock drainAndLoadForPartition and not even trigger it?

Instead, execute direct getLoadExecService().shutdown() and CompletableFuture.allOf.

your test should focus on ExecutionException and not the InterruptedException (based on PR description)

@deniskuzZ , when we call the submitQueueDrainThreads method - it will internally call drainAndLoadForPartition and this is what I want to test. If any error occurs in drainAndLoadForPartition, it will get caught and RuntimeException will be thrown which will be propagated as ExecutionException.
In my repro sceanrio the code flow was reaching till here but not being propagated properly.

here would be wrapped in ExecutionException.
Your test is checking for InterruptedException which was already covered in https://github.com/apache/hive/pull/5845/files#diff-573e1d79d59124631df039ba5fedfd037457f3e3c3b8fb5e349b4086191bc197L330

ExecutionException thrown = assertThrows(ExecutionException.class, () -> { CompletableFuture.allOf(loaderTasks.toArray(new CompletableFuture[0])) .get(2, TimeUnit.MINUTES); });

The above code block is checking for ExecutionException only

assertInstanceOf(InterruptedException.class, cause.getCause());

The above assertion is just extra assertion which tells the underlying cause of ExecutionException. I can remove it too

You should test the production code drainAndLoadForPartition, configure drainAndLoadForPartition to throw the exception, and not test the TEST

...org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/TestVectorMapJoinFastHashTableLoader.java

sonarqubecloud · 2025-06-12T19:14:28Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ · 2025-06-13T08:36:23Z

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java

          throw new HiveException("Failed to complete the hash table loader. Loading timed out.");
+        } catch (ExecutionException e) {
+          throw new HiveException("One of the loader threads failed", e.getCause());
+        } catch (InterruptedException e) {


you already have handling for InterruptedException and loadExecService shutdown here
no need to duplicate

please check the patch

Subject: [PATCH] cleanup --- Index: ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java --- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java (revision 7e356e06eee218e077586a52811a42449196152d) +++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java (date 1749805780973) @@ -88,10 +88,6 @@ } - ExecutorService getLoadExecService() { - return loadExecService; - } - public VectorMapJoinFastHashTableLoader(TezContext context, Configuration hconf, MapJoinOperator joinOp) { this.tezContext = context; this.hconf = hconf; @@ -114,7 +110,7 @@ this.htLoadCounter = tezContext.getTezProcessorContext().getCounters().findCounter(counterGroup, counterName); } - void initHTLoadingService(long estKeyCount) { + void initHTLoadingService(long estKeyCount) { if (estKeyCount < VectorMapJoinFastHashTable.FIRST_SIZE_UP) { // Avoid many small HTs that will rehash multiple times causing GCs this.numLoadThreads = 1; @@ -157,8 +153,8 @@ } } - List<CompletableFuture<Void>> submitQueueDrainThreads(VectorMapJoinFastTableContainer vectorMapJoinFastTableContainer) - throws InterruptedException, IOException, SerDeException { + List<CompletableFuture<Void>> submitQueueDrainThreads(VectorMapJoinFastTableContainer vectorMapJoinFastTableContainer) + throws InterruptedException { List<CompletableFuture<Void>> loaderTasks = new ArrayList<>(); for (int partitionId = 0; partitionId < numLoadThreads; partitionId++) { int finalPartitionId = partitionId; @@ -319,20 +315,11 @@ LOG.info("Finished loading the queue for input: {} waiting {} minutes for TPool shutdown", inputName, 2); addQueueDoneSentinel(); - loadExecService.shutdown(); try { CompletableFuture.allOf(loaderTasks.toArray(new CompletableFuture[0])) - .get(2, TimeUnit.MINUTES); - } catch (TimeoutException e) { - throw new HiveException("Failed to complete the hash table loader. Loading timed out."); - } catch (ExecutionException e) { - throw new HiveException("One of the loader threads failed", e.getCause()); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } finally { - if (loadExecService != null && !loadExecService.isTerminated()) { - loadExecService.shutdownNow(); - } + .get(2, TimeUnit.MINUTES); + } catch (ExecutionException | TimeoutException e) { + throw new HiveException("Failed to complete the hash table loader.", e.getCause()); } batchPool.clear(); LOG.info("Total received entries: {} Threads {} HT entries: {}", receivedEntries, numLoadThreads, totalEntries.get());

Ack, I'll update in the next revision

deniskuzZ · 2025-06-13T08:55:39Z

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java

@@ -83,6 +88,10 @@ public VectorMapJoinFastHashTableLoader() {

  }

+  ExecutorService getLoadExecService() {


remove this

I need this so I can actually shutdown the service in tearDown method after running unit test. Since loadExecService is a private member of the class, I thought to make a package-private getter for the same so that I can access it in unit test.
Other approach will be to change access modifier of loadExecService to package-private

drainAndLoadForPartition in the finally should shutdown the service

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java

HIVE-28735: Propagated the exception properly from hash table-loader …

0da6a43

…threads to main thread

asf-ci-hive added the tests pending label Jun 4, 2025

HIVE-28735: Propagated the exception properly from hash table-loader …

c80fe21

…threads to main thread

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Jun 4, 2025

HIVE-28735: Added unit tests

1789052

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Jun 6, 2025

deniskuzZ reviewed Jun 6, 2025

View reviewed changes

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java Outdated Show resolved Hide resolved

deniskuzZ reviewed Jun 6, 2025

View reviewed changes

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java Outdated Show resolved Hide resolved

Used CompletableFuture for collecting the work done via fast ht loade…

5bf9cea

…r threads

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Jun 8, 2025

Removed @VisibleForTesting methods and changed their visibility to pr…

a0d2fca

…otected

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending tests failed labels Jun 9, 2025

deniskuzZ reviewed Jun 11, 2025

View reviewed changes

...org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/TestVectorMapJoinFastHashTableLoader.java Show resolved Hide resolved

asf-ci-hive added tests failed and removed tests pending labels Jun 11, 2025

Made methods package-private instead of protected

7e356e0

asf-ci-hive added tests pending and removed tests failed labels Jun 12, 2025

asf-ci-hive added tests failed and removed tests pending labels Jun 13, 2025

deniskuzZ reviewed Jun 13, 2025

View reviewed changes

...ava/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HIVE-28735: Propagated the exception properly from hash table-loader threads to main thread #5845

HIVE-28735: Propagated the exception properly from hash table-loader threads to main thread #5845

Uh oh!

Paramvir109 commented Jun 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

deniskuzZ Jun 11, 2025 •

edited

Loading

Uh oh!

Paramvir109 Jun 11, 2025

Uh oh!

deniskuzZ Jun 13, 2025 •

edited

Loading

Uh oh!

Paramvir109 Jun 14, 2025 •

edited

Loading

Uh oh!

deniskuzZ Jun 16, 2025

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 12, 2025

Uh oh!

deniskuzZ Jun 13, 2025

Uh oh!

deniskuzZ Jun 13, 2025

Uh oh!

Paramvir109 Jun 14, 2025

Uh oh!

deniskuzZ Jun 13, 2025

Uh oh!

Paramvir109 Jun 14, 2025

Uh oh!

deniskuzZ Jun 16, 2025

Uh oh!

Uh oh!

Uh oh!

		@@ -83,6 +88,10 @@ public VectorMapJoinFastHashTableLoader() {

		}

		ExecutorService getLoadExecService() {

HIVE-28735: Propagated the exception properly from hash table-loader threads to main thread #5845

Are you sure you want to change the base?

HIVE-28735: Propagated the exception properly from hash table-loader threads to main thread #5845

Uh oh!

Conversation

Paramvir109 commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Is the change a dependency upgrade?

Does this PR introduce any user-facing change?

How was this patch tested?

Hive 3

Hive 4

Uh oh!

Uh oh!

Uh oh!

deniskuzZ Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Paramvir109 Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 12, 2025

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Paramvir109 commented Jun 4, 2025 •

edited

Loading

deniskuzZ Jun 11, 2025 •

edited

Loading

deniskuzZ Jun 13, 2025 •

edited

Loading

Paramvir109 Jun 14, 2025 •

edited

Loading