[Task]: Should run my hop pipeline on Beam Spark Pipeine Engine option without error

[Raja10D]

What needs to happen?

I need to run a Hop pipeline on Beam with Spark Runner

While I am executing this I'm encountering some errors. I will display it below, if somebody can execute this step without error, guide me how to do it?

The error I'm getting:

Copy code
2025/01/02 15:30:53 - Hop - Pipeline opened.
2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]...
2025/01/02 15:30:53 - Hop - Started the pipeline execution.
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/02 15:30:53 - Hop - Error preparing remote pipeline
2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam
2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/02 15:30:53 - Hop - at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the org.apache.spark.metrics.source.Source class not being found.

Issue Priority

Priority: 1

Issue Component

Component: Hop Run

[hansva]

duplicate of #4752