I am reaching out to seek your assistance with an issue I am encountering while running a simple Hop pipeline using the Beam Spark pipeline engine. The pipeline executes without any issues when using the default local runner, but it fails when I switch to the Beam Spark engine option. I want to run hop pipeline on Beam with Spark runner it can be either using hop gui or with spark-submit command.

[Raja10D]

What needs to happen?

Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.1
Apache Hop Version: apache-hop-client-2.11.0
The error logs are as follows:

yaml
Copy code
2025/01/02 15:30:53 - Hop - Pipeline opened.
2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]...
2025/01/02 15:30:53 - Hop - Started the pipeline execution.
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/02 15:30:53 - Hop - Error preparing remote pipeline
2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam
2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/02 15:30:53 - Hop - at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the org.apache.spark.metrics.source.Source class not being found.

I want to know whether the task can be completed with the above versions. if yes tell me how to do it?
if not what versions I can do?

Issue Priority

Priority: 1

Issue Component

Component: Actions

[hansva]

Please stop opening new tickets about the same subject

[hansva]

Duplicate of #4752

[Raja10D]

Please stop opening new tickets about the same subject

Ok, sure
Whether the task I have mentioned can be done or not