Don't show spark UI and Application ID when using Spark 3.5.0

[duc-dn]

Hi @askhatri, I built a snapshot master branch of LIVY with spark 3.5.0
After I replace them in helm chart, it worked well. However, when I checked LIVY UI, it doesn't show spark ui and application id.
I checked endpoint get /sessions, seem to not get info of spark session

{
    "id": 1,
    "name": null,
    "appId": null,
    "owner": null,
    "proxyUser": null,
    "state": "idle",
    "kind": "pyspark",
    "appInfo": {
        "driverLogUrl": null,
        "sparkUiUrl": null
    },
    "log": [
        "24/07/12 08:39:50 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:51 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:52 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:53 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:54 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:55 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:56 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:58 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)",
        "24/07/12 08:39:59 INFO LoggingPodStatusWatcherImpl: Application status for spark-c421e5052f844570bc681a897692dd91 (phase: Running)"
    ],
    "ttl": null,
    "idleTimeout": null,
    "driverMemory": null,
    "driverCores": 0,
    "executorMemory": null,
    "executorCores": 0,
    "archives": [],
    "files": [],
    "heartbeatTimeoutInSecond": 0,
    "jars": [],
    "numExecutors": 0,
    "pyFiles": [],
    "queue": null
}

I forward port of spark driver and can see spark UI of session.
In addition, when deploying with your image, I still see sparkUI and applicationID (build image following in Docker.md)
So Is this due to Livy Server (currently) not being compatible with Spark 3.5.0?
Thanks

[askhatri]

Hi @duc-dn , Which version of JRE/JDK are you using? Livy Server is currently not compatible with JDK 17. Spark 3.5.0 Docker images are built using JDK 17, and Livy is not compatible with JDK 17.

[duc-dn]

Below image is the version that I built snapshot LIVY

I only change versions of SPARK and HADOOP in pom file
And spark image is: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu
I find that they are the same java version

[askhatri]

Okay, I will test with Spark 3.5.0 and share my findings.

[duc-dn]

Hi again. Do you have any updates about above problem?
Besides, I have a curious problem. I find docs in Livy

By default Livy is built against Apache Spark 2.4.5, but the version of Spark used when running Livy does not need to match the version used to build Livy. 
Livy internally handles the differences between different Spark versions.
The Livy package itself does not contain a Spark distribution. It will work with any supported version of Spark without needing to rebuild.

So follow you, when building Livy with spark 3.2.3, can it run with Spark 3.5.0?
I tried running basic command pyspark with Livy image: asifkhatri/livy:spark3.2.3, spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu.
It works well and I can see spark ui link in LIVY UI.

[askhatri]

Hi @duc-dn , I am still working on it. Livy image: asifkhatri/livy:spark3.2.3 with spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu might work. We need to build a new Livy image with spark 3.5.0 for spark: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu.

[askhatri]

I have updated the Helm chart to support Spark 3.5.0.

You can use this repository for testing.

To build your own Docker image, refer to this commit in the incubator-livy repository.

Use the following command to build the Livy code:

mvn -e -Pthriftserver -Pscala-2.12 -Pspark3 -Phadoop3 -pl -:livy-python-api clean install

For the Docker image, use the following Dockerfile:

FROM apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu

ENV LIVY_VERSION=0.9.0-incubating-SNAPSHOT
ENV LIVY_PACKAGE=apache-livy-${LIVY_VERSION}_2.12-bin
ENV LIVY_HOME=/opt/livy
ENV LIVY_CONF_DIR=/conf
ENV PATH=$PATH:$LIVY_HOME/bin

USER root

COPY $LIVY_PACKAGE.zip /

RUN apt-get update && apt-get install -y unzip && \
    unzip /$LIVY_PACKAGE.zip -d / && \
    mv /$LIVY_PACKAGE /opt/ && \
    rm -rf $LIVY_HOME && \
    ln -s /opt/$LIVY_PACKAGE $LIVY_HOME && \
    rm -f /$LIVY_PACKAGE.zip

RUN mkdir /var/log/livy && \
    ln -s /var/log/livy $LIVY_HOME/logs

WORKDIR $LIVY_HOME

ENTRYPOINT ["livy-server"]

[duc-dn]

Hi @askhatri, I cloned this branch: https://github.com/askhatri/incubator-livy/commit/980a78db4ccae6e44afa960762b266e65ac38c4a and built in local using command: mvn -e -Pthriftserver -Pscala-2.12 -Pspark3 -Phadoop3 -pl -:livy-python-api clean install
However, I am facing errors:

[ERROR] Tests run: 19, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 29.218 s <<< FAILURE! - in org.apache.livy.rsc.TestSparkClient
[ERROR] testSparkSQLJob(org.apache.livy.rsc.TestSparkClient)  Time elapsed: 5.263 s  <<< ERROR!
java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.13 executor driver): java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/StreamReadConstraints
        at org.apache.spark.sql.catalyst.json.JSONOptions.buildJsonFactory(JSONOptions.scala:194)
        at org.apache.spark.sql.catalyst.json.JsonInferSchema.$anonfun$infer$1(JsonInferSchema.scala:83)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)

[askhatri]

Hi @duc-dn , It appears that there are some test failures. I will work on fixing them. In the meantime, you can use the -DskipITs -DskipTests flag to skip the tests as a workaround.

[askhatri]

Hi @duc-dn, I have tested the build and found that there is a build error. You can check the detailed logs at the following link: log link.

[duc-dn]

@askhatri, which version mvn and java do you use to build LIVY?
I'm using mvn 3.6.3, java 11.0.23. If not skipping testing, when testing with spark client, I faced errors in the log file
org.apache.livy.rsc.TestSparkClient-output.txt

[askhatri]

I am using "Apache Maven 3.3.9" and "Java version: 1.8.0_292". Same configuration that is used at https://github.com/apache/incubator-livy/actions/runs/9871334539/job/27259040026.

[duc-dn]

thanks @askhatri, let me check

[duc-dn]

@askhatri, I changed mvn 3.3.9 java1.8.0 but I faced same above error

java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/StreamReadConstraints

So is this problem due to the version of jackson?? I don't know why you don't face this problem
I tried changing jackson version to 2.15.2 following on StackOverflow
log.txt

[askhatri]

I'm not sure about the Jackson version. This is new to me.

[duc-dn]

Hi @askhatri, I found your latest update. Thanks
I still have the problem with auth livy from spark magic using LDAP.
I added additional config in livy.conf (configmap)

livy.server.auth.type=ldap
livy.server.auth.ldap.url=ldap://localhost:389
livy.server.auth.ldap.base-dn = ou=livy,dc=example,dc=org
livy.server.auth.ldap.username-domain = example.org
livy.server.auth.ldap.enable-start-tls=false
livy.server.auth.ldap.security-authentication=simple

In log of livy server. I don't see raise any error about LDAP, but livy server is restarted continuously

24/07/26 09:52:30 INFO LivyServer: LDAP auth enabled.
24/07/26 09:52:30 INFO WebServer: Starting server on http://livytest-livycluster-0.livytest-livycluster-headless.livy.svc.cluster.local:8998
24/07/26 09:52:31 DEBUG IOStatisticsContextIntegration: Created instance IOStatisticsContextImpl{id=3, threadId=28, ioStatistics=counters=();
gauges=();
minimums=();
maximums=();
means=();
}
24/07/26 09:52:35 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:35 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:38 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:38 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:45 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:45 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:46 DEBUG SparkKubernetesApp: getApplicationReport, applicationId: spark-0f59c651679748de9444a35a32fd1cd7, namespace: livy applicationTag: livy-batch-2-82hd2jb1
24/07/26 09:52:46 DEBUG DefaultSharedIndexInformer: Ready to run resync and reflector for v1/namespaces/livy/pods with resync 0
24/07/26 09:52:46 DEBUG DefaultSharedIndexInformer: Resync skipped due to 0 full resync period for v1/namespaces/livy/pods
24/07/26 09:52:46 DEBUG Reflector: Listing items (1) for v1/namespaces/livy/pods at v762663
24/07/26 09:52:46 DEBUG BatchSession: BatchSession 2 state changed from STARTING to FINISHED
24/07/26 09:52:46 DEBUG SparkKubernetesApp: spark-0f59c651679748de9444a35a32fd1cd7 FINISHED testbatch-a397cf90ee3f8789-driver.livy: 	node: docker-desktop 	hostname: null 	podIp: null 	startTime: 2024-07-26T08:54:20Z 	phase: Succeeded 	reason: null 	message: null 	labels: name=driver, spark-app-name=testbatch, spark-version=3.5.0, spark-role=driver, spark-app-selector=spark-0f59c651679748de9444a35a32fd1cd7, spark-app-tag=livy-batch-2-82hd2jb1 	containers: 		spark-kubernetes-driver: 			image: apache/spark:3.5.0-scala2.12-java11-python3-r-ubuntu 			requests: cpu=1, memory=1408Mi 			limits: memory=1408Mi 			command: [] [driver, --properties-file, /opt/spark/conf/spark.properties, --class, org.apache.spark.examples.SparkPi, local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar, 10000] 	conditions: 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:55Z, message=null, reason=null, status=False, type=PodReadyToStartContainers, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:54:20Z, message=null, reason=PodCompleted, status=True, type=Initialized, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:54Z, message=null, reason=PodCompleted, status=False, type=Ready, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:55:54Z, message=null, reason=PodCompleted, status=False, type=ContainersReady, additionalProperties={}) 		PodCondition(lastProbeTime=null, lastTransitionTime=2024-07-26T08:54:20Z, message=null, reason=null, status=True, type=PodScheduled, additionalProperties={})
24/07/26 09:52:48 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:48 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:55 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 DEBUG AuthenticationFilter: Got token null from httpRequest http://10.1.8.205:8998/version
24/07/26 09:52:55 DEBUG AuthenticationFilter: Request [http://10.1.8.205:8998/version] triggering authentication. handler: class org.apache.livy.server.auth.LdapAuthenticationHandlerImpl
24/07/26 09:52:55 INFO LivyServer: Shutting down Livy server.
24/07/26 09:52:55 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:529)); Key: (root (auth:SIMPLE))@file://; URI: file:///; Object Identity Hash: e82c2ff
24/07/26 09:52:55 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.RawLocalFileSystem.close(RawLocalFileSystem.java:895)); Key: null; URI: file:///; Object Identity Hash: 453d5f
24/07/26 09:52:55 DEBUG ShutdownHookManager: Completed shutdown in 0.004 seconds; Timeouts: 0
24/07/26 09:52:55 DEBUG ShutdownHookManager: ShutdownHookManager completed shutdown.