spark-operator: Documentation to integrate Spark Operator and Notebooks #4141
Conversation
|
Hi @fresende. Thanks for your PR. I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
🚫 This command cannot be processed. Only organization members or owners can use the commands. |
fresende
changed the title
fresende
force-pushed
the
doucmentation-spark
branch
from
7c38936 to
6495ba1
Compare
|
@fresende please sign your latest commits to fix the DCO CI issues. After installing Kubeflow's latest manifests and following the steps described above, the changes look good to me. I only encountered the issue mentioned in jupyter-server/enterprise_gateway#1405. While we wait for the new helm with the additional configuration, you should update your patch command with the suggested environment variable from the issue. |
| - Launch a notebook from the Kubeflow UI | ||
| - Select the `pyspark` kernel | ||
| - Write and run PySpark code | ||
| - Your notebook submits Spark jobs via JEG → Spark Operator → Kubernetes |
There was a problem hiding this comment.
Can you please add debugging using spark UI as well ?
content/en/docs/components/spark-operator/user-guide/notebooks-spark-operator.md
Show resolved
Hide resolved
Shekharrajak
left a comment
Shekharrajak
left a comment
There was a problem hiding this comment.
Let's merge this and later we can update any point.
Signed-off-by: Fellipe Resende <[email protected]>
Signed-off-by: Fellipe Resende <[email protected]>
Signed-off-by: Fellipe Resende <[email protected]>
Signed-off-by: Fellipe Resende <[email protected]>
fresende
force-pushed
the
doucmentation-spark
branch
from
65b3d95 to
c4740c1
Compare
andreyvelich
left a comment
andreyvelich
left a comment
There was a problem hiding this comment.
Thank you for this great content @fresende!
/assign @ChenYi015 @jacobsalway @vara-bonthu @yuchaoran2011 @ImpSy @nabuskey
Please can you check it ?
Signed-off-by: Fellipe Resende <[email protected]>
Signed-off-by: Fellipe Resende <[email protected]>
yuchaoran2011
left a comment
yuchaoran2011
left a comment
There was a problem hiding this comment.
Overall looks good. Left a minor comment. Thanks for the contribution
content/en/docs/components/spark-operator/user-guide/notebooks-spark-operator.md
Outdated
Show resolved
Hide resolved
Signed-off-by: Fellipe Resende <[email protected]>
|
/approve |
andreyvelich
left a comment
andreyvelich
left a comment
There was a problem hiding this comment.
Thanks for this awesome contribution @fresende!
/lgtm
/approve
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich, ChenYi015 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| image: elyra/kernel-image-puller:3.2.3 | ||
| imagePullPolicy: Always | ||
| pullPolicy: Always | ||
| defaultContainerRegistry: quay.io |
There was a problem hiding this comment.
Do we not need to add ingress here ? https://jupyter-enterprise-gateway.readthedocs.io/en/latest/operators/deploy-kubernetes.html#update-helm-deployment-to-enable-ingress
There was a problem hiding this comment.
We're currently setting the JUPYTER_GATEWAY_URL to the service address, which allows us to avoid enabling ingress. What is your current use case that requires ingress? @Shekharrajak
There was a problem hiding this comment.
Still thinking if this service will be available to call in another namespace:
kubectl -n enterprise-gateway get svc enterprise-gateway -o yaml
type: NodePort
7 participants
Description of your changes:
Added a new page that documents how to integrate Spark Operator and Notebooks