Support for Pyspark kernel (kernel may hang forever)

[chenshenwei1998]

Environment data

• VS Code version: 1.92.2 (Universal) fee1edb8d6d72a0ddff41e5f71a671c23ed924b9
• Jupyter Extension version (available under the Extensions sidebar): v2024.7.0
• Python Extension version (available under the Extensions sidebar): v2024.12.3
• OS (Windows | Mac | Linux distro) and version: MacOS 14.3.1 (23D60)
• Python and/or Anaconda version: Python 3.8.19
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Local

Expected behaviour

pysparkkernel (sparkmagic) respond normally like other kernels

Actual behaviour

After starting pysparkkernel and executing the first cell, the kernel may hang for a long time. Even if the first cell responds after several minutes, the execution of second cell may hang forever.

I observe the kernel logs and notice that when vscode first execute a cell, it will send some initial code like import sys as _VSCODE_sys\nprint(_VSCODE_sys.executable); del _VSCODE_sys to kernel. However, in pysparkkernel, executing common python code equals to sending code to remote livy server. When there is no spark session exists, the pysparkkernel will start a spark session. Hence, the initial code is wrongly recognized as user-executed code, then pysparkkernel begins to start spark session, which explains why kernel hangs for several minutes (kernel sends execution reply until spark session is ready). After that, the communication between vscode-jupyter and kernel becomes very unstable, sometimes cells can be executed, sometimes it will hang forever (maybe some channel is in wrong state because of long initializing time or incorrect kernel reply?). In contrast, jupyter lab works fine with pysparkkernel because it does not send initial code to kernel.

I patched the pysparkkernel and let it execute initial code sent by vscode jupyter locally, then everything works fine.

I think this is a compatibility issue between vscode-jupyter and pysparkkernel, initial code should not be sent to pysparkkernel.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Installing sparkmagic and pysparkkernel
2. Choose pysparkkernel in vscode (need to fill in pysparkkernel config, see sparkmagic project)
3. Execute code in vsocde.

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

23:16:09.233 [trace] Conda file is conda
23:16:11.109 [debug] Get Custom Env Variables, Class name = Sm, completed in 0ms, has a truthy return value, Arg 1: undefined, Arg 2: "RunPythonCode"
23:16:11.116 [debug] KernelProvider switched kernel to id = .undefined.sparkmagic.kernels.pysparkkernel.pysparkkernel./Users/~/miniforge3/envs/sparkmagictest/python./Users/~/miniforge3/envs/sparkmagictest/python.-m#sparkmagic.kernels.pysparkkernel.pysparkkernel
23:16:11.116 [debug] start the kernel, options.disableUI=true for ~/Desktop/livy-emr.ipynb
23:16:11.127 [trace] Registering commtarget jupyter.widget
23:16:11.128 [debug] Controller selection change completed
23:16:11.128 [info] Starting Kernel (Python Path: ~/miniforge3/envs/sparkmagictest/bin/python, Conda, 3.8.19) for '~/Desktop/livy-emr.ipynb' (disableUI=true)
23:16:11.128 [trace] Creating raw notebook for resource '~/Desktop/livy-emr.ipynb'
23:16:11.131 [debug] Get Custom Env Variables, Class name = Sm, completed in 1ms, has a truthy return value, Arg 1: "~/Desktop/livy-emr.ipynb", Arg 2: "RunPythonCode"
23:16:11.138 [debug] Launching kernel .undefined.sparkmagic.kernels.pysparkkernel.pysparkkernel./Users/~/miniforge3/envs/sparkmagictest/python./Users/~/miniforge3/envs/sparkmagictest/python.-m#sparkmagic.kernels.pysparkkernel.pysparkkernel for ~/Desktop/livy-emr.ipynb in ~/Desktop with ports 9001, 9000, 9004, 9002, 9003
23:16:11.138 [info] Launching Raw Kernel PySpark # /Users/~/miniforge3/envs/sparkmagictest/bin/python
23:16:11.138 [debug] Create ProcessService, Class name = Im (started execution), Arg 1: "~/Desktop/livy-emr.ipynb", Arg 2: {"a":false,"b":{"z":3,"w":{"i":-1,"end":0},"u":[{"id":2044},{"id":2045},null,{"id":2048}]}}
23:16:11.138 [debug] Get Custom Env Variables, Class name = Sm, completed in 0ms, has a truthy return value, Arg 1: "~/Desktop/livy-emr.ipynb", Arg 2: "RunNonPythonCode", Arg 3: {"a":false,"b":{"z":3,"w":{"i":-1,"end":0},"u":[{"id":2044},{"id":2045},null,{"id":2048}]}}
23:16:11.139 [debug] Create ProcessService, Class name = Im, completed in 1ms, has a truthy return value, Arg 1: "~/Desktop/livy-emr.ipynb", Arg 2: {"a":false,"b":{"z":3,"w":{"i":-1,"end":0},"u":[{"id":2044},{"id":2045},null,{"id":2048}]}}
23:16:11.968 [debug] Got env vars from Python Ext in 838ms for ~/miniforge3/envs/sparkmagictest/bin/python, with env var count 62.
23:16:11.968 [trace] Prepend PATH with python bin for ~/miniforge3/envs/sparkmagictest/bin/python
23:16:11.968 [debug] Getting activated env variables, Class name = pd, completed in 838ms, has a truthy return value, Arg 1: "~/Desktop/livy-emr.ipynb", Arg 2: "~/miniforge3/envs/sparkmagictest/bin/python", Arg 3: undefined
23:16:11.969 [info] Process Execution: ~/miniforge3/envs/sparkmagictest/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
23:16:11.970 [info] Process Execution: ~/miniforge3/envs/sparkmagictest/bin/python -m sparkmagic.kernels.pysparkkernel.pysparkkernel -f /Users/~/Library/Jupyter/runtime/kernel-v2-57879km7dCukFoby5.json
    > cwd: ~/Desktop
23:16:11.970 [debug] Kernel process 58066.
23:16:11.972 [debug] Got env vars from Python Ext in 863ms for ~/miniforge3/envs/sparkmagictest/bin/python, with env var count 62.
23:16:11.972 [trace] Prepend PATH with python bin for ~/miniforge3/envs/sparkmagictest/bin/python
23:16:11.972 [debug] Getting activated env variables, Class name = pd, completed in 863ms, has a truthy return value, Arg 1: undefined, Arg 2: "~/miniforge3/envs/sparkmagictest/bin/python", Arg 3: undefined
23:16:11.972 [info] Process Execution: ~/miniforge3/envs/sparkmagictest/bin/python -m pip list
23:16:12.208 [trace] ipykernel version & path 6.29.5, ~/miniforge3/envs/sparkmagictest/lib/python3.8/site-packages/ipykernel/__init__.py for ~/miniforge3/envs/sparkmagictest/bin/python
23:16:12.569 [debug] Handle Execution of Cells 0 for ~/Desktop/livy-emr.ipynb
23:16:13.570 [debug] Kernel output 58066: To connect another client to this kernel, use:
--existing kernel-v2-57879km7dCukFoby5.json
23:16:13.599 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{},"header":{"date":"2024-08-31T15:16:13.599Z","msg_id":"c190b899-c532-49c4-86bf-bb637c48412e","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:16:13.599 [debug] Waiting for Raw Session to be ready in postStartRawSession
23:16:13.599 [debug] Waiting for Raw session to be ready, status: connected
23:16:13.599 [trace] Raw session connected
23:16:13.599 [debug] Waited for Raw session to be ready & got status: connected
23:16:13.600 [debug] Successfully waited for Raw Session to be ready in postStartRawSession
23:16:13.600 [debug] Kernel status is 'unknown' before requesting kernel info and after ready
23:16:13.600 [debug] Sending request for kernelInfo
23:16:13.600 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"control","content":{},"header":{"date":"2024-08-31T15:16:13.600Z","msg_id":"578714bf-8878-47e3-8081-19f5bc608e5a","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:16:13.600 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{},"header":{"date":"2024-08-31T15:16:13.600Z","msg_id":"2c0fedf7-7e50-4019-a630-451777bf34b6","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:16:13.779 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_0","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.778702Z","version":"5.3"},"parent_header":{},"metadata":{},"content":{"execution_state":"starting"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.781 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_1","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.780216Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.599000Z","msg_id":"c190b899-c532-49c4-86bf-bb637c48412e","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.781 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_3","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.780830Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"578714bf-8878-47e3-8081-19f5bc608e5a","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.781 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_5","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.781315Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.599000Z","msg_id":"c190b899-c532-49c4-86bf-bb637c48412e","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.782 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_6","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.781565Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"578714bf-8878-47e3-8081-19f5bc608e5a","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.782 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_2","msg_type":"kernel_info_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.780434Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.599000Z","msg_id":"c190b899-c532-49c4-86bf-bb637c48412e","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"status":"ok","protocol_version":"5.3","implementation":"PySpark","implementation_version":"1.0","language_info":{"name":"pyspark","mimetype":"text/x-python","codemirror_mode":{"name":"python","version":3},"file_extension":".py","pygments_lexer":"python3"},"banner":"Python 3.8.19 | packaged by conda-forge | (default, Mar 20 2024, 12:49:57) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.\n","help_links":[{"text":"Python Reference","url":"https://docs.python.org/3.8"},{"text":"IPython Reference","url":"https://ipython.org/documentation.html"},{"text":"NumPy Reference","url":"https://docs.scipy.org/doc/numpy/reference/"},{"text":"SciPy Reference","url":"https://docs.scipy.org/doc/scipy/reference/"},{"text":"Matplotlib Reference","url":"https://matplotlib.org/contents.html"},{"text":"SymPy Reference","url":"http://docs.sympy.org/latest/index.html"},{"text":"pandas Reference","url":"https://pandas.pydata.org/pandas-docs/stable/"}]},"idents":[],"buffers":[],"channel":"shell"}
23:16:13.782 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_4","msg_type":"kernel_info_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.781043Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"578714bf-8878-47e3-8081-19f5bc608e5a","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"status":"ok","protocol_version":"5.3","implementation":"PySpark","implementation_version":"1.0","language_info":{"name":"pyspark","mimetype":"text/x-python","codemirror_mode":{"name":"python","version":3},"file_extension":".py","pygments_lexer":"python3"},"banner":"Python 3.8.19 | packaged by conda-forge | (default, Mar 20 2024, 12:49:57) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.\n","help_links":[{"text":"Python Reference","url":"https://docs.python.org/3.8"},{"text":"IPython Reference","url":"https://ipython.org/documentation.html"},{"text":"NumPy Reference","url":"https://docs.scipy.org/doc/numpy/reference/"},{"text":"SciPy Reference","url":"https://docs.scipy.org/doc/scipy/reference/"},{"text":"Matplotlib Reference","url":"https://matplotlib.org/contents.html"},{"text":"SymPy Reference","url":"http://docs.sympy.org/latest/index.html"},{"text":"pandas Reference","url":"https://pandas.pydata.org/pandas-docs/stable/"}]},"idents":[],"buffers":[],"channel":"control"}
23:16:13.782 [trace] Got response for requestKernelInfo
23:16:13.782 [debug] Successfully completed postStartRawSession after 1 attempt(s) in 182ms
23:16:13.783 [trace] Executing silently Code (idle) = import sys as _VSCODE_sys\nprint(_VSCODE_sys.executable); del _VSCODE_sys
23:16:13.783 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"silent":false,"store_history":false,"user_expressions":{},"allow_stdin":true,"stop_on_error":false,"code":"import sys as _VSCODE_sys\nprint(_VSCODE_sys.executable); del _VSCODE_sys"},"header":{"date":"2024-08-31T15:16:13.783Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:16:13.783 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_7","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.783007Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"2c0fedf7-7e50-4019-a630-451777bf34b6","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.784 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_9","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.783932Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"2c0fedf7-7e50-4019-a630-451777bf34b6","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.784 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_8","msg_type":"kernel_info_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.783083Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.600000Z","msg_id":"2c0fedf7-7e50-4019-a630-451777bf34b6","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"status":"ok","protocol_version":"5.3","implementation":"PySpark","implementation_version":"1.0","language_info":{"name":"pyspark","mimetype":"text/x-python","codemirror_mode":{"name":"python","version":3},"file_extension":".py","pygments_lexer":"python3"},"banner":"Python 3.8.19 | packaged by conda-forge | (default, Mar 20 2024, 12:49:57) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.\n","help_links":[{"text":"Python Reference","url":"https://docs.python.org/3.8"},{"text":"IPython Reference","url":"https://ipython.org/documentation.html"},{"text":"NumPy Reference","url":"https://docs.scipy.org/doc/numpy/reference/"},{"text":"SciPy Reference","url":"https://docs.scipy.org/doc/scipy/reference/"},{"text":"Matplotlib Reference","url":"https://matplotlib.org/contents.html"},{"text":"SymPy Reference","url":"http://docs.sympy.org/latest/index.html"},{"text":"pandas Reference","url":"https://pandas.pydata.org/pandas-docs/stable/"}]},"idents":[],"buffers":[],"channel":"shell"}
23:16:13.785 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_10","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.785061Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:16:13.786 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_11","msg_type":"execute_input","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:13.785143Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"code":"import sys as _VSCODE_sys\nprint(_VSCODE_sys.executable); del _VSCODE_sys","execution_count":1},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,101,120,101,99,117,116,101,95,105,110,112,117,116]}],"buffers":[],"channel":"iopub"}
23:16:14.924 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_12","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:16:14.922838Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","date":"2024-08-31T15:16:13.783000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stdout","text":"Starting Spark application\n"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,111,117,116]}],"buffers":[],"channel":"iopub"}
23:17:15.984 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_13","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:15.982094Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>0</td><td>spark-f58b418b85ef4fccae42b8b768ce7d54</td><td>pyspark</td><td>idle</td><td><a target=\"_blank\" href=\"https://pre-emr-spark-ui-cn-hangzhou.data.aliyun.com/runtime/jobs/?token=OG80czJocTM0NDBQZnhsL1c1OFRYZmI4SDdhYlc2am93VmszMEYxY3FSVldQVGpVbVgrOVFVTTFncFZFM01mcmZZRE8wb0s2dDNTcUtGTmZ1aW9VR2JmbWE5TjZ6azlmMGhCc2Z5VmNPZHBXdnBrTEhoU0dHYmNSZCsybGp0bVBaejlpZ3J3M1pKOXhOcVRXL1Zhamt6b2lYaEpQaHRMRWFvblRBWUpLZFphQ25TdkV2NjFieVJ1UmlFeXhGdlU1WE9DZm9KaDVOemt6NlhHb1ByVkhoUkM5YmNhd1Q4UU03NDd6cHdXNjFFWk1EWXZvZWJUVUJwV1BweDhoSGVDK084TDhEd1Vqd1ZsUnRuWGZNeHlNYmYyd1h1T1J4UTBHVmlaNTZJNmw5d2RSVnBnUFBMZTdEdFhTaEo0UEtabElzM2drd1gyWG9KU0o2M2FFak9XRHd0eXJZTVpkQ0Y4OGxQZXJWbVIweGcraUtiRDhkUGY2RU9TNWppZmZxaFRPMnA5UVpIN0h1OUU9\">Link</a></td><td></td><td>None</td><td>✔</td></tr></table>"},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:17:16.129 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_14","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:16.127631Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"2.0.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border_bottom":null,"border_left":null,"border_right":null,"border_top":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":"25px","justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"padding":null,"right":null,"top":null,"visibility":null,"width":"50%"},"buffer_paths":[]},"comm_id":"91c101c84433443cbda11a5e7f39f7c5","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,57,49,99,49,48,49,99,56,52,52,51,51,52,52,51,99,98,100,97,49,49,97,53,101,55,102,51,57,102,55,99,53]}],"buffers":[],"channel":"iopub"}
23:17:16.129 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_15","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:16.128196Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"StyleView","bar_color":null,"description_width":""},"buffer_paths":[]},"comm_id":"fa15423e60634a17b55eccaafafd451c","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,102,97,49,53,52,50,51,101,54,48,54,51,52,97,49,55,98,53,53,101,99,99,97,97,102,97,102,100,52,53,49,99]}],"buffers":[],"channel":"iopub"}
23:17:16.130 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_16","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:16.128295Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"2.0.0","_view_name":"ProgressView","bar_style":"info","description":"Progress:","description_allow_html":false,"layout":"IPY_MODEL_91c101c84433443cbda11a5e7f39f7c5","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_fa15423e60634a17b55eccaafafd451c","tabbable":null,"tooltip":null,"value":0},"buffer_paths":[]},"comm_id":"4a320c2c24fe4e438b1f5c71bb25e53d","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,52,97,51,50,48,99,50,99,50,52,102,101,52,101,52,51,56,98,49,102,53,99,55,49,98,98,50,53,101,53,51,100]}],"buffers":[],"channel":"iopub"}
23:17:16.130 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_17","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:16.129061Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"4a320c2c24fe4e438b1f5c71bb25e53d"}},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:17:16.131 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"91c101c84433443cbda11a5e7f39f7c5","data":{}},"header":{"date":"2024-08-31T15:17:16.130Z","msg_id":"192e2c7b-3b35-4ef3-a205-af7c640b254d","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:16.131 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"fa15423e60634a17b55eccaafafd451c","data":{}},"header":{"date":"2024-08-31T15:17:16.131Z","msg_id":"4064053d-63e7-4249-8ffc-e63a33c95419","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:16.131 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"4a320c2c24fe4e438b1f5c71bb25e53d","data":{}},"header":{"date":"2024-08-31T15:17:16.131Z","msg_id":"87a59b21-669e-4edb-aefb-7eeaa226eac4","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.585 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_18","msg_type":"comm_close","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.584706Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{},"comm_id":"4a320c2c24fe4e438b1f5c71bb25e53d"},"idents":[{"type":"Buffer","data":[99,111,109,109,45,52,97,51,50,48,99,50,99,50,52,102,101,52,101,52,51,56,98,49,102,53,99,55,49,98,98,50,53,101,53,51,100]}],"buffers":[],"channel":"iopub"}
23:17:18.586 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_19","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.585795Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","date":"2024-08-31T15:16:13.783000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stdout","text":"SparkSession available as 'spark'.\n"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,111,117,116]}],"buffers":[],"channel":"iopub"}
23:17:18.693 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_20","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.691338Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"2.0.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border_bottom":null,"border_left":null,"border_right":null,"border_top":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":"25px","justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"padding":null,"right":null,"top":null,"visibility":null,"width":"50%"},"buffer_paths":[]},"comm_id":"978d9f30c1504d65b2fc334894a1bff9","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,57,55,56,100,57,102,51,48,99,49,53,48,52,100,54,53,98,50,102,99,51,51,52,56,57,52,97,49,98,102,102,57]}],"buffers":[],"channel":"iopub"}
23:17:18.694 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_21","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.692253Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"StyleView","bar_color":null,"description_width":""},"buffer_paths":[]},"comm_id":"978741196212441fa81a2227247d16a7","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,57,55,56,55,52,49,49,57,54,50,49,50,52,52,49,102,97,56,49,97,50,50,50,55,50,52,55,100,49,54,97,55]}],"buffers":[],"channel":"iopub"}
23:17:18.694 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_22","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.692510Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"2.0.0","_view_name":"ProgressView","bar_style":"info","description":"Progress:","description_allow_html":false,"layout":"IPY_MODEL_978d9f30c1504d65b2fc334894a1bff9","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_978741196212441fa81a2227247d16a7","tabbable":null,"tooltip":null,"value":0},"buffer_paths":[]},"comm_id":"e3eda4b210ae4e6b9044bbe00d8c4075","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,101,51,101,100,97,52,98,50,49,48,97,101,52,101,54,98,57,48,52,52,98,98,101,48,48,100,56,99,52,48,55,53]}],"buffers":[],"channel":"iopub"}
23:17:18.694 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"978d9f30c1504d65b2fc334894a1bff9","data":{}},"header":{"date":"2024-08-31T15:17:18.694Z","msg_id":"5efa94b6-39bb-4b99-9162-4848cbe51868","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.695 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"978741196212441fa81a2227247d16a7","data":{}},"header":{"date":"2024-08-31T15:17:18.695Z","msg_id":"d928486e-5e27-442e-ac98-47f981a3d461","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.695 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"comm_id":"e3eda4b210ae4e6b9044bbe00d8c4075","data":{}},"header":{"date":"2024-08-31T15:17:18.695Z","msg_id":"779acb1a-3010-4fe4-85de-3351b3fa995b","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.695 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_23","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.693854Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e3eda4b210ae4e6b9044bbe00d8c4075"}},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:17:18.739 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_24","msg_type":"comm_close","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.738350Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{},"comm_id":"e3eda4b210ae4e6b9044bbe00d8c4075"},"idents":[{"type":"Buffer","data":[99,111,109,109,45,101,51,101,100,97,52,98,50,49,48,97,101,52,101,54,98,57,48,52,52,98,98,101,48,48,100,56,99,52,48,55,53]}],"buffers":[],"channel":"iopub"}
23:17:18.740 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_25","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.739004Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","date":"2024-08-31T15:16:13.783000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stdout","text":"/usr/bin/python3"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,111,117,116]}],"buffers":[],"channel":"iopub"}
23:17:18.741 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_27","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.741155Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.741 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_26","msg_type":"execute_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.740739Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:16:13.783000Z","msg_id":"5478cd0a-8efd-44f7-b5d0-0c2ff28f81b3","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"started":"2024-08-31T15:16:13.785124Z","dependencies_met":true,"engine":"44ca996b-4012-4d50-8721-633ecb536a60","status":"ok"},"content":{"status":"ok","execution_count":0,"user_expressions":{},"payload":[]},"idents":[],"buffers":[],"channel":"shell"}
23:17:18.742 [trace] Executing silently Code (completed) = import sys as _VSCODE_sys\nprint(_VSCODE_sys.executable); del _VSCODE_sys with 6 output(s)
23:17:18.742 [trace] Started running kernel initialization for ~/Desktop/livy-emr.ipynb
23:17:18.742 [trace] Executing silently Code (idle) = try:\nimport ipywidgets as _VSCODE_ipywidgets\nprint("e976ee50-99ed-4aba-9b6b-9dcd5634d07d:IPy
23:17:18.743 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"silent":false,"store_history":false,"user_expressions":{},"allow_stdin":true,"stop_on_error":false,"code":"try:\n    import ipywidgets as _VSCODE_ipywidgets\n    print(\"e976ee50-99ed-4aba-9b6b-9dcd5634d07d:IPyWidgets:\" + _VSCODE_ipywidgets.__version__)\n    del _VSCODE_ipywidgets\nexcept:\n    pass"},"header":{"date":"2024-08-31T15:17:18.743Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.743 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_28","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.742484Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.130000Z","msg_id":"192e2c7b-3b35-4ef3-a205-af7c640b254d","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.744 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_29","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.743770Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.130000Z","msg_id":"192e2c7b-3b35-4ef3-a205-af7c640b254d","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.745 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_30","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.744301Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.131000Z","msg_id":"4064053d-63e7-4249-8ffc-e63a33c95419","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.745 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_31","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.745342Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.131000Z","msg_id":"4064053d-63e7-4249-8ffc-e63a33c95419","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.747 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_32","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.746355Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.131000Z","msg_id":"87a59b21-669e-4edb-aefb-7eeaa226eac4","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.748 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_33","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.747241Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:16.131000Z","msg_id":"87a59b21-669e-4edb-aefb-7eeaa226eac4","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.748 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_34","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.747574Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.694000Z","msg_id":"5efa94b6-39bb-4b99-9162-4848cbe51868","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.749 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_35","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.748400Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.694000Z","msg_id":"5efa94b6-39bb-4b99-9162-4848cbe51868","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.750 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_36","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.749467Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.695000Z","msg_id":"d928486e-5e27-442e-ac98-47f981a3d461","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.751 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_37","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.750156Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.695000Z","msg_id":"d928486e-5e27-442e-ac98-47f981a3d461","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.751 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_38","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.750491Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.695000Z","msg_id":"779acb1a-3010-4fe4-85de-3351b3fa995b","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.753 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_39","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.751272Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.695000Z","msg_id":"779acb1a-3010-4fe4-85de-3351b3fa995b","msg_type":"comm_close","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.754 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_40","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.751883Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.754 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_41","msg_type":"execute_input","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.751997Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"code":"try:\n    import ipywidgets as _VSCODE_ipywidgets\n    print(\"e976ee50-99ed-4aba-9b6b-9dcd5634d07d:IPyWidgets:\" + _VSCODE_ipywidgets.__version__)\n    del _VSCODE_ipywidgets\nexcept:\n    pass","execution_count":1},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,101,120,101,99,117,116,101,95,105,110,112,117,116]}],"buffers":[],"channel":"iopub"}
23:17:18.849 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_42","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.847591Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"2.0.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border_bottom":null,"border_left":null,"border_right":null,"border_top":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":"25px","justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"padding":null,"right":null,"top":null,"visibility":null,"width":"50%"},"buffer_paths":[]},"comm_id":"142ec4a7f205410a945ce9165e1bb47f","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,49,52,50,101,99,52,97,55,102,50,48,53,52,49,48,97,57,52,53,99,101,57,49,54,53,101,49,98,98,52,55,102]}],"buffers":[],"channel":"iopub"}
23:17:18.849 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_43","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.848070Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"StyleView","bar_color":null,"description_width":""},"buffer_paths":[]},"comm_id":"e6cff3bc11be4096b2485320e805779c","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,101,54,99,102,102,51,98,99,49,49,98,101,52,48,57,54,98,50,52,56,53,51,50,48,101,56,48,53,55,55,57,99]}],"buffers":[],"channel":"iopub"}
23:17:18.849 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_44","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.848422Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"2.0.0","_view_name":"ProgressView","bar_style":"info","description":"Progress:","description_allow_html":false,"layout":"IPY_MODEL_142ec4a7f205410a945ce9165e1bb47f","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_e6cff3bc11be4096b2485320e805779c","tabbable":null,"tooltip":null,"value":0},"buffer_paths":[]},"comm_id":"bdd1fc6f09b04ef5966e4e775288a7db","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,98,100,100,49,102,99,54,102,48,57,98,48,52,101,102,53,57,54,54,101,52,101,55,55,53,50,56,56,97,55,100,98]}],"buffers":[],"channel":"iopub"}
23:17:18.850 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_45","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.849441Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"bdd1fc6f09b04ef5966e4e775288a7db"}},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:17:18.884 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_46","msg_type":"comm_close","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.883644Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{},"comm_id":"bdd1fc6f09b04ef5966e4e775288a7db"},"idents":[{"type":"Buffer","data":[99,111,109,109,45,98,100,100,49,102,99,54,102,48,57,98,48,52,101,102,53,57,54,54,101,52,101,55,55,53,50,56,56,97,55,100,98]}],"buffers":[],"channel":"iopub"}

[chenshenwei1998]

23:17:18.887 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_48","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.886686Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.888 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_47","msg_type":"execute_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.885898Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.743000Z","msg_id":"916ca4e6-3b89-4e91-8f5f-6c7a9b9d7b38","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"started":"2024-08-31T15:17:18.751968Z","dependencies_met":true,"engine":"44ca996b-4012-4d50-8721-633ecb536a60","status":"ok"},"content":{"status":"ok","execution_count":0,"user_expressions":{},"payload":[]},"idents":[],"buffers":[],"channel":"shell"}
23:17:18.888 [trace] Executing silently Code (completed) = try:\nimport ipywidgets as _VSCODE_ipywidgets\nprint("e976ee50-99ed-4aba-9b6b-9dcd5634d07d:IPy with 1 output(s)
23:17:18.888 [trace] Determined IPyWidgets Version as undefined
23:17:18.888 [trace] Executing silently Code (idle) = def __VSCODE_inject_module():\ndef __VSCODE_call_function(function, callback, data=None):
23:17:18.888 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"silent":false,"store_history":false,"user_expressions":{},"allow_stdin":true,"stop_on_error":false,"code":"\ndef __VSCODE_inject_module():\n\n    def __VSCODE_call_function(function, callback, data=None):\n        __VSCODE_send_chat_message__(function, data, callback=callback)\n\n    def __VSCODE_send_chat_message__(function, data, callback):\n        requests = {}\n        try:\n            requests = __VSCODE_send_chat_message__.__requests\n        except Exception:\n            __VSCODE_send_chat_message__.__requests = requests\n\n        import uuid as __VSCODE_send_chat_message__uuid\n        import IPython.display as __VSCODE_send_chat_message__ipython_display\n\n        id = str(__VSCODE_send_chat_message__uuid.uuid4())\n        requests[id] = callback\n        data_is_none = data is None\n        __VSCODE_send_chat_message__ipython_display.display({\"application/vnd.vscode.chat_message\": data}, metadata={\"id\":id, \"function\": function, \"dataIsNone\": data_is_none}, raw=True)\n\n        del __VSCODE_send_chat_message__ipython_display\n        del __VSCODE_send_chat_message__uuid\n\n    def __VSCODE_on_chat_message(id, data):\n        requests = {}\n        try:\n            requests = __VSCODE_send_chat_message__.__requests\n        except Exception:\n            __VSCODE_send_chat_message__.__requests = requests\n\n        if id in requests:\n            requests[id](data)\n            del requests[id]\n        else:\n            raise NotImplementedError(f\"Callback not found for message {id}\")\n\n    import sys as __VSCODE_send_chat_message__sys\n    import IPython as __VSCODE_send_chat_message__IPython\n    chat = type(__VSCODE_send_chat_message__IPython)(\"chat\")\n    chat.send_message = __VSCODE_send_chat_message__\n    chat.call_function = __VSCODE_call_function\n    chat.__on_message = __VSCODE_on_chat_message\n    __VSCODE_send_chat_message__sys.modules[\"vscode\"] = type(__VSCODE_send_chat_message__IPython)(\"vscode\")\n    __VSCODE_send_chat_message__sys.modules[\"vscode\"].chat = chat\n    del __VSCODE_send_chat_message__sys\n    del __VSCODE_send_chat_message__IPython\n\n\n__VSCODE_inject_module()\ndel __VSCODE_inject_module\n\n__vsc_ipynb_file__ = \"/Users/~/Desktop/livy-emr.ipynb\"\n%config Completer.use_jedi = False"},"header":{"date":"2024-08-31T15:17:18.888Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:18.891 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_49","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.889768Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:18.892 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_50","msg_type":"execute_input","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:18.889915Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"code":"\ndef __VSCODE_inject_module():\n\n    def __VSCODE_call_function(function, callback, data=None):\n        __VSCODE_send_chat_message__(function, data, callback=callback)\n\n    def __VSCODE_send_chat_message__(function, data, callback):\n        requests = {}\n        try:\n            requests = __VSCODE_send_chat_message__.__requests\n        except Exception:\n            __VSCODE_send_chat_message__.__requests = requests\n\n        import uuid as __VSCODE_send_chat_message__uuid\n        import IPython.display as __VSCODE_send_chat_message__ipython_display\n\n        id = str(__VSCODE_send_chat_message__uuid.uuid4())\n        requests[id] = callback\n        data_is_none = data is None\n        __VSCODE_send_chat_message__ipython_display.display({\"application/vnd.vscode.chat_message\": data}, metadata={\"id\":id, \"function\": function, \"dataIsNone\": data_is_none}, raw=True)\n\n        del __VSCODE_send_chat_message__ipython_display\n        del __VSCODE_send_chat_message__uuid\n\n    def __VSCODE_on_chat_message(id, data):\n        requests = {}\n        try:\n            requests = __VSCODE_send_chat_message__.__requests\n        except Exception:\n            __VSCODE_send_chat_message__.__requests = requests\n\n        if id in requests:\n            requests[id](data)\n            del requests[id]\n        else:\n            raise NotImplementedError(f\"Callback not found for message {id}\")\n\n    import sys as __VSCODE_send_chat_message__sys\n    import IPython as __VSCODE_send_chat_message__IPython\n    chat = type(__VSCODE_send_chat_message__IPython)(\"chat\")\n    chat.send_message = __VSCODE_send_chat_message__\n    chat.call_function = __VSCODE_call_function\n    chat.__on_message = __VSCODE_on_chat_message\n    __VSCODE_send_chat_message__sys.modules[\"vscode\"] = type(__VSCODE_send_chat_message__IPython)(\"vscode\")\n    __VSCODE_send_chat_message__sys.modules[\"vscode\"].chat = chat\n    del __VSCODE_send_chat_message__sys\n    del __VSCODE_send_chat_message__IPython\n\n\n__VSCODE_inject_module()\ndel __VSCODE_inject_module\n\n__vsc_ipynb_file__ = \"/Users/~/Desktop/livy-emr.ipynb\"\n%config Completer.use_jedi = False","execution_count":1},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,101,120,101,99,117,116,101,95,105,110,112,117,116]}],"buffers":[],"channel":"iopub"}
23:17:19.004 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_51","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.003200Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"2.0.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border_bottom":null,"border_left":null,"border_right":null,"border_top":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":"25px","justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"padding":null,"right":null,"top":null,"visibility":null,"width":"50%"},"buffer_paths":[]},"comm_id":"b1e830538dae466da9f2a4056a766b49","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,98,49,101,56,51,48,53,51,56,100,97,101,52,54,54,100,97,57,102,50,97,52,48,53,54,97,55,54,54,98,52,57]}],"buffers":[],"channel":"iopub"}
23:17:19.005 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_52","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.003605Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"StyleView","bar_color":null,"description_width":""},"buffer_paths":[]},"comm_id":"76f8b4fdd4b34f1880641bf9d187792c","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,55,54,102,56,98,52,102,100,100,52,98,51,52,102,49,56,56,48,54,52,49,98,102,57,100,49,56,55,55,57,50,99]}],"buffers":[],"channel":"iopub"}
23:17:19.005 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_53","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.003700Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"2.0.0","_view_name":"ProgressView","bar_style":"info","description":"Progress:","description_allow_html":false,"layout":"IPY_MODEL_b1e830538dae466da9f2a4056a766b49","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_76f8b4fdd4b34f1880641bf9d187792c","tabbable":null,"tooltip":null,"value":0},"buffer_paths":[]},"comm_id":"4834265a59434b0db5fde6489f42c7e5","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,52,56,51,52,50,54,53,97,53,57,52,51,52,98,48,100,98,53,102,100,101,54,52,56,57,102,52,50,99,55,101,53]}],"buffers":[],"channel":"iopub"}
23:17:19.005 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_54","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.004663Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"4834265a59434b0db5fde6489f42c7e5"}},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:17:19.848 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_55","msg_type":"comm_close","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.847393Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{},"comm_id":"4834265a59434b0db5fde6489f42c7e5"},"idents":[{"type":"Buffer","data":[99,111,109,109,45,52,56,51,52,50,54,53,97,53,57,52,51,52,98,48,100,98,53,102,100,101,54,52,56,57,102,52,50,99,55,101,53]}],"buffers":[],"channel":"iopub"}
23:17:19.849 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_56","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.848163Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","date":"2024-08-31T15:17:18.888000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stderr","text":"An error was encountered:\nunknown magic command 'config'\nUnknownMagic: unknown magic command 'config'\n\n"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,101,114,114]}],"buffers":[],"channel":"iopub"}
23:17:19.851 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_58","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.850454Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.851 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_57","msg_type":"execute_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.849493Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:18.888000Z","msg_id":"d6b43a61-3d07-4cf7-9d7d-df58bc052535","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"started":"2024-08-31T15:17:18.889903Z","dependencies_met":true,"engine":"44ca996b-4012-4d50-8721-633ecb536a60","status":"ok"},"content":{"status":"ok","execution_count":0,"user_expressions":{},"payload":[]},"idents":[],"buffers":[],"channel":"shell"}
23:17:19.851 [trace] Executing silently Code (completed) = def __VSCODE_inject_module():\ndef __VSCODE_call_function(function, callback, data=None): with 2 output(s)
23:17:19.852 [debug] Requesting Kernel info
23:17:19.852 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{},"header":{"date":"2024-08-31T15:17:19.852Z","msg_id":"d786b537-9492-4d4c-9ae7-6315bd6c9b22","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:19.852 [trace] Got Kernel info
23:17:19.852 [trace] End running kernel initialization, now waiting for idle
23:17:19.852 [trace] Waiting for idle on (kernel): 541de6cd-5a2e-40df-9ffe-912511b766f1 -> idle
23:17:19.853 [trace] Finished waiting for idle on (kernel): 541de6cd-5a2e-40df-9ffe-912511b766f1 -> idle
23:17:19.853 [trace] End running kernel initialization, session is idle
23:17:19.854 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"code":"__file__.","cursor_pos":9},"header":{"date":"2024-08-31T15:17:19.853Z","msg_id":"2a06149e-d6ed-4f7b-9680-0803d217f6b0","msg_type":"complete_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"parent_header":{}}
23:17:19.854 [trace] Registering Kernel Completion Provider from kernel PySpark (Python 3.8.19) for language python
23:17:19.854 [trace] IPyWidgetScriptSource.initialize
23:17:19.855 [info] Kernel successfully started
23:17:19.855 [warn] No nbextensions folder found for kernel .undefined.sparkmagic.kernels.pysparkkernel.pysparkkernel./Users/~/miniforge3/envs/sparkmagictest/python./Users/~/miniforge3/envs/sparkmagictest/python.-m#sparkmagic.kernels.pysparkkernel.pysparkkernel
23:17:19.857 [trace] Cell Index:0 sent to kernel
23:17:19.858 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"silent":false,"store_history":true,"user_expressions":{},"allow_stdin":true,"stop_on_error":false,"code":"%%configure -f\n{\n    \"driverCores\": 1,\n    \"driverMemory\": \"3g\",\n    \"executorCores\": 1,\n    \"executorMemory\": \"3g\",\n    \"numExecutors\": 2,\n    \"conf\" : {\n        \"spark.dynamicAllocation.enabled\":  \"true\",\n        \"spark.dynamicAllocation.initialExecutors\":  2,\n        \"spark.dynamicAllocation.minExecutors\": 2,\n        \"spark.dynamicAllocation.maxExecutors\":  4,\n        \"spark.sql.files.ignoreMissingFiles\": \"true\"\n    }\n}"},"header":{"date":"2024-08-31T15:17:19.858Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"cellId":"vscode-notebook-cell:/Users/~/Desktop/livy-emr.ipynb#W0sZmlsZQ%3D%3D"},"parent_header":{}}
23:17:19.859 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_60","msg_type":"kernel_info_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.854467Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.852000Z","msg_id":"d786b537-9492-4d4c-9ae7-6315bd6c9b22","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"status":"ok","protocol_version":"5.3","implementation":"PySpark","implementation_version":"1.0","language_info":{"name":"pyspark","mimetype":"text/x-python","codemirror_mode":{"name":"python","version":3},"file_extension":".py","pygments_lexer":"python3"},"banner":"Python 3.8.19 | packaged by conda-forge | (default, Mar 20 2024, 12:49:57) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.\n","help_links":[{"text":"Python Reference","url":"https://docs.python.org/3.8"},{"text":"IPython Reference","url":"https://ipython.org/documentation.html"},{"text":"NumPy Reference","url":"https://docs.scipy.org/doc/numpy/reference/"},{"text":"SciPy Reference","url":"https://docs.scipy.org/doc/scipy/reference/"},{"text":"Matplotlib Reference","url":"https://matplotlib.org/contents.html"},{"text":"SymPy Reference","url":"http://docs.sympy.org/latest/index.html"},{"text":"pandas Reference","url":"https://pandas.pydata.org/pandas-docs/stable/"}]},"idents":[],"buffers":[],"channel":"shell"}
23:17:19.861 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_59","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.853737Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.852000Z","msg_id":"d786b537-9492-4d4c-9ae7-6315bd6c9b22","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.861 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_61","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.855538Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.852000Z","msg_id":"d786b537-9492-4d4c-9ae7-6315bd6c9b22","msg_type":"kernel_info_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.863 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_62","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.857034Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.853000Z","msg_id":"2a06149e-d6ed-4f7b-9680-0803d217f6b0","msg_type":"complete_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.973 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_64","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.972415Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.853000Z","msg_id":"2a06149e-d6ed-4f7b-9680-0803d217f6b0","msg_type":"complete_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.973 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_63","msg_type":"complete_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.971988Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.853000Z","msg_id":"2a06149e-d6ed-4f7b-9680-0803d217f6b0","msg_type":"complete_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"matches":[],"cursor_end":9,"cursor_start":9,"metadata":{"_jupyter_types_experimental":[]},"status":"ok"},"idents":[],"buffers":[],"channel":"shell"}
23:17:19.973 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_65","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.973323Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"busy"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:17:19.974 [trace] Start cell 0 execution @ 1725117439974 (clear output)
23:17:19.974 [debug] Kernel acknowledged execution of cell 0 @ 1725117439974
23:17:19.983 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_66","msg_type":"execute_input","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:19.973406Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"code":"%%configure -f\n{\n    \"driverCores\": 1,\n    \"driverMemory\": \"3g\",\n    \"executorCores\": 1,\n    \"executorMemory\": \"3g\",\n    \"numExecutors\": 2,\n    \"conf\" : {\n        \"spark.dynamicAllocation.enabled\":  \"true\",\n        \"spark.dynamicAllocation.initialExecutors\":  2,\n        \"spark.dynamicAllocation.minExecutors\": 2,\n        \"spark.dynamicAllocation.maxExecutors\":  4,\n        \"spark.sql.files.ignoreMissingFiles\": \"true\"\n    }\n}","execution_count":1},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,101,120,101,99,117,116,101,95,105,110,112,117,116]}],"buffers":[],"channel":"iopub"}
23:17:21.441 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_67","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:17:21.439013Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","date":"2024-08-31T15:17:19.858000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stdout","text":"Starting Spark application\n"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,111,117,116]}],"buffers":[],"channel":"iopub"}
23:18:23.627 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_68","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:23.623393Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>1</td><td>spark-c1dd087d3a64446d82cbf00819a008b4</td><td>pyspark</td><td>idle</td><td><a target=\"_blank\" href=\"https://pre-emr-spark-ui-cn-hangzhou.data.aliyun.com/runtime/jobs/?token=VHpIT09lUkZJZk9aTVJpSCtwNENPaVVDUGlSY0RDU21CVGlTcGpMSC9OMGUyN01ZaUtlbS9aWlBBazFsd3k3alZhQU51NzBDcGdyeDBlSWhZbnI0N2hqUGN0NzViem5FZUFIV1pNeVgvRWEwWUdzeG1RVFpEWXpobEk2TlNUMUE5ay9EbEZvOGRuNVc3TUt6K1ZWNDJSeEc1U0dnRWZCM2dnMm5DSzdVNXI0a2FDakhlRW5CdkFtaUNtTWlTOGwrWTBGenlqb0FSWUlpS3JZR2c2REt5d0Zua210ZS9raERWWHhqa1RyNStxNkNobkxPekZjQ1p4VVBHck1iYXBMdis1VGx2anVjMzJkemZOVE96OFlHYkdMeGd5VFcxNzAyOEdQVVBSODdtd205c1hTYmovelovZExoNUZtczhpQ1RrY2xZTFltemtFcUNabi96d3BSd3dONW5yVVlFZmp4d0pKSktnU0hwTDg2M2xjdmFwenJTNXFsZUhSdWFFVm5ac2tISGd2RzVFNXM9\">Link</a></td><td></td><td>None</td><td>✔</td></tr></table>"},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:18:23.735 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_69","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:23.733292Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"2.0.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border_bottom":null,"border_left":null,"border_right":null,"border_top":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":"25px","justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"padding":null,"right":null,"top":null,"visibility":null,"width":"50%"},"buffer_paths":[]},"comm_id":"2fcf15deb0ad4ad0b2ae0735fff7f54b","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,50,102,99,102,49,53,100,101,98,48,97,100,52,97,100,48,98,50,97,101,48,55,51,53,102,102,102,55,102,53,52,98]}],"buffers":[],"channel":"iopub"}
23:18:23.736 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_70","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:23.734030Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"2.0.0","_view_name":"StyleView","bar_color":null,"description_width":""},"buffer_paths":[]},"comm_id":"d10a2d529c894fa7a5138b71123b1dcd","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,100,49,48,97,50,100,53,50,57,99,56,57,52,102,97,55,97,53,49,51,56,98,55,49,49,50,51,98,49,100,99,100]}],"buffers":[],"channel":"iopub"}
23:18:23.736 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_71","msg_type":"comm_open","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:23.734149Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"version":"2.1.0"},"content":{"data":{"state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"2.0.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"2.0.0","_view_name":"ProgressView","bar_style":"info","description":"Progress:","description_allow_html":false,"layout":"IPY_MODEL_2fcf15deb0ad4ad0b2ae0735fff7f54b","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_d10a2d529c894fa7a5138b71123b1dcd","tabbable":null,"tooltip":null,"value":0},"buffer_paths":[]},"comm_id":"943190c492df470a989b7c1989864af5","target_name":"jupyter.widget","target_module":null},"idents":[{"type":"Buffer","data":[99,111,109,109,45,57,52,51,49,57,48,99,52,57,50,100,102,52,55,48,97,57,56,57,98,55,99,49,57,56,57,56,54,52,97,102,53]}],"buffers":[],"channel":"iopub"}
23:18:23.737 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_72","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:23.735179Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"943190c492df470a989b7c1989864af5"}},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:18:26.348 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_73","msg_type":"comm_close","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.344916Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{},"comm_id":"943190c492df470a989b7c1989864af5"},"idents":[{"type":"Buffer","data":[99,111,109,109,45,57,52,51,49,57,48,99,52,57,50,100,102,52,55,48,97,57,56,57,98,55,99,49,57,56,57,56,54,52,97,102,53]}],"buffers":[],"channel":"iopub"}
23:18:26.352 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_74","msg_type":"stream","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.347812Z","version":"5.3"},"parent_header":{"msg_type":"execute_request","date":"2024-08-31T15:17:19.858000Z","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","version":"5.2","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968"},"metadata":{},"content":{"name":"stdout","text":"SparkSession available as 'spark'.\n"},"idents":[{"type":"Buffer","data":[115,116,114,101,97,109,46,115,116,100,111,117,116]}],"buffers":[],"channel":"iopub"}
23:18:26.353 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_75","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.352135Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"Current session configs: <tt>{'conf': {'spark.dynamicAllocation.enabled': 'true', 'spark.dynamicAllocation.initialExecutors': 2, 'spark.dynamicAllocation.minExecutors': 2, 'spark.dynamicAllocation.maxExecutors': 4, 'spark.sql.files.ignoreMissingFiles': 'true'}, 'driverCores': 1, 'driverMemory': '3g', 'executorCores': 1, 'executorMemory': '3g', 'numExecutors': 2, 'kind': 'pyspark'}</tt><br>"},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:18:26.705 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_76","msg_type":"display_data","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.703776Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>1</td><td>spark-c1dd087d3a64446d82cbf00819a008b4</td><td>pyspark</td><td>idle</td><td><a target=\"_blank\" href=\"https://pre-emr-spark-ui-cn-hangzhou.data.aliyun.com/runtime/jobs/?token=NnpubGVsc3ppQWdubjNBWWdYdjVLT0xhNm81bmV6b2Z1dDZJdVBpSTNUdUNsNWJOUEl0VFQ5ejZoSy9yRDgybU9remJGQXhlMXU5NGRkT1NDdTNWNE1JNXBwYzIrVTdZQ0JJY0Z1TFdsTmpWMS9vM21nK2tldDVJTzRNbm0yMWQ0ZS9hUWJtbXh5eHFLSnFVUzF1M2hYRW1UVzRjQ3pJOCtYTlhySWxrMEtLckg3N28yQkxKbzlyQXVicEVyb0lEeUM2T3BldytyUHVOemRVZDUwV1luanF2Y1FpQTFFRE00NUp4SGRqWSs2a3BsUVVkaExPVHBjSkFocUxLZGpXYWFvRm5OanlhSDhlRXUzbFVGTGI1M0hOc1dQczhjTEUySk9mL3lJMmVaWWlSUHl2eGlvUkFUd2xzZkxuQWordkx3eW9ROVN2akg0dE1OR2RvRG1sbDRDMnB3dDFaUWtNRVRsck1Icm5nOGVCbjM4aGpZWDdhUjAwVHM0azBleGR0NG5TWmtYekl5Y3M9\">Link</a></td><td></td><td>None</td><td>✔</td></tr></table>"},"metadata":{},"transient":{}},"idents":[{"type":"Buffer","data":[100,105,115,112,108,97,121,95,100,97,116,97]}],"buffers":[],"channel":"iopub"}
23:18:26.708 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_78","msg_type":"status","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.707716Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{},"content":{"execution_state":"idle"},"idents":[{"type":"Buffer","data":[107,101,114,110,101,108,46,52,52,99,97,57,57,54,98,45,52,48,49,50,45,52,100,53,48,45,56,55,50,49,45,54,51,51,101,99,98,53,51,54,97,54,48,46,115,116,97,116,117,115]}],"buffers":[],"channel":"iopub"}
23:18:26.710 [trace] [AnyMessage Event] [recv] [58066] {"header":{"msg_id":"fbace886-d2a65ca6647108231dd95353_58066_77","msg_type":"execute_reply","username":"~","session":"fbace886-d2a65ca6647108231dd95353","date":"2024-08-31T15:18:26.706799Z","version":"5.3"},"parent_header":{"date":"2024-08-31T15:17:19.858000Z","msg_id":"a80e0396-9a0f-4669-8c85-a64e1c26c4b5","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"started":"2024-08-31T15:17:19.973395Z","dependencies_met":true,"engine":"44ca996b-4012-4d50-8721-633ecb536a60","status":"ok"},"content":{"status":"ok","execution_count":1,"user_expressions":{},"payload":[]},"idents":[],"buffers":[],"channel":"shell"}
23:18:26.711 [trace] Cell 0 completed in 66.736s (start: 1725117439974, end: 1725117506710)
23:18:26.711 [trace] Cell 0 executed successfully
23:20:16.108 [debug] Handle Execution of Cells 1 for ~/Desktop/livy-emr.ipynb
23:20:16.109 [trace] Cell Index:1 sent to kernel
23:20:16.110 [trace] [AnyMessage Event] [send] [58066] {"buffers":[],"channel":"shell","content":{"silent":false,"store_history":true,"user_expressions":{},"allow_stdin":true,"stop_on_error":false,"code":"spark"},"header":{"date":"2024-08-31T15:20:16.110Z","msg_id":"54995aa1-abf5-449f-8447-57489669fb43","msg_type":"execute_request","session":"7e4ba6c2-8f38-4248-b77a-fc7954144968","username":"8a029f6d-8d16-4f86-ba6c-1d63fe1cb654","version":"5.2"},"metadata":{"cellId":"vscode-notebook-cell:/Users/~/Desktop/livy-emr.ipynb#W1sZmlsZQ%3D%3D"},"parent_header":{}}

[DonJayamanne]

Thank you for the details and the analysis.
I'm surprised that no one else has run into this, guess no one else is using the pysparkkernel with this exetnsion.

[mkc450]

@DonJayamanne I am facing the same issue. Is there any fix for the same?

[MarvinRoe]

@DonJayamanne Hi, I also have this issue. I had to go back to Jupyter Version 2023.1.2010391206 and VS Code 1.76.2 in order for the PySpark Kernel to work again. Hope this helps finding the error

[shlomota]

I just applied a patch to sparkmagic as well, thought I'd share here in case it might help others.

find $(dirname $(dirname $PYTHON_PATH)) -name "sparkkernelbase.py" -type f

Locate the _do_execute method (around line 100-120) in sparkkernelbase.py
Replace the entire _do_execute method with this patched version:

def _do_execute(self, code, silent, store_history, user_expressions, allow_stdin):
        # VS Code injection patterns that should be executed locally
        vscode_patterns = [
            'import sys as _VSCODE_sys',
            'def __VSCODE_inject_module():',
            'import ipywidgets as _VSCODE_ipywidgets',
            '__vsc_print_nbextension_widgets',
            '__VSCODE_inject_module()',
            '%config Completer.use_jedi',
            '_VSCODE_ipywidgets',
            '__vsc_ipynb_file__',
            'try:\n    import ipywidgets as _VSCODE_ipywidgets',
            'def __jupyter_exec_background__():',
            '__jupyter_exec_background__()',
            'do_complete(',
            'application/vnd.vscode.bg.execution'
        ]
        
        # Check if this is VS Code injection code
        is_vscode_injection = any(pattern in code for pattern in vscode_patterns)
        
        if is_vscode_injection:
            # Execute VS Code injection code locally using IPython
            self.logger.debug(f"Intercepting VS Code injection code: {code[:100]}...")
            
            try:
                # Execute locally instead of sending to Livy
                result = self._execute_cell_for_user(
                    code, silent, store_history, user_expressions, allow_stdin
                )
                return result
            except Exception as e:
                # If local execution fails, create a successful response to keep VS Code happy
                self.logger.debug(f"Local execution failed, creating mock response: {e}")
                return {
                    'status': 'ok',
                    'execution_count': self.execution_count,
                    'payload': [],
                    'user_expressions': {}
                }
        else:
            # Normal user code - send to Livy via sparkmagic
            code_to_run = self.user_code_parser.get_code_to_run(code)
            
            res = self._execute_cell(
                code_to_run, silent, store_history, user_expressions, allow_stdin
            )
            
            return res

[DonJayamanne]

I'm not keen on trying to add some work arounds here.
Pyspark kernel isn't technically a Jupyter compliant Kernel.
Trying to add work arounds in the Jupyter extension for something that isn't technically a Jupyter kernel doesn't feel like the right solution.

We have other extensions that use Jupyter to run against Pyspark kernels, and they handle this by intercepting the requests at a lower level (i.e. ensure Jupyter API works with Pyspark out side the Jupyter extension).

Hence closing this issue as not planned.

[devstein]

Hey @shlomota @MarvinRoe @mkc450 - a contributor made a PR to fix this in Sparkmagic. I do not have a good way to reproduce the issue.

Can any of you take a look and confirm the proposed PR fix works for your as well? Thank you!