Cannot connect to Sparkmagic kernels

[jvaesteves]

Hello, I've trying to use VSCode to accept the Sparkmagic kernels that I installed on a venv using Poetry, so I can connect to an EMR instance via Livy, but when listing the kernels on the notebook interface, all that appears are Python versions and venvs from my computer. I tried this on PyCharm Pro and works well.

Setup:

• VSCode version: 1.70.1
• Jupyter extension version: v2022.7.1102252217
• Python version: tried 3.10.5 and 3.7.13
• Poetry version: 1.1.14
• Packages: sparkmagic==0.20.0

Steps to reproduce:

SPARKMAGIC_LOCATION=$(pip show sparkmagic | grep Location | cut -d" " -f2)
jupyter nbextension enable --py --sys-prefix widgetsnbextension
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/pysparkkernel 
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/sparkrkernel 
jupyter serverextension enable --py sparkmagic

What is expected

For PySpark, Spark and SparkR to appear on VSCode kernel list

What is happening

[DonJayamanne]

@jvaesteves I've created an issue to discuss this issue, this looks like a bug. Lets continue the conversation there #11126

Note: This looks like a bug, hence the reason to move it into the issues.