Can't read google cloud storage data with pyspark in VSCode jupyter

[vikasd22]

Hello,

I have seen a peculiar issue in VScode Jupyter, where I can not read google cloud storage parquet files with Pyspark. It works in jupyterlab in browser with no problem. For example, if I do this in VScode jupyter:

# pyspark config
spark_config =  {
    "spark.sql.execution.arrow.pyspark.enabled": "true",
    "spark.driver.memory": "8g",
    "spark.executor.memory": "8g"
}

# reading gcs data
df = spark.read.parquet("gs://ds-source/datalake/ymd=20220301/")

I get the following error:

An error occurred while calling o78.parquet.\n: org.apache.hadoop.fs.UnsupportedFileSystemException: 
No FileSystem for scheme \"gs\"\n\tat org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)

Note that above code work fine as a python script.
I have already played around with the config and vscode Jupyter versions. All the version seems to have this issue. I am not sure if It is a bug. This seems to be pyspark specific. Pandas doesn't seem to have any problem reading the files.

I have already tried this stack overflow solution as well:

https://stackoverflow.com/questions/55595263/how-to-fix-no-filesystem-for-scheme-gs-in-pyspark

I would love if anyone can shed some light on it.

[rchiodo]

My guess is that there's some bash script that needs to run first. You might try closing all instances of VS code and then launching it from a bash shell. That should get VS code to inherit the environment of the bash shell.

[vikasd22]

@rchiodo I use a remote instance and login in there with remote-ssh extension. How do you propose I do this? Should I install VScode on the remote instance?

[rchiodo]

Installing VS code on the remote might work.

This might work too, but would probably be rather hard to get the environment correct:
https://superuser.com/questions/163167/when-sshing-how-can-i-set-an-environment-variable-on-the-server-that-changes-f

And that's presuming my assumption is correct, that there's some environment that isn't correct.