Support viewfs schema

[WhoisZihan]

Aarrow already supported viewfs for quite some time, since fsspec is built on top of arrow, it would be great for fsspec to support viewfs schema which is commonly used nowadays.

[martindurant]

Sure! We support arrow filesystems in general, so adding the protocol would not be hard - see what is done in fsspec.registry for "hdfs". I don't know whether viewfs would come with any extra concerns.

Note that fsspec is not built on arrow, but supports it as an optional backend, and many arrow functions accept fsspec.

[WhoisZihan]

Sure! We support arrow filesystems in general, so adding the protocol would not be hard - see what is done in fsspec.registry for "hdfs". I don't know whether viewfs would come with any extra concerns.

Great! I tried to call fsspec.register_implementation("viewfs", "fsspec.implementations.arrow.HadoopFileSystem"), but it will cause HDFS connection failure.

However, if I directly use fsspec.filesystem('hdfs'), I can connect to and read from viewfs cluster. Not sure how they differ in implementation, or if it is because of libhdfs3 initialization.

Note that fsspec is not built on arrow, but supports it as an optional backend, and many arrow functions accept fsspec.

I see, thanks for clarification.

[WhoisZihan]

Sure! We support arrow filesystems in general, so adding the protocol would not be hard - see what is done in fsspec.registry for "hdfs". I don't know whether viewfs would come with any extra concerns.

Great! I tried to call fsspec.register_implementation("viewfs", "fsspec.implementations.arrow.HadoopFileSystem"), but it will cause HDFS connection failure.

However, if I directly use fsspec.filesystem('hdfs'), I can connect to and read from viewfs cluster. Not sure how they differ in implementation, or if it is because of libhdfs3 initialization.

@martindurant Do you have any idea on how to get viewfs work by registry? Thanks.