Unable to install & load httpfs extension, file does not exist

[ahuarte47]

I can not load extensions using the web console or my custom jupyter-lite instance (I have successfully loaded the wheel duckdb-0.10.3-cp311-cp311-emscripten_3_1_46_wasm32.whl and duckdb is working).

I would like to load a parquet file from s3.

[ahuarte47]

Hi, I am trying to build the httpfs extension myself from duckdb source code (tag v0.10.3) without success.

I heave temporally changed the extension_config.cmake file to include this extension and the json as well.

Executing make wasm_eh the json extension is built, but httpfs fails:

The error is related with OpenSSL library, but I do not know how to resolve it.
Please can you help me?

[carlopi]

Hi! At the current moment extensions are not supported in the duckdb version of pyodide.

This should be more explicit, given here the error message is not really helping.

Networking on the pyodide side is still possible, possibly via fsspec as a workaround.

[ahuarte47]

Thanks Carlo, I will try to use fsspec.
Congratulations for this amazing software!

[marcus-cse]

@carlopi , any plans for extension support soon? Is it a heavy lift given httpfs is already supported by DuckDB-Wasm?

[fredguth]

How can I make it work with fsspec? I am not used to this extension.

[ahuarte47]

Finally I was able to build the httpfs extension for WASM. After installing EMSDK & VCPKG tools, make wasm_eh worked.

But do you know is there any plan to add support to build or include extensions for Pyodide?. Do you know how to build DuckDB for Pyodide? Is it possible to build the wheels adding extensions ?

[nicornk]

duckdb + httpfs + iceberg extension on pyodide would be super cool!

[fredguth]

I think duckdb is not looking for it, but a duckdb sync like the what is Turso for sqlite would be also very interesting. I don't need the fastest db for writing if I am building a local-first app but having a duckdb in the browser which can access data remotely and efficiently... great.

[ahuarte47]

I thought I compiled the httpfs extension for WASM successfully, but I was wrong. It seems it is not linking properly some necessary lib, maybe openssl?

The community http-client extension uses openssl as well, I am getting same error.

duckdb-iceberg uses openssl and I tried to mimic its makefile/CMakeLists.txt settings to fix that, but I was not able to load both extensions in WASM yet :-(

[timsurber]

I'm also interested in this.
@ahuarte47 Would you like to share your current progress for building?

[ahuarte47]

I'm also interested in this. @ahuarte47 Would you like to share your current progress for building?

I did not get the extension working properly :-(

[timsurber]

I found a workaround for reading files over network.
Example for parquet:

import pyodide.http
import duckdb
res = await pyodide.http.pyfetch("http://localhost:8000/test.parquet") # watch out for CORS
dst = open("/test.parquet", "wb")
dst.write(await res.bytes())
duckdb.sql("select * from read_parquet('/test.parquet')")
dst.close()

Is it worth it to explore this approach to better integrate it into duckdb pyodide or is it planned to use a whole different method for HTTP requests?