Duckdb parquet, what's under the hood ?

[GuillaumeChretien]

Description

Hello,
I am experimenting with DuckDB and the Parquet format. Everything works wonderfully, but I would like to understand what is happening under the hood. Here is an example code (in the example I used a viewof, but in my actual case myId is mutable from an interactive map).

db = DuckDBClient.of({ donnees : FileAttachment("data/donnees.parquet") }) 

viewof myId = Inputs.range([0, 1000], {value: 1, step: 1, label: "select id"})

res= db.sql` SELECT id, indicateur, valeur FROM donnees WHERE id =${myId} ` 
res

I then export my site and publish it.

My questions are:

1. Is the Parquet file fully loaded on the client-side, or is it just a connection that is available?
2. Do the queries fetch only what is necessary with each change of myId, or are the data integrated into the HTML during the export?
3. Is this method appropriate for any volume of data?

Thank you to the whole team for this amazing tool!

[cscheid]

Your code is in an OJS cell; that's a reactive Javascript framework we use from https://observablehq.com/, and there's no server involved on their code. So:

1. Is the Parquet file fully loaded on the client-side, or is it just a connection that is available?
2. Do the queries fetch only what is necessary with each change of myId, or are the data integrated into the HTML during the export?
3. Is this method appropriate for any volume of data?

1. If you deploy this to a public server, the data/donnees.parquet will exist on the server as well. The data is fully loaded by the web browser ("client-side") every time the web page is opened.
2. I believe the answer is a qualified "no". It will in general depend on how the DuckDB client is implemented. I think DuckDB's WASM implementation doesn't do partial requests.
3. It will depend on how the DuckDB implementation works in general, and I'm not certain. (This isn't part of Quarto; it's part of https://github.com/duckdb/duckdb-wasm, and you'll need to study their docs)

[GuillaumeChretien]

Thanks !

[jimjam-slam]

FWIW I've kicked the tyres on Quarto and Observable's DuckDB with respect to ranged HTTP reads (which are the feature that allow a partial read of a Parquet file), and I wasn't able to get them to work. When you open the page, the full Parquet file is downloaded from the server (unless you've written some code that delays the read, like putting it behind a button).

I asked around on the Observable forums and the DuckDB Discord. Everyone seems to think ranged HTTP reads should work with DuckDB WASM, but I tried hosting the Parquet file as (a) an Observable attachment, (b) on GitHub Pages, (c) on GitHub using a raw URL and (d) on CloudFlare Pages, and they all fell back to a full read (they report the fallback in the console):

Quarto test:

• Code: https://github.com/jimjam-slam/test-quarto-duckdb-parquet-ranged
• GitHub Pages: https://jimjam-slam.github.io/test-quarto-duckdb-parquet-ranged

Observable notebook: https://observablehq.com/@jimjamslam/ranged-parquet-duckdb-test

The other thing to note with the DuckDB ranged read support is that it defaults to doing a full read if ranged reads aren't available, rather than erroring. That might be a problem if you have, say, a 2GB Parquet file that you want people to read maybe a few hundred KB of — you might not want to subject users (or yourself) to a big download if it fails!

That said, I'm really keen to see this work at some point 🙏🏻 It seems like there have been a few kinks with Observable's deploy of DuckDB WASM versions that fix problems with ranged reads, so it could be that it's still on the way.

observablehq/feedback#588
duckdb/duckdb-wasm#1366

[GuillaumeChretien]

Thank you for sharing this, that's the strategy I've adopted for the GIS part of my quartowebsite with the pmtiles format. It would be great if we could actually do the same with duckdb and parquet file !

[jimjam-slam]

I should say, I tried changing the query on the above Observable notebook link to:

SELECT
  count(*)
FROM
  read_parquet(${`https://jimjam-slam.github.io/test-quarto-duckdb-parquet-ranged/data/acorn-sat.parquet`})

Instead of SELECT * FROM [...] WHERE station_num == '086338', and although the console reports it's falling back to a full read, I'm only seeing a few hundred bytes actually transferred.

So in conclusion, I'm very confused 😂 It could be that requests that be answered purely using the metadata in the HEAD request will skip the full read, but ones that required subsequent partial reads end up reading the full file instead...