Embedding compressed files - parquet or arrow?

[brichards920]

Embedding compressed files - parquet or arrow?

Hello - my main use case with Quarto involves writing self-contained files with OJS chunks that users would open from a SharePoint site.

I've been successful converting R data to OJS via "ojs_define" and also with "FileAttachment('myfile.arrow').arrow()", but I've noticed that for larger datasets the html files I produce get pretty big.

It would be helpful to be able to use parquet files or compressed feather/arrow files, and to be able to embed them in the document. The closest attempt I've had so far is below using duckdb where it works without a self-contained: true argument but it fails with it.

Does anyone happen to know if it is possible at this time to use compressed arrow files with Quarto + OJS. I know that in the Quarto documentation it says "Apache Arrow IPC file (uncompressed)" as one of the structured types, but since I've managed to get parquet files running with duckdb and Quarto, I'm wondering if things have moved forward to allow them now? The Observable documentation also seems to reference some use of other files types.

I've been trying various configurations of compressed, uncompressed, feather, arrow, parquet, and duckdb files without much success past what has already worked.

This works:

---
title: "Issue Example - This Works!"
---

```{r}
arrow::write_parquet(iris, 'irisParquet.parquet')
```

```{ojs}
db = DuckDBClient.of({
  myfile: FileAttachment("irisParquet.parquet")
})

Inputs.table(db.query("select * from myfile"))
```

This fails with "Error: Invalid Input Error: File 'irisParquet.parquet' too small to be a Parquet file."

---
title: "Issue Example - This Fails!"
self-contained: true
---

```{r}
arrow::write_parquet(iris, 'irisParquet.parquet')
```

```{ojs}
db = DuckDBClient.of({
  myfile: FileAttachment("irisParquet.parquet")
})

Inputs.table(db.query("select * from myfile"))
```

I'm on Windows 10, and I'm using quarto 1.3.450

[cderv]

my main use case with Quarto involves writing self-contained files with OJS chunks that users would open from a SharePoint site

I am curious about this as OJS does not work with self contained to embed resources

• Documents using Observable JS are not self-contained #1412

where it works without a self-contained: true argument but it fails with it.

Especially this make me think that this is indeed not working.

Did you tweak something to make it work ? Otherwise, I believe you hit some limitations of self-contained and OJS not working well together. I let you see the thread linked

[mcanouil]

@brichards920 Please note that self-contained is deprecated¹ as stated in Quarto documentation. Prefer the use of embed-resources which the current (for more than a year now) and correct option, see https://quarto.org/docs/reference/formats/html.html#rendering.

Footnotes

1. self-contained is internally translated to the correct option, i.e., embed-resources. ↩

[brichards920]

Thank you @cderv and @mcanouil for your feedback.

I realized that when I was doing most of my learning with Quarto and OJS I was actually on a past version of RStudio (2023.03.2) that shipped with Quarto 1.2.475.

With the older version I could compile a qmd file like this and then open the resulting html file without a supporting RStudio session.

---
title: "Example - This Works with embed-resources"
embed-resources: true
---
  
```{r}
ojs_define(irisO = iris)
```

```{ojs}
Inputs.table(transpose(irisO))
```

Shortly before I submitted my question I tried updating RStudio and Quarto to their newest versions, which seems to produce html files with data that fails to load correctly if I don't have a supporting RStudio session/server. I just rolled back to the old version and the resulting html files work again. @cderv that's likely why you wondered if I had tweaked something - I was on an older version that did work.

Anyways - I'm thrilled that I can produce flat html files that work with the embed-resources argument and that continue to work after I close down RStudio. The filesize is pretty manageable even with larger datasets.

Thanks for looking into this - sorry about any confusion from my older Quarto version.