Lua API to parse the code block special comments starting with #|

[whitphx]

Description

Looks like the special comment lines starting with #| are one of the standard features of Quarto.
However, it is hard to implement extensions/filters dealing with them because these is no packaged API to parse that syntax.
For example, Shinylive extension has its own parser implementation.
I think such a parser package should be provided from Quarto's official API.

Context: whitphx/quarto-stlite#1
I created a Quarto filter for Streamlit/stlite and would like to extend it to handle the special comment syntax.

[mcanouil]

#| is not a general syntax as it will depends on the comment symbol used for each languages.
For instance, in Observable code cell it is //|.

@cscheid Any opinion on this feature request regarding Quarto Lua API?

[cscheid]

I think it's a good request. It's not trivial to do because most of our YAML option processing happens in Typescript, where we have good YAML parsers. I don't know if a good YAML parser exists in Lua, and that would be a significant effort.

@whitphx what exactly are looking to do with those options?

[whitphx]

Now I'm creating quarto-stlite.
It's a Quarto filter to embed Streamlit app into Quarto document, with the Wasm port of Streamlit, "stlite".

It now supports normal code blocks like below:

```{stlite-python}
import streamlit as st

st.title("stlite Quarto embedding example")

st.write("# Welcome to Streamlit! 👋")
```

I want to make the filter support the special comment syntax like the followings.
I also think requiring the Node and/or Python runtime and some package installation is acceptable due to the supported language of this extension, like the Shinylive extension actually expects its Node package to be installed, while Lua implementation is the best for portability.

Specify YAML options

```{stlite-python}
# requirements: ["numpy"]
# embed: true
# embed_options:
# - light_theme
import streamlit as st
import numpy as np

st.title("stlite Quarto embedding example")

st.write("# Welcome to Streamlit! 👋")

st.write(np.random.randn(100, 2))
```

Mount multiple files

This is like what the Shinylive extension already does.
Both stlite and Shinylive are ports of Python frameworks, Streamlit and Shiny, and they have similar features. Handling multiple source files is one of them.

```{shinylive-python}
## file: app.py
import streamlit as st
from utils import square

x = st.slider("Select a value")
n = st.slider("Select a power")
st.write(f"{x}^{n} = {square(x, n)}")

with open("www/logo.png", "rb") as f:
    st.image(f.read())

## file: utils.py

def square(x, n):
    return x ** n

## file: www/logo.png
## type: binary
iVBORw0KGgoAAAANSUhEUgAAACgAAA ...
```

[cscheid]

Let's slow down a little. The second feature you're discussing behaves quite differently from the first.

The first feature is about extracting a single YAML object with cell metadata. This is directly analogous to how a Pandoc markdown document can have a single YAML object with document metadata. In Pandoc, if you write the following:

---
title: A title
---

## A section

some text

---
date: 2023-08-10
---

Some more text

You get a single YAML object. Quarto's cell metadata is even more restrictive: you can only declare those at the very top of a cell. If we were to extend this behavior, though, we would almost certainly make it compatible with Pandoc, and yield a single metadata entry.

The second feature is much more complex, and almost certainly outside of what we'd ever support in Quarto. Of course, as an extension author, you're welcome to treat the code however you want. But the management of splitting a single code cells into sub-cell and their associated metadata is not something that base Quarto supports now or would in the future.

[whitphx]

Makes sense.
While I just dumped everything I wanted, I understand the first one is general and worth considering in this context.
Still, it would be so helpful if Quarto releases its official API to deal with the first case because as an extension developer I can use it for the next version only covering the first one and can even see its behavior as an official reference when I write my own parser covering the second one.