Registering a custom code cell engine when using a lua filter and knitr backened

[coatless]

I'm trying to create a new code chunk engine, e.g.:

```{webr}
1 + 1
```

I mirrored how the shinylive extension modifies the CodeBlock when it detects the new class, c.f.

https://github.com/quarto-ext/shinylive/tree/main

However, this works well when the engine is python only. If I switch the engine to knitr, I get:

---
title: "Testing"
format: html
engine: knitr
filters:
- webr
---

```{webr}
1 + 1
```

Warning message:
In get_engine(options$engine) :
  Unknown language engine 'webr' (must be registered via knit_engines$set()).

Outside of creating a warning, this also causes the pandoc filter to receive a CodeBlock that has different attributes between engines:

jupyter

[trace] Parsed [CodeBlock ("",["{webr}"],[]) "1 + 1 "] at line 12
[trace] Parsed [CodeBlock ("",["{webr}"],[]) "message(\"hello\")"] at line 17

knitr

[trace] Parsed [CodeBlock ("",["webr","cell-code"],[]) "message(\"hello\")" at line 20
[trace] Parsed [Div ("",["cell"],[]) [CodeBlock ("",["webr","cell-code"],[] at line 23

This requires the filter checking inclusion on:

el.attr.classes:includes("{webr}") # Jupyter 
el.attr.classes:includes("webr")    # knitr

Is there a way to register engine requirements or options in a quarto extension, e.g. _extension.yaml?

The extension code can be found at: https://github.com/coatless/quarto-webr

[mcanouil]

For knitr, you can read the following https://bookdown.org/yihui/rmarkdown-cookbook/custom-engine.html to register new custom engine.
Note that it might be better to make a PR (https://github.com/yihui/knitr) to add that new engine to knitr, also they might have plans already to do that.

Also @coatless you might want to make a PR to register comment character for chunk option as discussed for prql in #4092.

[coatless]

@mcanouil I'm aware of:

https://bookdown.org/yihui/rmarkdown-cookbook/custom-engine.html

What I'm particularly interested in is being able to specify that outside of a code cell. Ideally, I would like it to be in the YAML:

---
title: "My Document"
format: html
knitr:
  knit_engines:  # desired key to load engine
    webr: "verbatim" 
  opts_chunk:   # Real
    comment: "#>" 
    R.options:
      knitr.graphics.auto_pdf: true
---

The example is based off of:

https://quarto.org/docs/computations/r.html#knitr-options

Unfortunately, the pass into pandoc comes much too late to be able to append the engine definition and, thus, avoid having the warning get raised.

[coatless]

Whoops, I didn't see that edit linking to the prql comment/engine discussion. Thanks for pointing that out! Glancing at the prql package, they include a specific knitr engine setup portion:

Construct the engine:

https://github.com/eitsupi/prqlr/blob/8c6fb5de7c85648acb16ab6eb6fae518456df3b7/R/knitr-engine.R#L7-L36

Load the engine during package load if knitr is detected:

https://github.com/eitsupi/prqlr/blob/8c6fb5de7c85648acb16ab6eb6fae518456df3b7/R/zzz.R#L2-L10

Though, I think this means that there isn't a good way to create a custom code engine at the moment for quarto?

[mcanouil]

"code engines", as you call them, are not defined by Quarto and are completely up to the computing engines, namely knitr and Jupyter.
Quarto only adds code cell options to pass through to the computing engines. This is why not all options are supported by both computing engines.

knitr:
  knit_engines:  # desired key to load engine
    webr: "verbatim" 
  opts_chunk:   # Real
    comment: "#>" 
    R.options:
      knitr.graphics.auto_pdf: true

The above is only possible if/because knitr has these options.

[coatless]

@mcanouil Thanks for the feedback and emphasis on terms.

So, I suppose what I'm asking for is if there is a way to modify the compute engine process inside of an extension. My guess is this is probably a new feature request... So, I would need @cderv or another quarto team member to move it over onto the tracker.

As it stands now, I'm only able to "clean up" by reading Pandoc's meta at a point that is much further down the line than I care for:

The goal would be to allow changes at:

And similarly for Jupyter:

[mcanouil]

Depends on what you want/need exactly, but maybe a Lua filter can help.

[cderv]

Hi @coatless,

Glad to see you jump into making this extension ! We were planning on doing it like shinylive as we thought that there would be some adjustment in quarto to make and this is effectively the case. You found some limitation that we need to think through.

Is there a way to register engine requirements or options in a quarto extension, e.g. _extension.yaml?

So to this question: Not yet - no there is not, and maybe there should be indeed. I remember there was once already the question about Quarto extension for custom cell handlers, but this was not yet something we could offer easily, as best integration would be internal to Typescript code base, like we did for ojs, mermaid and dot. Maybe a simplification of this could be made for a way to "prevent" knitr handling for Lua based Codeblock handler.

Below more more context on what you are encountering (sorry for the long answer, I wanted to give as much details as needed, also for later reference to this question)

The warning you see is thrown by knitr because it sees the code chunk with engine as it is using the same syntax as using code cells, and then apply logic as any other engine. But for this one, there is no specific R engine as the code cell should be handled directly by your extension.

Note one trick : Using - (dash) in an engine name will make knitr skip the cell because it is not supported - only [a-zA-Z0-9_]. So shinylive-python does not have the issue in R documents. Not sure it was on purpose though. That being said, this is just a workaround, and you don't want to change your name and process.

However, this works with . too. This works with your current extension with no error

---
title: "Testing"
format: html
engine: knitr
filters:
  - webr
---

```{.webr}
1 + 1
```

knitr will ignore those for sure because this is the usual Pandoc Markdown syntax for Codeblock with attributes, and your filter correctly handle the webr as a class with el.attr.classes:includes("webr"). shinylive-python could have go with that too, but they do differently (probably unaware of the knitr issue).

That would be the right way to go with Pandoc only to define such CodeBlock handler in Lua filter. And this goes with the idea of cells engine to knitr or jupyter for now, except internal ojs, mermaid and dot. As I said Quarto extension for cell handler is not something we have yet, but Lua filter to process CodeBlock with classes and attributes can be done. Difference is slim by rely mainly on YAML intelligence and comments in YAML for options.

I understand though that user experience and other cell handler like mermaid and dot justify to find a better way probably. That is worth re discussing for 1.4 or later.

About the context of current situation our internal cell handler.

When defining a cell language handler that is not supposed to be handled by knitr (like mermaid or dot internal to Quarto), knitr should be aware of those and effectively just pass through the content with attributes. This is what is done with those two internal mermaid and dot cell handers.

Having the extension registering this information to be pass through to knitr seems a good idea. But maybe we need some adjustment to knitr itself to ignore some code cells.

you might want to make a PR to register comment character for chunk option as discussed for prql in #4092.

Regarding this, it is not as simple. I don't know if we want to "register" every custom engine this way. prql was a first and a knitr engine only. But for example, shinylive handled this for example by parsing the args in the codeblock content iself. Maybe we want to offer such feature in our Lua Quarto API 🤔 This is also part of why we don't have yet Quarto extension for cell handlers yet. More to think through.

I'll open an issue to track and discuss further the general case. (#4839)

Hope it helps.

[cderv]

@coatless discussion in #4839 leads us to this advice

Custom engine that should not be processed by computation engine, because process in extension should be using a - (dash) in there name.

This is the convention that is using shinylive and reason is that Quarto VSCODE extension use the value after - to know what language and auto completion should be used. Ex:

• For python https://github.com/quarto-dev/quarto/blob/main/apps/vscode/syntaxes/build-lang.js#L317
• For R: https://github.com/quarto-dev/quarto/blob/5a91b81974508ce6afdbe6921c5ffd7e94e92ffe/apps/vscode/syntaxes/build-lang.js#L49

This means when

```{shinylive-python}
# the content will be highlighted for Python syntax
```

To have the same for webr, this means the usage should be

```{webr-r}
# the content will be highlighted for R syntax
```

This is a convention already in place in VSCODE used for shinylive . Easy to add the same for R and also to add autocompletion in those chunks as they would be known to be R.

This requires no change in knitr or Quarto itself. (as I explained a dash in engine is not valid knitr engine so skipped)

Is this something that would work for you ? using webr-r or webr ?

[coatless]

First, there's no real "rush" to implement as the extension is live and accomplishes its main goal: reducing the barrier of entry to using webr for teaching. In fact, I'd prefer to avoid rushing a solution for v1.3 quarto release window.

With this in mind, I'm okay with either naming scheme (webr, webr-r, web-r, ...). There is a preference toward webr, but we're early days so changing it will only affect early adopters.

I suppose what I'm more interested in is being able to obtain the runtime output of a code cell. That is, I would like to pre-populate the HTML document with a webr backed code chunk.

Another potential idea I've toyed with is splitting across a tabset panel:

From my understanding, the compute engine needs to handle that earlier on in the document rendering workflow. Thus, I would need to be able to interact with knitr::knit_engines$set() (and the Jupyter equivalent).

One other portion that has arisen is the need to be able to specify execution options, e.g. using a default echo: true remark (c.f. the server:shiny interactivity setting echo: false on all code cells coatless/quarto-webr#4)

[jjallaire]

We've now got the dash based variations (e.g. shinylive-python or web-r) working as full peers in the Quarto VS Code extension (they get highlighting, execution, and completion, in both source and visual mode).

For now on options I think the best way is to parse them within your extension (we will hopefully in v1.4 do this for you automagically but that's not 100% guaranteed so it might be better just to get it working in the meantime)