Improving code chunk manipulation

[PMassicotte]

I made some changes in this branch: https://github.com/R-nvim/R.nvim/tree/feature/improve-chunk-handling

Basically, removing many uses of regex in favour of using Treesitter for navigating and sending code chunks/lines to the console.

Please feel free to test it and comment.

[jalvesaq]

With this file:

First paragraph.

```{r}
x <- 1
y <- 2
```

Second paragraph.

```{r}
z <- 3
```

Trying <LocalLeader>cc (RSendChunk) with the cursor on "x", I get:

No R or Python code chunk found at the cursor position.

Trying <LocalLeader>ch (RSendChunkFH) with the cursor on "z", I get:

E5108: Error executing lua R.nvim/lua/r/send.lua:346: attempt to call field 'filter_code_chunks_by_lang' (a nil value)
stack traceback:
        R.nvim/lua/r/send.lua:346: in function 'chunks_up_to_here'
        [string ":lua"]:1: in main chunk

[PMassicotte]

Good tx. Should be ok now.

[jalvesaq]

I noted only one more problem, a missing feature. If we have this child.Rmd:

First child paragraph.

```{r}
xc <- 1
```

And this parent.qmd:

First paragraph.

```{r, child="child.Rmd"}
```

Second paragraph.

```{r}
x <- xc + 2
```

The value of x is `r x`.

And if the cursor is over the header of the first chunk of parent.qmd, R.nvim on the main branch sends this command to R:

require(knitr); knit('child.Rmd', output=NULL)

On the main branch, we have this:

        if cline:find("^```.*child *= *") then
            return "chunk_child"
        [...]

[PMassicotte]

Nice, was not even aware of this feature. Should be fixed now.

[jalvesaq]

Still not working...

[PMassicotte]

Humm it works on my side

[PMassicotte]

ah ok

because I have an extra line. in the chunk because of my formatter. Fix incoming.

[jalvesaq]

It's very interesting to have the fields info_string_params and comment_params on the chunk table. They can be used in other situations. I'll see the possibility of using your code to replace regular expression searches at https://github.com/R-nvim/cmp-r/blob/main/lua/cmp_r/figtbl.lua

[PMassicotte]

It's very interesting to have the fields info_string_params and comment_params on the chunk table. They can be used in other situations. I'll see the possibility of using your code to replace regular expression searches at https://github.com/R-nvim/cmp-r/blob/main/lua/cmp_r/figtbl.lua

Very nice!

[PMassicotte]

Just looking at https://github.com/R-nvim/cmp-r/blob/main/lua/cmp_r/figtbl.lua, indeed, you could do awesome stuff there.

[PMassicotte]

I had to change the TS query to account for empty content in chunks. Doing some tests and will send a push.

[jalvesaq]

Testing here... There is a code that is not supported on the main branch, but is valid Quarto code:

First paragraph.

```{r}
#| child: child.Rmd
xc <- 9
```

Second paragraph.

```{r}
x <- xc + 2
```

The value of x is `r x`.

Note: the line xc <- 9 (that is, the contents of a chunk with the child field) is ignored.

[PMassicotte]

Ok, I will add a check to see if we have child in either places.

[PMassicotte]

What should be the behaviour if we have this

First paragraph.

```{r, child="child.Rmd"}
x <- 1
```

Second paragraph.

```{r}
x <- xc + 2
```

The value of x is `r x`.

```{r}
y <- 2
```

if the cursor is inside the first chunk. Should if knit the child, send x <- 1 or both? At the moment, it send x <- 1

Also if the cursor is on the header of the first chunk? Just knit?

[jalvesaq]

When we render the document, any R code within the block is skipped, but I think we can ignore this for now and treat it as normal R code.

[jalvesaq]

Idea of behavior change:

• When the cursor is over the chunk header, instead of the message "Not inside R or Python code chunk [within chunk_header]", we could send the whole chunk.
• When the cursor is over the end of chunk delimiter, jump to the next chunk header (not to the first line of the next code block).

This change would allow easily to send code blocks one at a time and jump to the next. What do you think? Is the current behavior preferable?

[PMassicotte]

ah, that is a good idea.

When the cursor is over the chunk header, instead of the message "Not inside R or Python code chunk [within chunk_header]", we could send the whole chunk.

What if that chunk header contains a child? knit it and then send the whole chunk?

[jalvesaq]

I think we should imitate the behavior of rendering (rmarkdown::render() or quarto::quarto_render()) which is to ignore the contents of the chunk. So, we would just send require(knitr); knit('child.Rmd', output=NULL).

[PMassicotte]

Can you test with

---
---

# H1

```{r}
library(tidyverse)
```

## H2

First paragraph.

```{webr}
x <- 1L
y <- 2L
```

### With child

```{r, child='child.Rmd'}
z <- 3
```

Second paragraph.

```{r}
#| eval:false
z <- 3L
```

```{r}
#| child: child.Rmd
xc <- 9
```

```{r}
mtcars |>
  head()
```

and child.rmd

First child paragraph.

```{r}
xc <- 1
```

[PMassicotte]

The thing I am not sure is the eval. Should we ignore it and presume it is only for knitr and run it?

[jalvesaq]

Only now I could try it...

It's excellent!

• Cursor over the block header: send the block and jump to the next block.
  • If the next block has eval: false, it is skipped.
  • If the block has child: ..., knitr() is called instead of sending the block code.
  • The "language" webr is recognized.
• In any case, if the cursor is inside the block, the code is sent one line at a time.

The only thing that doesn't seem correct that I noted was that <LocalLeader>ch sends the contents of the block with the child field instead of knit() the child file.

[PMassicotte]

Thank you!

Good call for <localLeader>ch. I shall fix that tomorrow.

Things to do/discuss

• What if a chunk has child defined in both the info_string and comments? Which one should have precedence? I think we just have to decide and document it.
• We also have code scattered in rmd.lua and quarto.lua that might benefit from to be glued together.

[PMassicotte]

The only thing that doesn't seem correct that I noted was that <LocalLeader>ch sends the contents of the block with the child field instead of knit() the child file.

This should be working now.

[jalvesaq]

I added the highlighting of code blocks. Please, pull the changes and let me know whether it works on your side. Any suggestion of improvement?

[PMassicotte]

I really like it. If fell it makes the document easier to read. Good job. Should we merge it?

[jalvesaq]

Should we merge it?

The behavior of sending a line when the cursor is over the chunk's header has changed. I will document it in the README. Also, I'll try to highlight comments starting with #| with a different color.

[jalvesaq]

Did it. But now I have found a bug: sometimes, markdown code is recognized as "r" lang by get_lang(). I'm trying to create a small reproducible example.

[PMassicotte]

I also found it. It is like the TS parser or something is not fully initialized. Hope you can find more.

[jalvesaq]

With this simple Quarto document:

---
author: "Someone"
title: "Testing"
---

If the cursor is on the last line (an empty one) and we type <M-r>, R.nvim inserts the text "<M-r>" instead of creating a new chunk. I added a vim.notify(get_lang()) to the beginning of rmd.write_chunk() and it shows "yaml" when it should be "markdown".

[jalvesaq]

Should we create an option to disable the different highlighting of comments prefixed with #| or #%%?

[PMassicotte]

I think the user might like that, what do you think? Aslo, what is the #%% I am not familiar with that comment type.

With our latest push, this is what I have

[PMassicotte]

really like the code block background

[PMassicotte]

One thing to fix is when the cursor is on a python chunk header. It now tries to send the python code as R code.

[jalvesaq]

I think the user might like that, what do you think?

I also think everybody will like it and nobody will request an option to disable it. We can add the option in the future if someone requests it.

what is the #%% I am not familiar with that comment type.

The #%% prefix is a Jupyter delimiter to allow running parts of a cell separately. I've never used Jupyter, but I've found this information somewhere and added the highlighting to not leave this case uncovered.

[PMassicotte]

I also think everybody will like it and nobody will request an option to disable it. We can add the option in the future if someone requests it.

Sounds good.

I am refactoring the get_lang(), will see how it works.

[PMassicotte]

I have modified the get_lang() function for the chunk. It now really returns the chunk language (r. python, ...). I have added another function that tells where the cursor is within a chunk. header, end or child.

We can now send the whole chunk if the cursor is on a python chunk.

[jalvesaq]

I fixed a bug. In parser:language_for_range(pos):lang(), pos should be a table with four elements, not two:

diff --git a/lua/r/utils.lua b/lua/r/utils.lua
index 137246b..ddc27f0 100644
--- a/lua/r/utils.lua
+++ b/lua/r/utils.lua
@@ -98,8 +98,7 @@ function M.get_lang()
     local parser = require("nvim-treesitter.parsers").get_parser(buf)

     if not parser then return "" end
-
-    return parser:language_for_range(pos):lang()
+    return parser:language_for_range({ pos[1], pos[2], pos[1], pos[2] }):lang()
 end

[PMassicotte]

Great, do you think that fixed the issue you mentioned earlier?

[jalvesaq]

I'm testing it now...

And there is another change that I want to make: don't highlight as the Ignore highlighting group the line with the comment child: child.Rmd.

[jalvesaq]

I improved the highlighting of chunks with the child field:

The line of the child field isn't highlighted with the color of the Ignore highlighting group. I think it's more correct now since this line is the only one not ignored by knitr.

[PMassicotte]

Yes it works on my side and it help to understand what will be knitted. One thing we could eventually do it eroding when the user tries to run code in a child chunk. This is what happens when you try to knit a child chunk with code in the body.

[jalvesaq]

I think we can merge it now...

[PMassicotte]

Great, will initiate a PR

[jalvesaq]

Great, do you think that fixed the issue you mentioned earlier?

Yes, I think so.