Use kitty capability to display image in the terminal?

[PMassicotte]

[PMassicotte]

Proof of concept:

show_plot_in_kitty <- function(expr) {
  png_file <- tempfile(fileext = ".png")
  png(png_file, width = 800, height = 600)
  force(expr)
  dev.off()
  system(sprintf("kitty +kitten icat %s", shQuote(png_file)))
}

show_plot_in_kitty(plot(pressure, col = "red"))

library(ggplot2)

p <- cars |>
  ggplot(aes(x = speed, y = dist)) +
  geom_point(size = 5L)

show_plot_in_kitty(print(p))

[PMassicotte]

Original idea from: https://rstats.me/@adamhsparks/114993016239999921

[jalvesaq]

It would be great to have a graphics device that saves PNG images and runs an application to display them.

We could split Kitty or WezTerm window to reserve a space for the graphics.

The system would be perfect if the graphics had a black background and the palette of colors were adjusted accordingly.

[jalvesaq]

Instead of splitting the terminal, displaying the graphics mixed with text output as in your example, might be equally or even better.

[PMassicotte]

I think we could add an option to use terminal capability to display the plot. What would be the best way?

1. When sourcing, parse the document with TS and just embed plot code inside show_plot_in_kitty()?

[PMassicotte]

Even easier, just a hook?

if (interactive() && Sys.getenv("TERM") == "xterm-kitty") {

  show_plot_in_kitty <- function(expr, width = 800, height = 600, dpi = 100) {
    png_file <- tempfile(fileext = ".png")
    png(png_file, width = width, height = height, res = dpi)
    on.exit(dev.off(), add = TRUE)
    force(expr)
    dev.off()
    system2("kitty", c("+kitten", "icat", png_file))
  }


  base_plot <- base::plot
  plot <- function(...) {
    show_plot_in_kitty(base_plot(...))
  }


  if (requireNamespace("ggplot2", quietly = TRUE)) {
    ggplot2_print <- getS3method("print", "ggplot")
    assign("print.ggplot", function(x, ...) {
      show_plot_in_kitty(ggplot2_print(x, ...))
    }, envir = globalenv())
  }
}

I will make an attempt

[jalvesaq]

I'm trying your code, and it works great!

In my opinion, it's already good enough, and we don't need a graphics device.

[djvanderlaan]

For reference: here is some code I use to create plots in kitty from R:

https://codeberg.org/djvanderlaan/kitty.r

One of the things this adds, is that it automatically tries to scale the plot and resolution (size of of the fonts) such that it matches those in the terminal. It can also match the colours to those in the terminal.

[PMassicotte]

Do you think this could be implemented in R.nvim?

[PMassicotte]

@jalvesaq can you try the terminal-plot branch? Sorry for all the change noise in the files, due to my formatter. You will have to set terminal_plot = true in your config.

I have tested it with a file

library(ggplot2)

plot(1)

ggplot(mtcars, aes(x = mpg, y = cyl)) +
  geom_point()

[jalvesaq]

I'll try it. I didn't have time look at @djvanderlaan's kitty.r package, but it seems to do everything that we need.

[PMassicotte]

Just tried it with a more "complex" plot.

[jalvesaq]

@PMassicotte, it works! Good job! However, it's not the same as a graphics device. That is, it will work only for predefined functions, not, for example, for hist(), barplot(), etc...

[jalvesaq]

kitty.r works in the code below:

library(kitty.r)
kitty_plot(hist(mtcars$qsec))

# Example from kitty.r README
kitty_plot({
  data(iris)
  plot(iris$Sepal.Width, iris$Sepal.Length, 
    col = kitty_palette()[iris$Species], 
    pch = 20)
  grid(lty = 2)
  legend("topright", legend = levels(iris$Species), 
    col = kitty_palette()[1:3], pch = 20)
})

# ggplot example
p <- ggplot(mtcars, aes(mpg, hp)) +
        geom_point()
kitty_plot(print(p))

However, this doesn't work:

kitty_plot({
    ggplot(mtcars, aes(mpg, hp)) +
        geom_point()
})

From the kitty.r README example, I realized that if the plot is immediately displayed in the terminal, we can't increment it step by step as can be done in an interactive graphics device such as X11. The complete sequence of commands to produce the plot must be within a single call to kitty_plot.

For example, the terminal_plot config option will not work well with this:

plot(1)
title("Plot One")

kitty_plot works:

library(kitty.r)
kitty_plot({
    plot(1)
    title("Plot One")
})

But, then, the script could not be shared with someone not running R in a Kitty terminal. Moreover, we can't call kitty_plot() in a Quarto document.

[jalvesaq]

In my opinion, ideally, we would have a system combining the features of @PMassicotte's approach and @djvanderlaan's kitty.r:

• Work with any function that produces graphics (base R function, ggplot, or whatever).
• Update the old image or display a new one with new elements (such as the title).
• Use a palette of colors that is appropriate for a terminal with a dark background.
• Don't require any change to the code; that is, the same code would run with other graphics devices (pdf or png when rendering a Quarto document, X11, Rstudio, XQuartz, Windows, etc).

[PMassicotte]

These are nice points. Not sure how to implement that thought :) But I think that could be a nice addition to R.nvim.

[jalvesaq]

These are nice points. Not sure how to implement that thought :) But I think that could be a nice addition to R.nvim.

Maybe we could start with support for ggplot and, perhaps, for plot, as you have done.

This function works on both WezTerm and Kitty:

# Adapted from https://codeberg.org/djvanderlaan/kitty.r
png2terminal <- function (filename) {
    d <- base64enc::base64encode(filename, 4096)
    for (i in seq_along(d)) {
        cat(paste0("\033_Ga=T,f=100,m=1;", d[i], "\033\\"))
    }
    cat("\033_Ga=T,f=100,m=0\033\\")
    cat("\n")
    invisible(NULL)
}

png2terminal("filename.png")

I think there is no need to check for available tools or run any system command such as kitty +kitten icat or wezterm imgcat.

[PMassicotte]

I have pushed your suggestion

R.nvim/nvimcom/R/misc.R

Lines 425 to 436 in 16387b4

	# Display PNG image in terminal using Kitty graphics protocol
	# Adapted from https://codeberg.org/djvanderlaan/kitty.r
	png2terminal <- function(filename) {
	d <- base64enc::base64encode(filename, 4096L)
	for (i in seq_along(d)) {
	cat(paste0("\033_Ga=T,f=100,m=1;", d[i], "\033\\"))
	}
	cat("\033_Ga=T,f=100,m=0\033\\")
	cat("\n")
	invisible(NULL)
	}

[adamhsparks]

Thanks for taking my idea and running with it! This is really great, I appreciate the work here.

[jalvesaq]

TERM in WezTerm is "xterm-256color", but we could use TERM_PROGRAM, which is "WezTerm". Something like this would work:

  # Check terminal support first
  ok <- Sys.getenv("TERM") == "xterm-kitty" ||
    Sys.getenv("TERM_PROGRAM") == "WezTerm"

  if (!ok) {
    warning("Terminal plot display not supported.")
    return(invisible(NULL))
  }

[jalvesaq]

We can see warnings about terminal plot not enabled in :RDebugInfo output. I thinks it's better to do nothing if not running interactively:

setup_terminal_plotting <- function() {
    if (!interactive()) {
        return(invisible(NULL))
    }

[jalvesaq]

Various terminal emulators support Kitty's graphics protocol (see https://sw.kovidgoyal.net/kitty/graphics-protocol/):

• Ghostty
• Konsole
• st (with a patch)
• Warp
• wayst
• WezTerm

I think we should either check all of them (not just Kitty and WezTerm) or not check the terminal at all.

[PMassicotte]

dd77e26

[jalvesaq]

Even if running in WezTerm, the plot can't be displayed if R is running in the built-in Neovim's terminal emulator. Also, the expression "base R plots" refers to all plotting functions from the base library. It will be more precise to refer specifically to base::plot. So, I suggest the following changes to the documentation:

diff --git a/doc/R.nvim.txt b/doc/R.nvim.txt
index 520f21a..fe4b0d1 100644
--- a/doc/R.nvim.txt
+++ b/doc/R.nvim.txt
@@ -1612,9 +1612,10 @@ config:
 ------------------------------------------------------------------------------
 6.17. Terminal plotting                                       *terminal_plot*
 
-R.nvim can display plots directly in the terminal for interactive R sessions.
-This feature automatically detects the terminal size and renders plots using
-terminal capabilites.
+
+R.nvim can display plots directly in the terminal for interactive R sessions
+when R is running in an external terminal emulator. This feature automatically
+detects the terminal size and renders plots using terminal capabilites.
 
 To enable terminal plotting functionality, add to your config:
 >lua
@@ -1624,9 +1625,8 @@ When enabled, plots will be displayed in the terminal using the current
 terminal dimensions for optimal sizing. 
 
 Note: This feature is experimental and only work in interactive R sessions.
-It works with base R plots and `ggplot2` but may not support all plotting
-libraries or plot types. Incremental plots (using `plot()` followed by
-`points()`, `lines()`, etc.) are not supported.
+It works with `base::plot` and `ggplot2::ggplot`. Incremental plots (using
+`plot()` followed by `points()`, `lines()`, etc.) are not supported.
 
 
 ------------------------------------------------------------------------------

[jalvesaq]

We are creating three objects in the user workspace: .base_plot, plot, and print.ggplot. At least .base_plot can be avoided:

diff --git a/nvimcom/R/misc.R b/nvimcom/R/misc.R
index 90db4dc..5d3d61c 100644
--- a/nvimcom/R/misc.R
+++ b/nvimcom/R/misc.R
@@ -515,10 +515,8 @@ setup_terminal_plotting <- function() {
   }
 
   if (getOption("nvimcom.terminal_plot", FALSE)) {
-    assign(".base_plot", base::plot, envir = .GlobalEnv)
-
     plot <- function(...) {
-      show_plot_in_terminal(.base_plot(...))
+      show_plot_in_terminal(base::plot(...))
     }
 
     assign("plot", plot, envir = .GlobalEnv)

[jalvesaq]

Problem: We get the base64 code and not the actual PNG image with the following example.Rmd file after <LocalLeader>kr:

---
title: "testing terminal plot"
---

A simple plot:

```{r}
plot(1)
```

[PMassicotte]

Problem: We get the base64 code and not the actual PNG image with the following example.Rmd file after <LocalLeader>kr:

---
title: "testing terminal plot"
---

A simple plot:

```{r}
plot(1)

I think I have managed to make it work in qmd and rmd document. I am using active binding to switch behavior depending on if we are in the process of knitting a document.

[PMassicotte]

There is also a new major version of ggplot2 coming out soon which now use S7 instead of S3. So likely this will need some more tweaking.

[jalvesaq]

I think I have managed to make it work in qmd and rmd document. I am using active binding to switch behavior depending on if we are in the process of knitting a document.

I confirm that it works!

[djvanderlaan]

Inspired by what you all are doing for R.nvim: I started work on something that was in de freezer for a while: an actual graphics device. I have a first version working. See the screenshot below. The code for this is in the development branch in https://codeberg.org/djvanderlaan/kitty.r/src/branch/dev/R/tgp.R.

What also might be of interest: I also started on supporting sixel output. (So I will probably rename the package to something else). This is, for example, supported by recent versions of Microsoft Terminal. Not yet in de repo.

[djvanderlaan]

sixel is another/older protocol for displaying images in the terminal (https://en.wikipedia.org/wiki/Sixel). Quite a bit more limited than the terminal graphics protocol. Here is a (not completely up to date) list of terminals and whether they support sixel or not.

[djvanderlaan]

What my current implementation does, is it opens a ragg::agg_capture device. This creates an in memory bitmap with the output of the plot commands. It also registers a call back that is called after every command in the main session. This call back checks if the agg_capture device changed and if so it will dump the contents into the terminal.

[PMassicotte]

What my current implementation does, is it opens a ragg::agg_capture device. This creates an in memory bitmap with the output of the plot commands. It also registers a call back that is called after every command in the main session. This call back checks if the agg_capture device changed and if so it will dump the contents into the terminal.

That is interesting. I do not full grasp all the details, but does it use the kitty protocol at the end or another mechanism to plot inside the terminal?

[djvanderlaan]

Yes it does.

[trevorld]

I also started on supporting sixel output.

FYI: In case you didn't know {bittermelon} can also pretty print raster images to the terminal wrapping {cli}: https://trevorldavis.com/R/bittermelon/dev/reference/bm_print.html

• Besides one pixel per character it can also render two or four pixels per character using Unicode Block Elements i.e. compress = "vertical" will render using two different colored pixels per character
• If the suggested package {magick} is installed it can auto-downscale the image so it fits in the terminal width i.e. downscale = requireNamespace("magick", quietly = TRUE)
• The package is MIT licensed.

Obviously using Kitty terminal graphics protocol will result in higher resolution images and should be preferred when supported by the terminal...

[jalvesaq]

Thanks, @djvanderlaan! I just tried the tgp graphics device in Kitty, and it works!

~/.Rprofile:

if (interactive()) {
    options(device = kitty.r::tgp)
}

I was waiting for this for years! Could we have some control over the graphics dimensions, as @PMassicotte has achieved in his code?

I couldn't check in WezTerm because I'm travelling. I'll try it on Saturday.

[djvanderlaan]

You can set the size, but currently only when you manually open the device:

tgp(width = 1200, height = 800)
plot(...)

I need to add something to allow users to change the default size; probably through options, e.g. options(tgp_size = 1200).

This morning I pushed some changes: the device will now try to guess a reasonable size for the device; set the resolution to match the font size in the terminal; and when running kitty (I have to look if and how you can query the theme in other terminals) will set the colors to those in the terminal.

Also there is a has_tgp_support() function that tries to determine if the terminal supports the terminal graphics protocol, so you might want to change the lines in .Rprofile to:

if (interactive() && has_tgp_support()) {
  options(device = kitty.r::tgp)
}

[jalvesaq]

This morning I pushed some changes

I tried it now. Excellent!

[jalvesaq]

@djvanderlaan, it works in WezTerm! Congratulations!

@PMassicotte, I think we can cancel the implementation of the terminal_plot option and recomend the use of options(device = tgp).

[PMassicotte]

Yes perfectly fine

[jalvesaq]

Thanks, @PMassicotte!

[jalvesaq]

kitty.r was renamed as terminalgraphics, and I put this in my ~/.Rprofile:

if (interactive()) {
  setHook("before.plot.new", function(...) {
    palette(c(
      "#ffffff",
      "#e05050",
      "#50e050",
      "#4040e0",
      "#50e0e0",
      "#e050e0",
      "#e0e050",
      "#505050",
      "#e0e0e0"
    ))
    par(
      bg = "#222222",
      fg = "#e0e0e0",
      col.axis = "#e0e0e0",
      col.lab = "#e0e0e0",
      col.main = "#e0e0e0",
      col.sub = "#e0e0e0"
    )
  })
  suppressPackageStartupMessages(
    library(ggplot2, quietly = TRUE)
  )
  suppressMessages(
    ggplot2::theme_set(theme_gray() + ggdark::dark_mode())
  )
  options(device = terminalgraphics::tgp)
}