ark?

[r2evans]

I just watched a 2024 introduction video to Positron, and I heard Lionel mention that he was excited that he might be able to integrate ark into ESS. And then he switched to Positron :-(

Is there any progress towards supporting Ark in emacs? Does it enable, solve, or simplify any pain points?

[vspinu]

It's LSP server so lsp-mode or eglot are the way forward here.

[r2evans]

Thanks @vspinu !

I don't see a lot of documentation about shifting from "classic" (my term) to LSP for interacting with the R process. I found https://nonvalet.com/posts/20211011_emacs_ess_lsp/, and I do see https://github.com/emacs-lsp/lsp-mode/. I'm not familiar enough with LSP to know ...

• is there any interaction between ESS and LSP, or is it a completely different way of interacting with the inferior process?
• if completely separate, are there known incompatibilities that preclude trying both simultaneously?
• I found @lionel- made https://lionel-.github.io/slidedecks/2024-07-11-ark/#/lsp-completion, though it seems to talk about an "approach" with ESS, some challenges (raw text), and shifts focus to Positron (which makes sense given lionel's recent efforts). Have you (vspinu) tried using lsp/ark?

I'm willing to "play" but have found it difficult to find any lessons learned with LSP with R in general and Ark specifically.

Background: I do a lot of remote work and need "detachable/reattachable" sessions, which is supported by Ark and not typical SSH modes (and this is not a tmux/screen/byobu thing). I have scripts that do most of what I want, but one of the bigger pain points is graphics, where native graphics do not work, and httpgd looks promising but has a tendency to get stuck in an uninterruptible state.

[vspinu]

I haven't used positron/ark but I am relying on lsp-mode/eglot with python (with variable success). I hope installation of ark will eventuallyas straightforward as installing an R package. For now it's a no-go for all those companies which have restricted software access (such as mine). I cannot even try it locally ATM.

Once it's mature enough, integrating with eglot, lsp-mode and dap-mode shouldn't be a major issue. I would expect most of the stuff should be working out of the box.

If ark can have super features compared to other LSP clients and could connect to a running R process to explore the dynamic environment, it might make sense to do some light integration on ESS side relying on emacs lsp clients, otherwise it's lsp client's business entirely as it looks to me.

Background: I do a lot of remote work and need "detachable/reattachable" sessions, which is supported by Ark

Where can I read more on this? How to setup ark remotely and how does it work?

[lionel-]

Ark is a Jupyter kernel that we use as an execution engine. The idea is that Positron is interacting with multiple interpreters (R and Python to start with) via the Jupyter protocol. I think that idea is very relevant for ESS as an interactive development tool for multiple statistical languages. It solves the issues we've had with raw IPC interaction (e.g. figuring out output boundaries for each input).

We have an LSP in Ark but the long term goal is to move most of it to Air. Whereas the Ark LSP inspects the R session dynamically and is paired with a Jupyter kernel running the R session, the Air LSP will be pure static analysis and very easy to integrate in other IDEs. Only downside is that it doesn't have access to the state of the session, so the Ark LSP is still needed to get completions for dynamic objects (e.g. data frames created by the user).

[lionel-]

Just adding that someone from the nvim community managed to get Ark working in nvim: https://www.reddit.com/r/neovim/comments/1j8msnt/new_experimental_r_plugin_arknvim/

[r2evans]

the long term goal is to move most of it to Air

If this is the future, does this mean that something non-Positron that wants to use LSP will use Air, Air and optional Ark, or always Air+Ark? Is this related to the Ark page saying "The LSP and DAP features are currently only available in Positron but will be made available to other frontends in the future"?

Thanks for the neovim link, its use of lua is an interesting twist.

I'm really excited by the notion of ArkI can see it working (at various levels) in ein, lsp-mode, possible even emacs-jupyter, but those don't seem to use more than one part of the Air+Ark set of features, do know of anybody trying to link to them?

[vspinu]

Ark is a Jupyter kernel that we use as an execution engine. The idea is that Positron is interacting with multiple interpreters

Hmm, this is getting exiting indeed. So would it be possible to say start jupyter console shell within emacs and connect to it dynamically as a side-car layer to ESS?

I am already relying on jupyter console for python from emacs. Do you guys planning similar stack for python as well?

[lionel-]

If this is the future, does this mean that something non-Positron that wants to use LSP will use Air, Air and optional Ark, or always Air+Ark?

I think mostly Air. Integrating with Ark is optional and harder.

So would it be possible to say start jupyter console shell within emacs and connect to it dynamically as a side-car layer to ESS?

Yes but you'd be back into a situation of interacting with raw IPC. Also jupyter-console wouldn't offer you the level of control needed to create the LSP comm. So a better setup would be to create your own jupyter client using the framework provided by the https://github.com/emacs-jupyter people. Or something derived from their REPL client.

I am already relying on jupyter console for python from emacs. Do you guys planning similar stack for python as well?

We have a fork of the python extension for vscode that contains a modified ipykernel: https://github.com/posit-dev/positron/tree/main/extensions/positron-python. It's not really meant to be used in other IDEs though.

[r2evans]

Who of the ESS devs are planning to travel to either useR! in Aug (Duke Univ, Durham, NC) or posit::conf in Sep (Atlanta, GA)? I believe @lionel- is speaking at one if not both. At a minimum, it's an opportunity for me to say "thanks" and buy you a beer/latte/something-else as a token of appreciation. I'd also be interested in hearing thoughts on ESS dev and its future, especially among the growth of other methods of interacting with inferior-processes (LSPs, including Air and Ark). Perhaps a BoF, but no need for formality. First round (of whatever) is on me.

I'm currently planning to go to posit::conf; I can swing useR! but would appreciate earlier notice if it's likely to happen :-)

@vspinu
@mmaechler
@rsparapa

I picked a few of the recent contributors; (1) my apologies if this poll is unwelcome; (2) please help if there are others I missed who should be polled

[rsparapa]

Sorry, I wish I was going to Duke: my alma mater. But, I'm not going to be present at either conference. Development has largely ceased. We are mainly in bug-fix mode.

[r2evans]

@rsparapa thanks, that is what I've been inferring for a bit. It's been a long and impactful journey! I would hate to see it wane ...

Are you still using emacs/ESS in your research? Do you have any experience with emacs and LSP? I feel that if there is going to be development in emacs and "R", it may be good to investigate if going the route of LSP could be fruitful, though I've not found a lot of it in the wild.

[rsparapa]

Of course, I use ESS every day! And I force my students to learn it as well. Sorry, I don't know what LSP is. A lot of my colleagues use it too. But I really don't know how many others there are out there.

[r2evans]

"force my students"

I'm both impressed by this, and for the non-emacs users, horrified ;-) ... many colleagues are not as keyboard-centric as I am, and learning a chording way of life can be a hurdle. My "OR" post-grad cohort had many people with non-science-y undergrad degrees, only two of them had formal CS training and one formal Math. Learning programming in general was as new as the stats/optimization we were learning/modeling (and not formally taught).

LSP = language server protocol; there is lsp-mode, and I have yet to see "R" in anybody's uses for it. The only emacs/LSP/R discussions I've seen use the languageserver R package (aptly named) which is nearing 2y since its last release (it's difficult to not read something into 12 commits in a year+ span).

LSP is how Air is envisioned to be used; Ark does LSP but also much more (such that it may not be well-suited for anything outside of Positron).

[rsparapa]

Oh, I would have thought LSP had something to do with lisp by that name. No, we are not really interested. Keep in mind that ESS is not just R. We also support SAS: mainly in batch. But the LSP project did give me an idea. We should have our own gofundme ;o)

Getting back to Ark. Also, not interested. Personally, I find the Jupyter notebooks to be more hassle than they are worth. What is wrong with an R script? And that IDE is so complicated. I usually just close all of the annoying windows. So what is the point?

[r2evans]

I'm also not using the jupyter kernel mindset, though it's not just about that: Positron (Posit's vscode-derivative that does most things RStudio does plus some more) uses Ark, and its R interface is the same REPL as before, not jupyter-like (i.e., it's still the > prompt, + continuation, Browse[1]> debugger, etc, and it is not "rmarkdown-like" where you have alternating code-chunk/output-chunk pairs).

I think the benefit of LSPs is that they relieve the editor of many of the language-specifics such as dynamic object/variable/function completion (and many other things) in a standardized way, meaning if an editor really supports the LSP, it "should" work equally well with any language that has a server-layer that implements this standardized protocol.

Especially considering the reduced efforts in maintaining ESS as its own REPL, style, language-interaction thing, I think there can be utility in offloading some of that functionality to lsp-mode.

Things I don't know, and frankly only somebody well-enough informed on the internals of both LSPs and ESS may be able to address them (I'm thinking of you, @lionel- :-)

• what are the real benefits of LSPs over the legacy ESS method?
• is there a clear distinction of functionality where ESS and lsp-mode are not just compatible but complementary?
• how different is that relationship based on the current ESS implementation? aka, what would it take to migrate this?
• is shoe-horning the ESS/lsp-mode interaction over-complicating things, i.e., is lsp-mode capable of supporting the basic REPL and functionality of R-code buffers and R-REPL interactions?

I still prefer emacs for many things. I am grateful for all of the past and current maintainers of ESS, and while I have no need of change for change's sake, I am interested in finding a way out of the slowed-rate of maintenance/development.

I'm willing to participate (though my elisp is weak) and fund as needed; I think your mention of "gofundme" is appropriate.

Thanks for the discourse!