Remove usage of `r_task()` in the LSP

[lionel-]

Parent issue: Remove usage of r_task() #691

We should consider two setups:

• Ark + Oak. The LSP runs alongside a user-operated Jupyter kernel. It has access to the user's R session.
• Oak. The LSP runs in standalone mode. It still has access to a sidecar R session, with or (preferably) without the Jupyter harness.

There are 3 categories of handlers:

• Static: can be rewritten with static analysis (package metadata, workspace index, semantic index).
• Help: depend on R's help server (see RHtmlHelp), and will remain as R queries.
• Dynamic: require runtime object state (data frames defined in the session, environment variables, R options, ...)

The most important category is the first one. Rewriting on top of static analysis:

• Makes the standalone LSP actually useful
• Makes the LSP in Positron more accurate, and sensitive to where cursor position is (e.g. package completions provided only when cursor is after corresponding library() call).

Rewriting with static analysis

Main sources of static information:

• Package metadata: reading from installed package files like NAMESPACE and DESCRIPTION, building on the existing Library / Package infrastructure used for diagnostics.

• Source code: Whereas information like exports is readily available from package metadata, a lot of context is missing from these files, such as whether an export is a function, what its formals are, what are the internal symbols in a namepsace, etc. This information requires analysing top-level assignments in R source files, which aren't readily available in installed packages. That's the main blocking point. Some ideas in Ark: Download package source code from CRAN tarballs positron#2286.

• Semantic index: a new index modelled after ty internals that tracks scopes and usage/definitions. Needed for position-sensitive completions and symbol resolution.

Package info will need to be merged with local scope information from the semantic index for callee resolution, function lookup, etc.

How can we set up dynamic queries

Help and dynamic queries need to be resolved by R, from a session whose .libPaths() matches the LSP. In packages, help for development documentation provided via pkgload needs some more thinking.

The simplest setup is to have dynamic features only available when R is idle. The queries would be sent from the LSP to the Console event loop and processed in between execute requests.

But this could be improved:

• Help: Serve help from a side car process? Fine with standalone Oak, as the side car runtime can live in the Oak process itself, but not ideal in Ark mode where the user R session already lives in the process. We could spawn a side car process but we should be mindful of process proliferation (e.g. in the context of multi-session, notebooks, etc). Maybe we can reuse the same side car process for multiple Ark instances, if same version of R and same libPaths? Interestingly, such a reusable side car process could be structured as a Jupyter kernel with Ark instances as clients.

• Dynamic state: We could create a static view of global objects in the R session and update it between execute requests.

My feeling is that it's more important to get Help to work while R is busy than dynamic queries. We shouldn't worry too much about the latter, especially at first.

Current r_task() usages

Here's a quick survey of r_task() calls in the LSP.

Static: can be rewritten with static analysis

• .libPaths() initialization in GlobalState::new(). Already has a FIXME. Resolvable from env vars / R install path. Small, self-contained change.

• Roxygen tag completions in completions_from_comment(). system.file("roxygen2-tags.yml"). Just a file path lookup into the library. Trivial once .libPaths() is resolved statically.

• Custom completions: library() / require() in completions_from_custom_source(). Installed package listing comes from scanning .libPaths() directories on disk. Library already has the infrastructure.

• Namespace :: / ::: completions in completions_from_namespace(). getNamespace(), R_lsInternal(), promise forcing. Exports are already in Package::namespace and Package::exported_symbols. Main gap: determining whether each export is a function (for () appending) needs new metadata constructed from analysing top-level assignments. Note: ::: lists all internal symbols, not just exports. That info isn't in NAMESPACE, it also requires analysing top-level assignments.

• Call argument completions (session) in completions_from_session_arguments(). parse_eval of callee + .ps.completions.formalNames(). Workspace fallback already static. Needs a static index of package function formals (not yet in Package), analysed from top-level assignments. S3 dispatch part is a complication.

• Signature help (formals + matching) in r_signature_help(). Callee lookup, r_formals(), argument matching. Same static formals index as call argument completions, but also needs default values for signature display. Param docs part is Help category.

• Search path completions in completions_from_search_path(). Needs detection of library()/require() calls and position-sensitive scope analysis. Static analysis is actually better here: completions can be position-sensitive (before vs. after a library() call). The only truly dynamic residual is objects created in the console.

Help: will remain as R queries

• Completion resolve in resolve_completion(). RHtmlHelp::from_function(), from_topic(), parameter() for functions, packages, and parameters.

• Hover in r_hover(). RHtmlHelp::from_function() to show function docs on hover.

• Signature help (param docs) in r_signature_help(). RHtmlHelp::from_function() + parameter(). Same r_task() as signature help item in the Static section.

Dynamic: require R runtime state

• $ / @ extractor completions in completions_from_extractor_object(). Evaluates LHS object, calls .DollarNames() / .AtNames(). Requires runtime object structure.

• Pipe root column completions in find_pipe_root(). Evaluates pipe root variable, calls names(). Requires runtime object structure.

• Subset [ / [[ completions in completions_from_subset() / completions_from_evaluated_object_names(). Evaluates object, calls names() / colnames(). Requires runtime object structure.

• Custom completions: env vars, options in completions_from_custom_source(). Sys.getenv() -> env vars, getOption() / options() -> R options. Truly runtime state.

• First argument eval (S3 dispatch) in get_first_argument(). Evaluates first arg to get class for method dispatch.

• Backend lifecycle in report_crash() and start_lsp(). Crash reporting UI message, LSP channel wiring. Infrastructure, not a query.

• Input boundaries (parse status) in input_boundaries(). Not really dynamic state but requires calling R_ParseVector().

[lionel-]

Here is one usage of r_task() that seems legit: showing a crash report in the frontend:

fn report_crash() {
    let user_message = concat!(
        "The R language server has crashed and has been disabled. ",
        "Smart features such as completions will no longer work in this session. ",
        "Please report this crash to https://github.com/posit-dev/positron/issues ",
        "with full logs (see https://positron.posit.co/troubleshooting.html#python-and-r-logs)."
    );

    r_task(|| {
        Console::get()
            .ui_comm()
            .show_message(String::from(user_message));
    });
}

The Console owns the UI comm so we need to send the message via Console. There is no execution of R code or access to R state at all, so running at interrupt time is perfectly fine.

We could have interrupt tasks take a Console subset as argument that only exposes safe methods. The idle tasks would get a larger subset, more similar to what comm handlers get.

In general, we'd avoid Console::get() and instead take the relevant Console subset as argument, containing only safe operations.

[DavisVaughan]

@lionel- for this particular example I feel like this could also just be a crossbeam channel

i.e.

fn report_crash() {
    lsp_crash_tx.send(())
}

and the lsp_crash_rx receiver lives on Console and will check if there are any messages either in the idle time loop or at each interrupt time check

[lionel-]

You're right of course. We should do that if that allows us to remove r_task(), even if in that case the usage is sound.