feat: DH-20578: Groovy remote file sourcing #7451

bmingles · 2025-11-26T22:40:53Z

DH-20578: This PR implements a RemoteFileSource plugin in Deephaven Core that can request file sources from a remote client. Key points of interest

RemoteFileSourceClassLoader.java - registered with the Groovy session to request remote file sources
RemoteFileSourceCommandResolver.java - resolves a flight command to an instance of the plugin service. This allows us to not have to register a magic variable in the session. It exports a PluginMarker object that is associated with a plugin name that the RemoteFileSourcePlugin later recognizes as an object it can suppport
RemoteFileSourcePlugin - the plugin
PluginMarker - named object used to match the flight command with the plugin. Needed since all plugins require an exported object of some sort

Related PRs

deephaven/vscode-deephaven/pull/273 implements the client side code to be used by the VS Code extension.
deephaven-ent/iris/pull/3878 is needed to enable this in Core+

Testing

Install the client side code in VS Code vscode-deephaven-1.0.10-remote.0.vsix

Community

Run Core Groovy server from this branch (I use IntelliJ debug config :web-client-api:gwtCompile server-jetty-app:run -Panonymous -Pgroovy)
Clone this repo https://github.com/bmingles/deephaven-groovy-remote-docs-demo and open the folder in VS Code
In the Deephaven view, expand the Workspace/Groovy/groovy-remote-docs-demo folder under REMOTE IMPORT SOURCES and click the + button to add test folder.
Open docs-sample.groovy file and run against the Deephaven server. Should see testTable panel show up.

Core+

Changes are deployed to bmingles-remote-file-source vm. Can also configure VS Code to connect to https://bmingles-remote-file-source.int.illumon.com:8123/ and run same code

github-actions · 2025-11-26T22:41:04Z

No docs changes detected for `325f688`

bmingles · 2026-01-02T21:50:11Z

server/jetty-app-11/build.gradle

    implementation project(':server-jetty-11')

    implementation project(':extensions-flight-sql')
+    implementation project(':plugin-remotefilesource')


Not sure if these should have been added here since I don't see any other plugins.

bmingles · 2026-01-02T21:50:18Z

server/jetty-app/build.gradle

    implementation project(':server-jetty')

    implementation project(':extensions-flight-sql')
+    implementation project(':plugin-remotefilesource')


Not sure if these should have been added here since I don't see any other plugins.

…H-20578)

…0578)

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

niloc132 · 2026-01-06T15:52:15Z

web/client-api/src/main/java/io/deephaven/web/client/api/JsProtobufUtils.java

+/**
+ * Utility methods for working with protobuf messages in the client-side JavaScript environment.
+ */
+public class JsProtobufUtils {


...how sure are we that we actually need any/all of this? doesnt the generated message types give us this for free?

I thought we needed it in order to support the Any wrapper?

niloc132 · 2026-01-06T15:52:49Z

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

+@JsType(namespace = "dh.remotefilesource", name = "RemoteFileSourceService")
+public class JsRemoteFileSourceService extends HasEventHandling {
+    /** Event name for generic messages from the server */
+    @JsProperty(namespace = "dh.remotefilesource.RemoteFileSourceService")


this annotation shouldnt be necessary, should get it from the type's namespace+name, ditto below

I got warnings for the @link #EVENT_MESSAGE in the header comment. Claude suggested this fix which worked. Is there a better way?

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

plugin/src/main/java/io/deephaven/plugin/type/PluginMarker.java

niloc132 · 2026-01-06T16:28:45Z

...efilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceCommandResolver.java

+                    "RemoteFileSourcePluginFetchRequest must contain a valid result_id");
+        }
+
+        final String pluginName = request.getPluginName();


it appears that DeephavenRemoteFileSourcePlugin is always the pluginName set by the client, so we end up with exactly one PluginMarker. Do we anticipate more of these, or specific clients who can control this? Nothing seems able to clear old instances, do we want a mechanism (liveness?) for that?

I anticipate that the Python version of the plugin may make use of this at some point.

Clearing instances is probably a good idea. Any suggestions on best way to do that?

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch 3 times, most recently from a48276f to fc106d6 Compare December 24, 2025 16:23

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from 777b0b4 to 4d1b51c Compare January 2, 2026 17:53

bmingles commented Jan 2, 2026

View reviewed changes

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from 3d7a03e to f0ced57 Compare January 5, 2026 21:14

bmingles and others added 22 commits January 5, 2026 15:15

Started building out RemoteFileSource plugin (#DH-20578)

6ec786d

Command resolver now fetches plugin (#DH-20578)

bd985ac

RemoteFileSourceTicketResolverFactoryService (#DH-20578)

51bb7b5

Added stub for JsRemoteFileSourceService (#DH-20578)

c282d40

Generated GWT bindings

1735733

Fetching plugin service working (#DH-20578)

c051a01

Wiring up messagestream (#DH-20578)

ae7d97e

test bidirectional communication (#DH-20578)

ef53eeb

set connection id (#DH-20578)

d3fe8cd

Moved clientSessionId to plugin fetch instead of separate message (#D…

f8e6dc4

…H-20578)

Cleanup (#DH-20578)

e5d8725

set execution context (#DH-20578)

361019a

Simplified execution context (#DH-20578)

2a1525a

Basic file sourcing is working (#DH-20578)

36143fa

Comments and cleanup (#DH-20578)

8246036

Made method private (#DH-20578)

1998da3

Made method static (#DH-20578)

74e193e

onResourceRequest method (#DH-20578)

59afc07

JS API types (#DH-20578)

af3eef0

Refactored to use PluginMarker (#DH-20578)

07f6b8e

Re-using JsWidget for message stream (#DH-20578)

3d7ebc7

Added pluginType field to PluginMarker (#DH-20578)

eea7214

bmingles added 16 commits January 5, 2026 15:15

Cleanup (#DH-20578)

51e4f9d

Split out JsProtobufUtils (#DH-20578)

434f5c8

Made canSourceResource synchronous (#DH-20578)

cf85dbd

renamed arg (#DH-20578)

8298fef

Cleanup (#DH-20578)

097607b

Sorted members (#DH-20578)

bf47495

Cleanup RemoteFileSourceCommandResolver (#DH-20578)

803a9ce

Cleanup RemoteFileSourceMessageStream (#DH-20578)

47c1ca2

Cleanup RemoteFileSourcePlugin (#DH-20578)

ac126af

Cleanup RemoteFileSourceTicketResolverFactoryService (#DH-20578)

23b14f9

Cleanup PluginMarker (#DH-20578)

440bd23

Changed back to resourceName convention to match class loaders (#DH-2…

9e74429

…0578)

Moved method (#DH-20578)

140b3ac

Cleanup JsProtobufUtils (#DH-20578)

77ff6b3

Cleanup JsRemoteFileSourceService (#DH-20578)

df05a62

Applying Colin's client proto generation (#DH-20578)

b70f606

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from f0ced57 to b70f606 Compare January 5, 2026 21:15

Changed to runtime dependencies and fixed docs links (#DH-20578)

52feab6

bmingles marked this pull request as ready for review January 5, 2026 23:23

bmingles requested review from cpwright, devinrsmith, niloc132 and rcaudy as code owners January 5, 2026 23:23

bmingles requested a review from mofojed January 5, 2026 23:23

bmingles mentioned this pull request Jan 5, 2026

feat: DH-20578: Groovy remote file source deephaven/vscode-deephaven#273

Draft

niloc132 reviewed Jan 6, 2026

View reviewed changes

bmingles added 4 commits January 6, 2026 12:47

Addressed review comments in RemoteFileSourceClassLoader (#DH-20578)

f46e97d

Addressed review comments in PluginMarker (#DH-20578)

5cc3673

Addressed review comments in RemoteFileSourceMessageStream (#DH-20578)

7ad0a36

Addressed review comments in JsRemoteFileSourceService (#DH-20578)

325f688

feat: DH-20578: Groovy remote file sourcing #7451

Are you sure you want to change the base?

feat: DH-20578: Groovy remote file sourcing #7451

Conversation

bmingles commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related PRs

Testing

Community

Core+

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No docs changes detected for 325f688

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bmingles commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

No docs changes detected for `325f688`