Separate out the frontend and the backend

[davidbrochart]

Problem

jupyter-collaboration is currently under the jupyterlab organization. It includes the JupyterLab frontend and the jupyter_server backend. The backend part used to be jupyter_server_ydoc, under the jupyter-server organization. I think it was a good choice to separate out the frontend and the backend, since some extensions may want to have access to the shared models without bringing the frontend stuff. Also it would allow jupyverse to not depend (or depend less) on jupyter-server.

Proposed Solution

We could split this repository in two, and have only the JupyterLab frontend stuff in a new repository jupyterlab/jupyterlab-collaboration, which would have the backend as a dependency.
The backend part would be in jupyter-server/jupyter-collaboration.
What do you think?

cc @krassowski

[jtpio]

We could split this repository in two, and have only the JupyterLab frontend stuff in a new repository jupyterlab/jupyterlab-collaboration, which would have the backend as a dependency.
The backend part would be in jupyter-server/jupyter-collaboration.

Just to understand the issue correctly. Is the issue that the current jupyter-collaboration package includes the @jupyter/collaboration-extension JupyterLab extension by default, because it's included in the wheel as data files?

If so, another way would be to have a Python package for the backend logic, and another Python package to include both the backend and the JupyterLab extension? That way we can avoid splitting the repos again, and reduce maintenance overhead by keeping the packages in the same repo.

[davidbrochart]

Is the issue that the current jupyter-collaboration package includes the @jupyter/collaboration-extension JupyterLab extension by default, because it's included in the wheel as data files?

Yes, but also the other way around: if I just want the frontend, I get the backend too. This is how jupyverse depends on the whole jupyter-server ecosystem in general, while it should just replace it.

If so, another way would be to have a Python package for the backend logic, and another Python package to include both the backend and the JupyterLab extension? That way we can avoid splitting the repos again, and reduce maintenance overhead by keeping the packages in the same repo.

To me it's still weird to have something JupyterLab-independent and something JupyterLab-specific in the same repo.

[jtpio]

The backend part used to be jupyter_server_ydoc, under the jupyter-server organization

Are we able to find the issue / PR that initiated the move to a single jupyter-collaboration repo? To understand the original motivations for this.

[davidbrochart]

I can't find an issue or a PR, but I guess it was for ease of releasing and synchronization between the frontend and the backend.

[jtpio]

Looks like jupyter_server_ydoc was moved the jupyterlab org, renamed, and then the frontend packages were added in #76 and #77

[davidbrochart]

Thanks for digging into the history. Unfortunately, they don't say much about the motivations, or am I missing something?

[jtpio]

Some more context in jupyterlab/jupyterlab#13527.

Maybe one motivation was to make it easier for users to enable RTC, by just installing the jupyter-collaboration package.

[davidbrochart]

Probably, but this is also an issue today. We would like to install jupyterlab-collaboration to enable RTC in JupyterLab, and jupyter-collaboration for just the backend part, right? The former would also install the latter.

[krassowski]

So what is the ideal package structure then?

• jupyter-collaboration: empty python package, what most users would install, depends on:
  • jupyter-collaboration-ui: jupyterlab/notebook frontend widgets
  • jupyter-server-ydoc: jupyter-server specific: handlers, spawns websocket, implements public server-side API (#270), depends on:
    • jupyter-ydoc for shared models data structures
    • pycrdt-websocket for the plain websocket server implementation

Then jupyverse can depend on only jupyter-collaboration-ui and provide a custom equivalent of jupyter-server-ydoc, say jupyverse-ydoc. It could also have a top-level package jupyverse-collaboration?

Or should we use a different name than jupyter-server-ydoc:

• to avoid confusion with the old package
• because it is not really descriptive? It feels like jupyter-server- prefix should stay. I would say it should not be named jupyter-server-collaboration (nor -rtc because this package will be used for other things too).

Also, if #237 which adds the chat function, it will make depending on jupyter-server-ydoc still harder because it brings a dependency on jupyter-chat and thus on langchain (which I think could be removed as it is only used to import a version of pydantic and a validation error, but that's a separate story).

Or maybe:

• jupyter-collaboration, depends on:
  • jupyter-collaboration-ui
  • jupyter-server-ydoc: core shared docs handlers for jupyter-server
  • jupyter-server-chat: chat handlers, depends on:
    • jupyter-server-ydoc

Thoughts?

[krassowski]

It would also be worth thinking where suggestions (#239) come in. Naturally, it would be another package like jupyter-server-chat, say jupyter-server-suggestions. Maybe to avoid having too many packages we only extract out jupyter-server-ydoc and keep the rest of the handlers in jupyter-collaboration?

[davidbrochart]

Suggestions are deeply tied to jupyter-collaboration and jupyter_ydoc at the moment. It's not clear to me if it could be a separate thing, or just built-in.

I know it's annoying to rename things, but I would like to have things more explicit, like:

• jupyterlab-collaboration
  • jupyterlab-collaboration-ui
  • jupyter-server-something

[jtpio]

Including @fcollonval in the conversation, since he was involved in the original development of jupyter-collaboration.

[krassowski]

Though jupyterlab-collaboration and jupyterlab-collaboration-ui work with both JupyterLab and Notebook 7+, right?