Migrate datalayer specific code to core, update collaboration, state and kernels! (#389)

Author: goanpeca

CLAUDE.md

@@ -1,66 +1,66 @@
-# Jupyter UI - AI Assistant Guide
+# Jupyter UI - Quick Reference
 
-## Quick Overview
+## Overview
 
-React component library for Jupyter notebooks. Monorepo with 4 packages managed by Lerna.
+React component library for Jupyter notebooks. Monorepo with 4 main packages.
 
-## Core Packages
-
-- `@datalayer/jupyter-react` - React components for notebooks, cells, terminals
-- `@datalayer/jupyter-lexical` - Rich text editor integration
-- `@datalayer/jupyter-docusaurus-plugin` - Docusaurus plugin
-- `datalayer-jupyter-vscode` - VS Code extension
-
-## Essential Commands
+## Commands
 
 ```bash
-npm install                   # Install dependencies
-npm run build                 # Build all packages
-npm run jupyter:server        # Start Jupyter server (port 8686)
-npm run storybook            # Start Storybook (port 6006)
-npm run lint                 # Check errors only (--quiet)
-npm run lint:fix             # Auto-fix issues
-npm run format               # Format code
-npm run type-check           # TypeScript checking
-npm run check                # Run all checks (format, lint, type)
-npm run check:fix            # Auto-fix and check all
-npm test                     # Run tests
+# Setup
+npm install
+npm run build
+
+# Development
+npm run jupyter:server       # Start Jupyter (port 8686)
+npm run storybook           # Start Storybook (port 6006)
+
+# React package dev (cd packages/react)
+npm run start               # Remote server config
+npm run start-local         # Local server (webpack + jupyter)
+npm run start-local:webpack # Only webpack
+
+# Code quality
+npm run check               # Format, lint, type-check
+npm run check:fix          # Auto-fix and check
 ```
 
-## Requirements
+## Key Info
 
-- Node.js >= 20.0.0 (use .nvmrc)
-- npm (not yarn)
-- Server token: `60c1661cc408f978c309d04157af55c9588ff9557c9380e4fb50785750703da6`
+- **Node.js**: >= 20.0.0 (use .nvmrc)
+- **Server token**: `60c1661cc408f978c309d04157af55c9588ff9557c9380e4fb50785750703da6`
+- **Webpack entry**: Edit `packages/react/webpack.config.js` → `ENTRY` variable
+- **Jupyter config**: `dev/config/jupyter_server_config.py`
 
-## Key Files
+## Collaboration Setup
 
-- `eslint.config.js` - ESLint v9 flat config
-- `.prettierrc.json` - Formatter config
-- `.prettierignore` - Excludes MDX files
-- `patches/` - Third-party fixes (auto-applied)
-- `packages/react/webpack.config.js` - Build config
+1. Install: `pip install jupyter-collaboration jupyterlab`
+2. Enable: Set `c.LabApp.collaborative = True` in jupyter config
+3. Test: Open http://localhost:3208/ in multiple windows
 
-## Recent Fixes (2024)
+## Collaboration Providers
 
-- MDX comments: `{/_` → `{/** **/}` in 13 files
-- Node requirement: 18 → 20+
-- Webpack warnings: 7 → 2 (source-map exclusions)
-- @jupyterlite patch for missing logos
-- ESLint v9 flat config migration
-- React 18 deprecations fixed
-- Storybook CI: Added wait-on and --url for test reliability
-- Terminal component: Fixed BoxPanel initialization issue
+```tsx
+// Basic usage
+const provider = new JupyterCollaborationProvider();
+<Notebook collaborationProvider={provider} path="notebook.ipynb" />;
+
+// With config
+const provider = new JupyterCollaborationProvider({
+  path: 'notebook.ipynb',
+  serverSettings: mySettings,
+});
+```
 
-## Common Issues
+## Troubleshooting
 
-1. **Storybook errors**: Check MDX syntax, run `npx patch-package`
-2. **Node version**: Use Node 20+ (`nvm use`)
-3. **Lint errors**: Run `npm run lint:fix`
-4. **Build fails**: Run `npm run type-check`
+- **Build fails**: Run `npm run type-check`
+- **Lint errors**: Run `npm run lint:fix`
+- **Node version**: Use Node 20+ (`nvm use`)
+- **Collaboration issues**: Check WebSocket connections and jupyter-collaboration installation
 
-## AI Assistant Notes
+## Development Tips
 
-- Always use npm, not yarn
+- Use npm, not yarn
 - Prefer editing over creating files
-- Run lint/type checks before committing
+- Run checks after changes: `npm run check:fix`

README.md

@@ -20,12 +20,12 @@ Jupyter UI is a set of [React.js](https://react.dev) components that allow a fro
 
 ## 📦 Packages
 
-| Package                                                              | Version                                                                                                                                                                                         | Description                                   |
-| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
-| [@datalayer/jupyter-react](./packages/react)                         | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-react)](https://www.npmjs.com/package/@datalayer/jupyter-react)                                                                         | Core React components for Jupyter integration |
-| [@datalayer/jupyter-lexical](./packages/lexical)                     | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-lexical)](https://www.npmjs.com/package/@datalayer/jupyter-lexical)                                                                     | Rich text editor with Lexical framework       |
-| [@datalayer/jupyter-docusaurus-plugin](./packages/docusaurus-plugin) | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-docusaurus-plugin)](https://www.npmjs.com/package/@datalayer/jupyter-docusaurus-plugin)                                                 | Docusaurus plugin for Jupyter notebooks       |
-| [datalayer-jupyter-vscode](./packages/vscode)                        | [![marketplace](https://img.shields.io/visual-studio-marketplace/v/datalayer.datalayer-jupyter-vscode)](https://marketplace.visualstudio.com/items?itemName=datalayer.datalayer-jupyter-vscode) | VS Code extension                             |
+| Package                                                              | Version                                                                                                                                                                                         | Description                             |
+| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
+| [@datalayer/jupyter-react](./packages/react)                         | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-react)](https://www.npmjs.com/package/@datalayer/jupyter-react)                                                                         | Generic React components for Jupyter    |
+| [@datalayer/jupyter-lexical](./packages/lexical)                     | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-lexical)](https://www.npmjs.com/package/@datalayer/jupyter-lexical)                                                                     | Rich text editor with Lexical framework |
+| [@datalayer/jupyter-docusaurus-plugin](./packages/docusaurus-plugin) | [![npm](https://img.shields.io/npm/v/@datalayer/jupyter-docusaurus-plugin)](https://www.npmjs.com/package/@datalayer/jupyter-docusaurus-plugin)                                                 | Docusaurus plugin for Jupyter notebooks |
+| [datalayer-jupyter-vscode](./packages/vscode)                        | [![marketplace](https://img.shields.io/visual-studio-marketplace/v/datalayer.datalayer-jupyter-vscode)](https://marketplace.visualstudio.com/items?itemName=datalayer.datalayer-jupyter-vscode) | VS Code extension                       |
 
 ## 🚀 Quick Start
 
@@ -38,21 +38,65 @@ npm install @datalayer/jupyter-react
 ### Basic Usage
 
 ```tsx
-import { Jupyter, Notebook } from '@datalayer/jupyter-react';
+import { JupyterReactTheme, Notebook } from '@datalayer/jupyter-react';
 
 function App() {
   return (
-    <Jupyter
-      jupyterServerUrl="http://localhost:8686"
-      jupyterServerToken="your-token"
-      startDefaultKernel
-    >
-      <Notebook />
-    </Jupyter>
+    <JupyterReactTheme>
+      <Notebook path="notebook.ipynb" id="notebook-id" startDefaultKernel />
+    </JupyterReactTheme>
   );
 }
 ```
 
+### Collaborative Editing
+
+Jupyter UI supports real-time collaboration through a pluggable provider system:
+
+```tsx
+import {
+  Notebook,
+  JupyterCollaborationProvider,
+} from '@datalayer/jupyter-react';
+
+function CollaborativeNotebook() {
+  const collaborationProvider = new JupyterCollaborationProvider();
+
+  return (
+    <Notebook
+      path="notebook.ipynb"
+      collaborationProvider={collaborationProvider}
+    />
+  );
+}
+```
+
+#### Creating Custom Collaboration Providers
+
+You can create your own collaboration provider by extending `CollaborationProviderBase`:
+
+```tsx
+import { CollaborationProviderBase } from '@datalayer/jupyter-react';
+
+class MyCustomProvider extends CollaborationProviderBase {
+  constructor(config) {
+    super('my-provider-type');
+    // Initialize your provider
+  }
+
+  async connect(sharedModel, documentId, options) {
+    // Implement your connection logic
+    // Set up WebSocket, authenticate, etc.
+  }
+}
+
+// Use it with any Notebook component
+const provider = new MyCustomProvider({
+  /* config */
+});
+<Notebook collaborationProvider={provider} path="notebook.ipynb" />;
+```
+
 ### Development Setup
 
 As a developer start with the [setup of your environment](https://jupyter-ui.datalayer.tech/docs/develop/setup) and try [one of the examples](https://jupyter-ui.datalayer.tech/docs/category/examples). We have [documentation](https://jupyter-ui.datalayer.tech) for more details.
@@ -100,12 +144,21 @@ We host a Storybook on ✨ https://jupyter-ui-storybook.datalayer.tech that show
 ### Advanced Features
 
 - **🔌 IPyWidgets Support** - Full support for interactive widgets
-- **👥 Collaborative Editing** - Real-time collaboration using Y.js
+- **👥 Collaborative Editing** - Pluggable provider system supporting:
+  - Jupyter collaboration (WebSocket-based with Y.js)
+  - Custom providers via `ICollaborationProvider` interface
 - **🎨 Theming** - JupyterLab theme support with dark/light modes
 - **🔧 Extensible** - Plugin system for custom functionality
 - **🚀 Performance** - Virtual scrolling, lazy loading, and optimizations
 - **🔒 Security** - Token authentication, CORS, XSS protection
 
+### Architecture Highlights
+
+- **🏗️ Clean Architecture** - Modular, composable components with clear interfaces
+- **🔄 Composition Pattern** - Components compose rather than inherit for maximum flexibility
+- **🔌 Provider System** - Pluggable collaboration providers for different backends
+- **📦 One-way Dependencies** - Core depends on jupyter-react, not vice versa
+
 <div align="center" style="text-align: center">
   <img alt="Jupyter UI Gallery" src="https://datalayer-jupyter-examples.s3.amazonaws.com/jupyter-react-gallery.gif" />
 </div>
@@ -151,7 +204,7 @@ We maintain a plugin for [Docusaurus](https://docusaurus.io) in the [docusaurus-
 
 ## 📋 Requirements
 
-- **Node.js** >= 18.0.0
+- **Node.js** >= 20.0.0
 - **npm** >= 8.0.0
 - **Python** >= 3.8 (for Jupyter server)
 - **JupyterLab** >= 4.0.0

SUMMARY.md

@@ -4,6 +4,15 @@
 
 Jupyter UI is an open-source React.js component library that bridges the gap between the Jupyter ecosystem and modern web development frameworks. It provides React components that are 100% compatible with Jupyter, allowing developers to build custom data products without being constrained by the traditional JupyterLab interface.
 
+## Recent Updates (January 2025)
+
+### Collaboration Provider System
+
+- Implemented plugin-based architecture for extensible collaboration
+- Built-in providers: `JupyterCollaborationProvider`, `NoOpCollaborationProvider`
+- Direct instantiation pattern for simplicity
+- Added collaboration support to `Notebook2` component
+
 ### Core Problem Solved
 
 Traditional JupyterLab uses the Lumino widget toolkit, an imperative UI framework that isn't compatible with modern declarative frameworks like React. This forces developers to either:
@@ -23,7 +32,7 @@ The project uses Lerna to manage a monorepo structure with the following organiz
 ```
 jupyter-ui/
 ├── packages/           # Core library packages
-│   ├── react/         # Main React component library
+│   ├── react/         # Main React component library (generic)
 │   ├── lexical/       # Rich text editor integration
 │   ├── docusaurus-plugin/  # Docusaurus integration
 │   └── vscode/        # VS Code extension
@@ -60,12 +69,15 @@ The main package providing React components for Jupyter functionality.
 - Provides React context providers for state management
 - Supports both local and remote Jupyter servers
 - Implements WebSocket communication for real-time updates
+- Plugin-based collaboration provider system
+- Extensible without platform-specific code
 
 **Key Files:**
 
 - `src/jupyter/JupyterContext.tsx` - Core context provider
-- `src/components/notebook/Notebook.tsx` - Main notebook component
+- `src/components/notebook/Notebook.tsx` - Main notebook component (accepts collaborationProvider)
 - `src/providers/ServiceManagerProvider.tsx` - Service management
+- `src/jupyter/collaboration/ICollaborationProvider.ts` - Provider interface
 
 ### 2. @datalayer/jupyter-lexical (v1.0.3)
 
@@ -454,11 +466,11 @@ The repository includes several example implementations:
 
 ## Community & Ecosystem
 
-- Active development by Datalayer, Inc.
 - MIT licensed
 - Integration with major React frameworks
 - Storybook for component documentation
 - Comprehensive documentation site
+- Active development by Datalayer, Inc.
 
 ## Future Roadmap (Based on Code Structure)
 

dev/config/jupyter_server_config.py

@@ -106,4 +106,4 @@
 # JupyterLab
 #################
 
-c.LabApp.collaborative = False
+c.LabApp.collaborative = True

dev/notebooks/collaboration.ipynb

@@ -0,0 +1,93 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Collaboration Example\n",
+    "This notebook is for testing real-time collaboration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-08-18T06:03:49.869760Z",
+     "iopub.status.busy": "2025-08-18T06:03:49.869446Z",
+     "iopub.status.idle": "2025-08-18T06:03:49.875095Z",
+     "shell.execute_reply": "2025-08-18T06:03:49.874611Z",
+     "shell.execute_reply.started": "2025-08-18T06:03:49.869729Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Hello from collaboration notebook!\n"
+     ]
+    }
+   ],
+   "source": [
+    "print('Hello from collaboration notebook!')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-08-18T06:03:51.258535Z",
+     "iopub.status.busy": "2025-08-18T06:03:51.258136Z",
+     "iopub.status.idle": "2025-08-18T06:03:51.262464Z",
+     "shell.execute_reply": "2025-08-18T06:03:51.261483Z",
+     "shell.execute_reply.started": "2025-08-18T06:03:51.258506Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "x + y = 30\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test collaboration by editing this cell\n",
+    "x = 10\n",
+    "y = 20\n",
+    "print(f'x + y = {x + y}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}