chore: Update setup for local development for Deepnote engineers

Author: OlegWock

CONTRIBUTING.md

@@ -151,64 +151,23 @@ $ ./bin/test-local
 
 ## Development Workflow for Deepnote maintainers
 
-### Using in Deepnote Projects
+### Local Toolkit Development with Webapp
 
-When you push a commit, a new version of `deepnote/jupyter-for-local` is built with your commit hash (shortened!). Use it in projects by updating `common.yml`:
+To develop toolkit changes against a local webapp with hot-reload:
 
-```yaml
-jupyter:
-  image: "deepnote/jupyter-for-local:SHORTENED_COMMIT_SHA"
-```
-
-Alternatively, to develop against a local copy of Deepnote Toolkit, first run this command to build the image:
-
-```bash
-docker build \
-  --build-arg "FROM_PYTHON_TAG=3.11" \
-  -t deepnote/deepnote-toolkit-local-hotreload \
-  -f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
-```
-
-Then start the container:
-
-```bash
-# To include server logs in the output add this argument
-#  -e WITH_SERVER_LOGS=1 \
-
-# Some toolkit features (e.g. feature flags support) require
-# DEEPNOTE_PROJECT_ID to be set to work correctly. Add this
-# argument with your project id
-# -e DEEPNOTE_PROJECT_ID=981af2c1-fe8b-41b7-94bf-006b74cf0641 \
-
-docker run \
-  -v "$(pwd)":/deepnote-toolkit \
-  -v /tmp/deepnote-mounts:/deepnote-mounts:shared \
-  -p 8888:8888 \
-  -p 2087:2087 \
-  -p 8051:8051 \
-  -w /deepnote-toolkit \
-  --add-host=localstack.dev.deepnote.org:host-gateway \
-  --rm \
-  --name deepnote-toolkit-local-hotreload-container \
-    deepnote/deepnote-toolkit-local-hotreload
-```
-
-This will start a container with Deepnote Toolkit mounted inside and expose all required ports. If you change code that runs in the kernel (e.g. you updated the DataFrame formatter), you only need to restart the kernel from Deepnote's UI. If you update code that starts Jupyter itself, you need to restart the container. And if you add or modify dependencies you need to rebuild the image.
-
-Now, you need to modify `common.yml` in the Deepnote app. First, replace `jupyter` service with noop image:
+1. Build the local development image:
+    ```bash
+    docker build -t deepnote/jupyter-for-local:local -f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
+    ```
 
-```yml
-jupyter:
-    image: 'screwdrivercd/noop-container'
-```
+2. Setup `DEEPNOTE_TOOLKIT_SOURCE_PATH` env variable pointing to folder with toolkit source. This can go either in `.zshrc` (or similar file for your shell) or set per shell session with `export DEEPNOTE_TOOLKIT_SOURCE_PATH=...`. If not set, webapp will try to resolve it to `../deepnote-toolkit` relative to webapp root folder.
 
-And change `JUPYTER_HOST` variable of executor to point to host machine:
+3. In the webapp repository, run:
+    ```bash
+    pnpm dev:app:local-toolkit
+    ```
 
-```yml
-executor:
-  environment:
-    JUPYTER_HOST: host.docker.internal
-```
+This mounts your toolkit source into the container and installs it in editable mode. Toolkit module code changes are reflected after kernel restart (use "Restart kernel" action in the webapp).
 
 ### Review Applications
 
@@ -229,6 +188,8 @@ We use Docker to ensure reproducible environments due to Jupyter libraries' bina
 
 - `jupyter-for-local.Dockerfile`: Creates development environment with Jupyter integration, used for local development from docker-compose used in main monorepo.
 
+- `jupyter-for-local-hotreload.Dockerfile`: Creates development environment which expectes toolkit source to be mounted at `/toolkit`. Used for development in main monorepo.
+
 ### Production Releases
 
 To release a new version to production:

dockerfiles/jupyter-for-local-hotreload/Dockerfile

@@ -1,32 +1,52 @@
-ARG FROM_PYTHON_TAG
+# Dockerfile for local development with hot-reload support
+# This container expects the toolkit source to be mounted at /toolkit
+# and installs it in editable mode for live code changes
+#
+# Build with:
+#   docker build -t deepnote/jupyter-for-local:local -f dockerfiles/jupyter-for-local-hotreload/Dockerfile .
+
+ARG FROM_PYTHON_TAG=3.12
 FROM deepnote/python:${FROM_PYTHON_TAG}
 
+ARG FROM_PYTHON_TAG
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install system dependencies
 RUN apt-get update && \
-    apt-get install -y openjdk-17-jdk && \
+    apt-get install --no-install-recommends -y \
+    rsync \
+    git \
+    # Required for pymssql
+    freetds-dev \
+    # Required for database connectivity through ODBC
+    unixodbc-dev \
+    # Required for secure connections (SSL/TLS)
+    libssl-dev && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
-RUN pip install poetry==2.2.0
+# Install Poetry and required plugins
+RUN pip install --no-cache-dir poetry==2.2.0 && \
+    poetry self add 'poetry-dynamic-versioning[plugin]>=1.0.0,<2.0.0'
 
-WORKDIR /deepnote-toolkit
+# Configure Poetry to create virtualenv outside the mounted source directory
+RUN poetry config virtualenvs.path /opt/venvs
 
-ENV POETRY_NO_INTERACTION=1 \
-    POETRY_VIRTUALENVS_CREATE=1 \
-    POETRY_VIRTUALENVS_IN_PROJECT=0 \
-    POETRY_CACHE_DIR=/tmp/poetry_cache
+# Create toolkit directory (will be mounted over, but needed for initial setup)
+RUN mkdir -p /toolkit /opt/venvs
 
-COPY pyproject.toml poetry.lock poetry.toml ./
+WORKDIR /toolkit
 
-RUN poetry install --no-interaction --no-ansi --with server --with dev
+# Environment variables for development mode
+ENV DEEPNOTE_RUNNING_IN_DEV_MODE=true \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
 
-ENV PYTHONPATH=/deepnote-toolkit:/deepnote-toolkit/installer:$PYTHONPATH \
-    TOOLKIT_BUNDLE_PATH=/deepnote-toolkit \
-    TOOLKIT_VERSION="local-build" \
-    USERNAME=user \
-    PASSWORD=password \
-    DEEPNOTE_RUNNING_IN_DEV_MODE=true \
-    DEEPNOTE_WEBAPP_URL="http://host.docker.internal:3002"
+# Copy the entrypoint script
+COPY dockerfiles/jupyter-for-local-hotreload/entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
 
-COPY dockerfiles/jupyter-for-local-hotreload/run-installer.sh /usr/local/bin/run-installer.sh
+EXPOSE 8888
 
-ENTRYPOINT ["/usr/local/bin/run-installer.sh"]
+ENTRYPOINT ["/entrypoint.sh"]

dockerfiles/jupyter-for-local-hotreload/entrypoint.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+# Entrypoint script for local development container
+# Installs toolkit in editable mode and starts servers via deepnote-toolkit CLI
+
+echo "[local-toolkit] Starting local development environment..."
+
+# Check if toolkit source is mounted
+if [ ! -f "/toolkit/pyproject.toml" ]; then
+    echo "[local-toolkit] ERROR: Toolkit source not found at /toolkit"
+    echo "[local-toolkit] Make sure to mount the deepnote-toolkit directory to /toolkit"
+    exit 1
+fi
+
+cd /toolkit
+
+# Mark git directory as safe (needed for poetry-dynamic-versioning)
+git config --global --add safe.directory /toolkit
+
+# Install dependencies and toolkit in editable mode
+echo "[local-toolkit] Installing toolkit in editable mode..."
+poetry install --extras server --no-interaction
+
+echo "[local-toolkit] Starting servers..."
+
+# Start servers using the toolkit CLI (handles Jupyter, LSP, config, etc.)
+exec poetry run deepnote-toolkit server "$@"