Skip to content

chore: Update setup for local development for Deepnote engineers #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
OlegWock wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

chore: Update setup for local development for Deepnote engineers #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 14 additions & 53 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,64 +151,23 @@ $ ./bin/test-local

## Development Workflow for Deepnote maintainers

### Using in Deepnote Projects
### Local Toolkit Development with Webapp

When you push a commit, a new version of `deepnote/jupyter-for-local` is built with your commit hash (shortened!). Use it in projects by updating `common.yml`:
To develop toolkit changes against a local webapp with hot-reload:

```yaml
jupyter:
image: "deepnote/jupyter-for-local:SHORTENED_COMMIT_SHA"
```

Alternatively, to develop against a local copy of Deepnote Toolkit, first run this command to build the image:

```bash
docker build \
--build-arg "FROM_PYTHON_TAG=3.11" \
-t deepnote/deepnote-toolkit-local-hotreload \
-f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
```

Then start the container:

```bash
# To include server logs in the output add this argument
# -e WITH_SERVER_LOGS=1 \

# Some toolkit features (e.g. feature flags support) require
# DEEPNOTE_PROJECT_ID to be set to work correctly. Add this
# argument with your project id
# -e DEEPNOTE_PROJECT_ID=981af2c1-fe8b-41b7-94bf-006b74cf0641 \

docker run \
-v "$(pwd)":/deepnote-toolkit \
-v /tmp/deepnote-mounts:/deepnote-mounts:shared \
-p 8888:8888 \
-p 2087:2087 \
-p 8051:8051 \
-w /deepnote-toolkit \
--add-host=localstack.dev.deepnote.org:host-gateway \
--rm \
--name deepnote-toolkit-local-hotreload-container \
deepnote/deepnote-toolkit-local-hotreload
```

This will start a container with Deepnote Toolkit mounted inside and expose all required ports. If you change code that runs in the kernel (e.g. you updated the DataFrame formatter), you only need to restart the kernel from Deepnote's UI. If you update code that starts Jupyter itself, you need to restart the container. And if you add or modify dependencies you need to rebuild the image.

Now, you need to modify `common.yml` in the Deepnote app. First, replace `jupyter` service with noop image:
1. Build the local development image:
```bash
docker build -t deepnote/jupyter-for-local:local -f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
```

```yml
jupyter:
image: 'screwdrivercd/noop-container'
```
2. Setup `DEEPNOTE_TOOLKIT_SOURCE_PATH` env variable pointing to folder with toolkit source. This can go either in `.zshrc` (or similar file for your shell) or set per shell session with `export DEEPNOTE_TOOLKIT_SOURCE_PATH=...`. If not set, webapp will try to resolve it to `../deepnote-toolkit` relative to webapp root folder.

And change `JUPYTER_HOST` variable of executor to point to host machine:
3. In the webapp repository, run:
```bash
pnpm dev:app:local-toolkit
```
Comment on lines +158 to +168
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add blank lines around code fences (MD031).

Lines 159–161 and 166–168 need blank lines before and after the fenced code blocks per markdown linting standards.

Apply this diff:

 1. Build the local development image:
+
     ```bash
     docker build -t deepnote/jupyter-for-local:local -f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
     ```
+

 3. In the webapp repository, run:
+
     ```bash
     pnpm dev:app:local-toolkit
     ```
+
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)

159-159: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

166-166: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

🤖 Prompt for AI Agents 
In CONTRIBUTING.md around lines 158 to 168, the two fenced code blocks do not
have blank lines immediately before and after them (MD031); add one blank line
before the opening ```bash and one blank line after the closing ``` for both the
docker build block (lines ~159–161) and the pnpm block (lines ~166–168),
adjusting the surrounding list spacing so the numbered list remains correct and
the markdown linter no longer flags MD031.


```yml
executor:
environment:
JUPYTER_HOST: host.docker.internal
```
This mounts your toolkit source into the container and installs it in editable mode. Toolkit module code changes are reflected after kernel restart (use "Restart kernel" action in the webapp).

### Review Applications

Expand All @@ -229,6 +188,8 @@ We use Docker to ensure reproducible environments due to Jupyter libraries' bina

- `jupyter-for-local.Dockerfile`: Creates development environment with Jupyter integration, used for local development from docker-compose used in main monorepo.

- `jupyter-for-local-hotreload.Dockerfile`: Creates development environment which expectes toolkit source to be mounted at `/toolkit`. Used for development in main monorepo.

### Production Releases

To release a new version to production:
Expand Down
58 changes: 39 additions & 19 deletions dockerfiles/jupyter-for-local-hotreload/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,52 @@
ARG FROM_PYTHON_TAG
# Dockerfile for local development with hot-reload support
# This container expects the toolkit source to be mounted at /toolkit
# and installs it in editable mode for live code changes
#
# Build with:
# docker build -t deepnote/jupyter-for-local:local -f dockerfiles/jupyter-for-local-hotreload/Dockerfile .

ARG FROM_PYTHON_TAG=3.12
FROM deepnote/python:${FROM_PYTHON_TAG}

ARG FROM_PYTHON_TAG

ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies
RUN apt-get update && \
apt-get install -y openjdk-17-jdk && \
apt-get install --no-install-recommends -y \
rsync \
git \
# Required for pymssql
freetds-dev \
# Required for database connectivity through ODBC
unixodbc-dev \
# Required for secure connections (SSL/TLS)
libssl-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

RUN pip install poetry==2.2.0
# Install Poetry and required plugins
RUN pip install --no-cache-dir poetry==2.2.0 && \
poetry self add 'poetry-dynamic-versioning[plugin]>=1.0.0,<2.0.0'

WORKDIR /deepnote-toolkit
# Configure Poetry to create virtualenv outside the mounted source directory
RUN poetry config virtualenvs.path /opt/venvs

ENV POETRY_NO_INTERACTION=1 \
POETRY_VIRTUALENVS_CREATE=1 \
POETRY_VIRTUALENVS_IN_PROJECT=0 \
POETRY_CACHE_DIR=/tmp/poetry_cache
# Create toolkit directory (will be mounted over, but needed for initial setup)
RUN mkdir -p /toolkit /opt/venvs
Comment on lines 16 to +37
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider consolidating RUN instructions to reduce layer count.

The multiple RUN commands (lines 16, 30, 34, 37) can be consolidated into fewer layers. For example, combining apt-get install, Poetry installation, Poetry config, and directory creation into two or three RUN instructions would improve build efficiency and reduce image size.

Consolidation example:

-# Install system dependencies
-RUN apt-get update && \
+# Install system dependencies and Poetry
+RUN apt-get update && \
     apt-get install --no-install-recommends -y \
     rsync \
     git \
-    # Required for pymssql
     freetds-dev \
-    # Required for database connectivity through ODBC
     unixodbc-dev \
-    # Required for secure connections (SSL/TLS)
     libssl-dev && \
     apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/* && \
+    pip install --no-cache-dir poetry==2.2.0 && \
+    poetry self add 'poetry-dynamic-versioning[plugin]>=1.0.0,<2.0.0' && \
+    poetry config virtualenvs.path /opt/venvs && \
+    mkdir -p /toolkit /opt/venvs

This reduces from 4+ RUN layers to 1, improving build time and image size.

Committable suggestion skipped: line range outside the PR's diff.

🧰 Tools
🪛 Hadolint (2.14.0)

[warning] 16-16: Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

(DL3008)

[info] 37-37: Multiple consecutive RUN instructions. Consider consolidation.

(DL3059)


COPY pyproject.toml poetry.lock poetry.toml ./
WORKDIR /toolkit

RUN poetry install --no-interaction --no-ansi --with server --with dev
# Environment variables for development mode
ENV DEEPNOTE_RUNNING_IN_DEV_MODE=true \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1

ENV PYTHONPATH=/deepnote-toolkit:/deepnote-toolkit/installer:$PYTHONPATH \
TOOLKIT_BUNDLE_PATH=/deepnote-toolkit \
TOOLKIT_VERSION="local-build" \
USERNAME=user \
PASSWORD=password \
DEEPNOTE_RUNNING_IN_DEV_MODE=true \
DEEPNOTE_WEBAPP_URL="http://host.docker.internal:3002"
# Copy the entrypoint script
COPY dockerfiles/jupyter-for-local-hotreload/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

COPY dockerfiles/jupyter-for-local-hotreload/run-installer.sh /usr/local/bin/run-installer.sh
EXPOSE 8888

ENTRYPOINT ["/usr/local/bin/run-installer.sh"]
ENTRYPOINT ["/entrypoint.sh"]
28 changes: 28 additions & 0 deletions dockerfiles/jupyter-for-local-hotreload/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash
set -e

# Entrypoint script for local development container
# Installs toolkit in editable mode and starts servers via deepnote-toolkit CLI

echo "[local-toolkit] Starting local development environment..."

# Check if toolkit source is mounted
if [ ! -f "/toolkit/pyproject.toml" ]; then
echo "[local-toolkit] ERROR: Toolkit source not found at /toolkit"
echo "[local-toolkit] Make sure to mount the deepnote-toolkit directory to /toolkit"
exit 1
fi

cd /toolkit

# Mark git directory as safe (needed for poetry-dynamic-versioning)
git config --global --add safe.directory /toolkit

# Install dependencies and toolkit in editable mode
echo "[local-toolkit] Installing toolkit in editable mode..."
poetry install --extras server --no-interaction

echo "[local-toolkit] Starting servers..."

# Start servers using the toolkit CLI (handles Jupyter, LSP, config, etc.)
exec poetry run deepnote-toolkit server "$@"
42 changes: 0 additions & 42 deletions dockerfiles/jupyter-for-local-hotreload/run-installer.sh
View file
Open in desktop

This file was deleted.

Loading