Skip to content

Commit b0142d0

Browse files
ZsailerZsailer
committed
first draft of jupyter-server proposal
1 parent 5a24471 commit b0142d0

File tree

1 file changed

+216
-0
lines changed

1 file changed

+216
-0
lines changed

‎jupyter-server/jupyter-server.md

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# Standalone Jupyter server enhancement proposal [active]
2+
3+
## Problem
4+
5+
There are now multiple frontends that talk to the backend services provided by the notebook server: the legacy Notebook, the dashboards, JupyterLab, standalone widgets and more. The configuration of legacy notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server.
6+
7+
## Proposed Enhancement
8+
9+
Decouple the backend server (and its configuration) from the classic notebook frontend. This will require the following steps:
10+
11+
1. Create `jupyter_server` repository
12+
- Fork of `notebook` repository.
13+
- Pure Python package with notebook backend services.
14+
- `notebook` tornado handlers and frontend logic stay in `notebook` repo.
15+
- Deprecated notebook server APIs do not move to `jupyter_server`.
16+
2. New server extensions mechanism.
17+
- server extensions move to `jupyter_server`.
18+
- new base classes to create applications from server extensions.
19+
- nbextensions stay in `notebook`.
20+
- `jupyter_notebook_config.d` folder becomes `jupyter_server_config.d`
21+
- `notebook` becomes a server extension.
22+
3. Services become server extensions
23+
- Services are just serve extensions with dependencies.
24+
- Add dependency injection system.
25+
5. Namespacing static files and REST API urls.
26+
- Each extension serves static files at the `/static/<extension>` url.
27+
-
28+
6. Migrate configuration.
29+
- Notebook App configuration stays in `jupyter_notebook_config.py`
30+
- Server and services configurations move to `jupyter_server_config.py`
31+
- Add a `migrate` application to automate this migration.
32+
7. Add necessary documentation to notebook and jupyter_server repos.
33+
34+
## Detailed Explanation
35+
36+
### Create a `jupyter_server` repository
37+
38+
The first thing to do is fork `notebook`. A new `jupyter_server` repo will keep the server specific logic and remove:
39+
1. the notebook frontend code,
40+
2. deprecated notebook server APIs, and
41+
3. tornado handlers to the classic notebook interface. These pieces will stay in the `notebook` repository.
42+
43+
Things that stay in notebook:
44+
45+
- `edit` module: the handlers for the classic notebook text editor.
46+
- `templates` directory: all html templates for the classic notebook
47+
- `terminal` module: handlers for the classic notebook terminal application.
48+
- `view` module: handlers for file viewer component.
49+
- `static` directory: all js and style sheets for notebook frontend.
50+
- `tree` module: a classic notebook file browser+handlers.
51+
- `auth` module? *(Should this move to jupyter_server?)*
52+
53+
Things that move to jupyter_server:
54+
- `services` module: all jupyter server services, managers, and handlers
55+
- `bundler`: handlers for building download bundles
56+
- `files`: handlers for serving files from contents manager.
57+
- `kernelspec`: handler for getting kernelspec
58+
- `base`: base handler for jupyter apps.
59+
- `i18n`: module for internationalizing the jupyter server
60+
61+
62+
Preliminary work resides in [jupyter_server](https://github.com/jupyter/jupyter_server).
63+
64+
### Server Extensions
65+
66+
The extension mechanism for the *jupyter server* will be the main area where server extensions differ from notebook's server extensions.
67+
68+
Enabled server extensions will still be loaded using the `load_jupyter_server_extension` approach when the jupyter server is started.
69+
70+
**Server extensions as applications**
71+
72+
In the proposed jupyter_server, extension developers may also create an application from their extension, using a new `JupyterServerExtensionApp` class. Extension developers can subclass the new base class to make server extensions into Jupyter applications (configurable and launchable from the commmand line). This new class loads extension config from file and parses configuration set from the command line.
73+
74+
For example, the legacy notebook could be started: 1) as an enabled extension or 2) by running the traitlets application from the command line.
75+
76+
Example extension:
77+
```python
78+
from .extension import load_jupyter_server_extension
79+
80+
class NotebookApp(JupyterServerExtensionApp):
81+
82+
name = 'notebook'
83+
description = 'NotebookApp as a server extension.'
84+
load_jupyter_server_extension = staticmethod(load_jupyter_server_extension)
85+
```
86+
87+
`JupyterServerExtensionApp` subclasses are configurable using Jupyter's configuration system. Users can generate config files for each extension in the Jupyter config path (see `jupyter --paths`). Each extension's configuration is loaded when the server is initialized or the extension application is launched.
88+
89+
As will all jupyter applications, users can autogenerate a configuration file for their extension using `jupyter my_extension --generate-config`.
90+
91+
Initial experimental work resides in [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension).
92+
93+
**Classic notebook server extension**
94+
95+
The classic `NotebookApp` will become a server extension. It will inherit `JupyterServerExtensionApp`. Users can enable the notebook using the extension install/enable mechanism or enable the `notebook` in their `jupyter_server_config.py` file:
96+
```python
97+
c.ServerApp.jpserver_extensions = {
98+
'notebook': True
99+
}
100+
```
101+
Users can also launch the notebook application using the (usual)`jupyter notebook` command line interface.
102+
103+
**Extension installing/enabling mechanism**
104+
105+
The new extension mechanism in the *jupyter server* will differ from notebook's server extensions.
106+
107+
* The `--sys-prefix` installation would become the default. (Users are confused when enabling an extension requires more permissions than the installation of the package). Installation in system-wide directories would be done with the `--system` option.
108+
* Installing an extension will include the addition of a 'manifest' file into a conf.d directory (under one of the Jupyter configuration directories, `user / sys-prefix / system`). The placement of such an extension manifest provided by a Python package can be done with `jupyter server extension install --py packagename [--user / --system / --sys-prefix]`. Packages (conda or wheels) carrying server extensions could place such manifests in the sys-prefix path by default effectively installing them. Development installations would also require the call to the installation command.
109+
110+
* Enabling an extension is separate from the installation. Multiple scenarios are possible:
111+
- enabling an extension at the same level (user / sys-prefix / system) as where it was installed, or at a higher level (user for sys-prefix and system, or sys-prefix for system).
112+
- forcibly disabling an extension that was enabled at a lower level of precedence.
113+
- forcibly enabling an extension that was disabled at a lower level of precedence.
114+
This would be done via two `conf.d` configuration directories managing a list of disabled extensions and list of enabled extensions in the form of empty files having the name of the corresponding extension. If an extension is both disabled and enabled at the same level of precedence, disabling has precedence. Packages (conda or wheels) could place such a enabler file in the sys-prefix path by default. The `jupyter server extension enable` command would be required for development installations.
115+
116+
(Possibly) when an extension is enabled at a given precedence level, it may only look for the version of the extension installed at the same or lower precedence level. For example, if an extension `foobar` is installed and enabled system wide, but a user installs a version with `--user`, this version will only be picked up if the user also enables it with `--user`.
117+
118+
### Services become server extensions (with dependency injection)
119+
120+
Right now, there are two ways to extend and customize the jupyter server: services and server extensions. This is a bit confusing for new contributors. The main difference is that services often (but not always) **depend on other services**.
121+
122+
On example is the `sessions` service, which depends on the `kernels` and `kernelspec` services. Without these two services, the `sessions` service doesn't work (or even make sense).
123+
124+
We could reduce complexity around `jupyter_server` by making *everything* a server extension. We would need to add a **dependency injection** system to allow extensions to depend on other extensions. Some good options are [pyinject](https://github.com/google/pinject) or [python-dependency-injector](https://github.com/ets-labs/python-dependency-injector).
125+
126+
To port a service, it will need to be refactored using extension mechanism mentioned above.
127+
128+
### Add namespacing to `static` endpoints and REST API urls.
129+
130+
Currently, the notebook tornado application serves all static files underneath the `/static/` prefix. Jupyter server will add namespacing under the static url and extension REST API urls. Each extension will serve their static files under the `/static/<extension-name>` prefix and their API handlers behind a `/extension/api/<extension-name>` prefix.
131+
132+
For example, the classic notebook server extension will add static handlers that reroute requests to the `/static/notebook/` endpoints.
133+
134+
The jupyter_server will provide a new `JupyterExtensionHandler` base class that reroute requests to the extension's namespaced static and REST API endpoints.
135+
136+
Preliminary experimental work resides in the [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension) repository.
137+
138+
### Configuration System
139+
140+
The configuration of the server and the legacy notebook are currently tightly coupled in a single NotebookApp configurable. The proposed jupyter server has server-specific configurations in a separate config system from the classic notebook (and jupyterlab) application configurations.
141+
142+
The table below show all the classic notebook traits and where they will live after migration.
143+
144+
**Overview of configuration structure**
145+
146+
The notebook and server configuration both live in the `jupyter_notebook_config.py` file. Extensions are configured in separate files (named after the extension) in the `jupyter_notebook_config.d` directory:
147+
```
148+
~/.jupyter/
149+
├── jupyter_notebook_config.py
150+
└── jupyter_notebook_config.d
151+
└── my_extension.json
152+
```
153+
154+
**New proposed configuration structure**
155+
156+
The jupyter_server configuration lives in the `jupyter_server_config.py` file. Extensios are configured in separate files in the `jupyter_server_config.d` folder. The notebook configuration is stored in a `notebook.py` file, just like other extensions.
157+
```
158+
~/.jupyter/
159+
├── jupyter_server_config.py
160+
└── jupyter_server_config.d
161+
├── notebook.py|json
162+
├── lab.py|json
163+
└── my_extension.py|json
164+
```
165+
166+
**Migration application**
167+
168+
To make migration easier on users, the jupyter server will include a `migrate` application to split notebook and server configurations into their appropriate locations (listed above). This application will find any `jupyter_notebook_config.py|json` files in `jupyter --paths`, read configured traits, sort server vs. notebook traits, and write them to the appropriate `jupyter_server_config.py|json` and `jupyter_notebook_config.py|json` files.
169+
170+
171+
### How this effects other projects
172+
173+
[**Classic notebook**]()
174+
In short, the classic notebook will become a server extension application. The rest of this proposal describes the details behind what will change in the notebook repo.
175+
[`JupyterServerExtensionApp`]().
176+
177+
[**Jupyter Lab**]()
178+
Jupyter lab will also become a server extension application. The new classes described above should simplify the way JupyterLab interfaces with the server.
179+
180+
[**Kernel Gateway**]()
181+
Kernel Gateway writes custom `kernel` and `kernelmanager` services and load them as server extensions.
182+
183+
[**Kernel Nanny**]()
184+
185+
## Pros and Cons
186+
187+
**Pros** associated with this implementation include
188+
189+
* Allow various frontends to use the backend services of the jupyter server.
190+
* Provides base classes for extension developers writing server extension applications and handlers.
191+
* Reduce complexity when extending the server by making everything a server extension.
192+
* Organizes the configuration for server and extension in a sane and logical manner (`conf.d` approach).
193+
194+
**Cons** associated with this implementation include:
195+
196+
* Break the classic notebook in a backwards incompatible way.
197+
* Affects many projects. The transition may be painful?
198+
* Adding a dependency injection system adds new complexity.
199+
200+
## Relevent Issues, PRs, and discussion
201+
202+
Moving to a `conf.d` approach.
203+
* PR [3116](https://github.com/jupyter/notebook/pull/3116), `jupyter/notebook`: extension config in `config.d` directory.
204+
* PR [3782](https://github.com/jupyter/notebook/issues/3782), `jupyter/notebook`: server extension use `conf.d` approach
205+
* PR [2063](https://github.com/jupyter/notebook/issues/2063), `jupyter/notebook`: config merging problems.
206+
207+
Conversation on server/notebook extensions:
208+
* PR [1706](https://github.com/jupyter/notebook/issues/1706), `jupyter/notebook`: proposal to improve server/notebook extensions
209+
* PR [2824](https://github.com/jupyter/notebook/issues/2824), `jupyter/notebook`: enable nbextensions by default
210+
211+
Static Namespace:
212+
* PR [21](https://github.com/jupyter/enhancement-proposals/pull/21#issuecomment-248647152)`jupyter/enhancement-proposals`: mention namespacing.
213+
214+
## Interested
215+
216+
@Zsailer, @SylvainCorlay, @ellisonbg, @blink1073, @kevin-bates

0 commit comments

Comments
 (0)