Merge pull request #185 from minrk/kv-abstract

reduce requirements of KV store implementations and custom methods

Author: minrk

docs/source/details.md

@@ -0,0 +1,83 @@
+# Implementation details
+
+## Traefik API
+
+traefik-proxy uses the [Traefik API](https://doc.traefik.io/traefik/operations/api/) to monitor routes and configurations.
+
+Because of **security** concerns, in traefik-proxy implementation, traefik api endpoint isn't exposed on the public http endpoint. Instead, it runs on a dedicated **authenticated endpoint** that's on localhost by default.
+
+The port on which traefik-proxy's api will run, as well as the username and password used for authenticating, can be passed to the proxy through `jupyterhub_config.py`, e.g.:
+
+```
+c.TraefikFileProviderProxy.traefik_api_url = "http://127.0.0.1:8099"
+c.TraefikFileProviderProxy.traefik_api_password = "admin"
+c.TraefikFileProviderProxy.traefik_api_username = "admin"
+```
+
+Check out TraefikProxy's [API Reference](TraefikProxy) for more configuration options.
+
+## Class structure
+
+A JupyterHub Proxy implementation must implement these methods:
+
+- `start` / `stop` (starting and stopping the proxy)
+- `add_route`
+- `delete_route`
+- `get_all_routes`
+- `get_route` (a default implementation is provided by the base class, based on get_all_routes)
+
+Additionally, for traefik we need to set up the separate "static config" (API access and where routes will be stored) and "dynamic config" (the routing table itself, which changes as servers start and stop).
+Where and how dynamic_config is stored is the ~only difference between TraefikProxy subclasses.
+
+Setting up traefik configuration is in these methods:
+
+- `_setup_traefik_static_config` - must be extended by subclasses to add any provider-specific configuration to `self.static_config`
+- `_setup_traefik_dynamic_config` - usually not modified
+
+TraefikProxy is organized into three levels:
+
+First, is [](TraefikProxy). This class is responsible for everything traefik-specific.
+It implements talking to the traefik API, and computing _what_ dynamic configuration is needed for each step.
+This base class provides implementations of `start`, `stop`, `add_route`, `delete_route`, and `get_all_routes`.
+
+TraefikProxy subclasses must implement _how_ dynamic config is stored, and set the appropriate static config to tell traefik how to load the dynamic config.
+Specifically, the generic methods:
+
+- `_setup_traefik_static_config` should extend `self.static_config` to configure the traefik [configuration discovery provider](https://doc.traefik.io/traefik/providers/overview/)
+- `_apply_dynamic_config` stores a given dynamic config dictionary in the appropriate config store
+- `_delete_dynamic_config` removes keys from the dynamic config store
+- `_get_jupyterhub_dynamic_config` reads the whole jupyterhub part (not read by traefik itself)
+
+We have two classes at this level:
+
+- TraefikFileProviderProxy, which stores dynamic config in a toml or yaml file, and
+- TKvProxy - another base class, which implements the above, based on a generic notion of a key-value store
+
+TKvProxy is an adapter layer, implementing all of the above methods, based on a few basic actions on key-value stores:
+
+- `_kv_atomic_set` should take a _flat dictionary_ of key _paths_ and string values, and store them.
+- `_kv_atomic_delete` should delete a number of keys in a single transaction
+- `_kv_get_tree` should recursively read everything in the key-value store under a prefix (returning the _flattened_ dictionary)
+
+TKvProxy is responsible for translating between key-value-friendly "flat" dictionaries and the 'true' nested dictionary format of the configuration (i.e. the nested dictionary `{"a": {"b": 5}}` will be flattened to `{"a/b": "5"}`).
+
+Finally, we have our specific key-value store implementations: [](TraefikEtcdProxy) and [](TraefikConsulProxy).
+These classes only need to implement:
+
+1. configuration necessary to connect to the key-value provider
+2. `_setup_traefik_static_config` to tell traefik how to talk to the same key-value provider
+3. the above three `_kv_` methods for reading, writing, and deleting keys
+
+## Testing jupyterhub-traefik-proxy
+
+You can then run the all the test suite from the _traefik-proxy_ directory with:
+
+```
+$ pytest -v ./tests
+```
+
+Or you can run a specific test with:
+
+```
+$ pytest -v ./tests/<test-file-name>
+```

docs/source/index.md

@@ -44,6 +44,7 @@ consul
 
 api/index
 changelog
+details
 ```
 
 ## Indices and tables

docs/source/install.md

@@ -96,119 +96,9 @@ c.JupyterHub.proxy_class = "traefik_file"
 ```
 c.JupyterHub.proxy_class = "traefik_etcd"
 # will configure JupyterHub to run with TraefikEtcdProxy
-
 ```
 
 ```
 c.JupyterHub.proxy_class = "traefik_consul"
 # will configure JupyterHub to run with TraefikConsulProxy
-
-```
-
-## Implementation details
-
-1. **Traefik API**
-
-   traefik-proxy uses the [Traefik API](https://doc.traefik.io/traefik/operations/api/) to monitor routes and configurations.
-
-   Because of **security** concerns, in traefik-proxy implementation, traefik api endpoint isn't exposed on the public http endpoint. Instead, it runs on a dedicated **authenticated endpoint** that's on localhost by default.
-
-   The port on which traefik-proxy's api will run, as well as the username and password used for authenticating, can be passed to the proxy through `jupyterhub_config.py`, e.g.:
-
-   ```
-   c.TraefikFileProviderProxy.traefik_api_url = "http://127.0.0.1:8099"
-   c.TraefikFileProviderProxy.traefik_api_password = "admin"
-   c.TraefikFileProviderProxy.traefik_api_username = "admin"
-   ```
-
-   Check out TraefikProxy's [API Reference](TraefikProxy) for more configuration options.
-
-2. **TKvProxy class**
-
-   TKvProxy is a JupyterHub Proxy implementation that uses traefik and a key-value store.
-   **TraefikEtcdProxy** and **TraefikConsulProxy** are proxy implementations that sublass `TKvProxy`.
-   Other custom proxies that wish to implementat a JupyterHub Trafik KV store Proxy can sublass `TKvProxy`.
-   **TKvProxy** implements JupyterHub's Proxy public API and there is no need to override these public methods.
-   The methods that **must be implemented** by the proxies that sublass `TKvProxy` are:
-
-   - **_\_define_kv_specific_static_config()_**
-     - Define the traefik static configuration that configures
-       traefik's communication with the key-value store.
-     - Will be called during startup if should_start is True.
-     - Subclasses must define this method if the proxy is to be started by the Hub.
-     - In order to be picked up by the proxy, the static configuration
-       must be stored into `proxy.static_config` dict under the `kv_name` key.
-   - **_\_kv_atomic_add_route_parts(jupyterhub_routespec, target, data, route_keys, rule)_**
-     - Add the key-value pairs associated with a route within a key-value store transaction.
-     - Will be called during add_route.
-     - When retrieving or deleting a route, the parts of a route are expected to have the following structure:
-       ```
-       [ key: jupyterhub_routespec            , value: target ]
-       [ key: target                          , value: data   ]
-       [ key: route_keys.backend_url_path     , value: target ]
-       [ key: route_keys.frontend_rule_path   , value: rule   ]
-       [ key: route_keys.frontend_backend_path, value: route_keys.backend_alias]
-       [ key: route_keys.backend_weight_path  , value: w(int) ]
-       # where w is the weight of the backend to be used during load balancing)
-       ```
-     - Returns:
-       - result (tuple):
-         - The transaction status (int, 0: failure, positive: success)
-         - The transaction response(str)
-   - **_\_kv_atomic_delete_route_parts(jupyterhub_routespec, route_keys)_**
-     - Delete the key-value pairs associated with a route, within a key-value store transaction (if the route exists).
-     - Will be called during delete_route.
-     - The keys associated with a route are:
-       - jupyterhub_routespec
-       - target
-       - route_keys.backend_url_path
-       - route_keys.frontend_rule_path
-       - route_keys.frontend_backend_path
-       - route_keys.backend_weight_path
-     - Returns:
-       - result (tuple):
-         - The transaction status (int, 0: failure, positive: success)
-         - The transaction response (str)
-   - **_\_kv_get_target(jupyterhub_routespec)_**
-     - Retrive the target from the key-value store.
-     - The target is the value associated with `jupyterhub_routespec` key.
-     - Returns:
-       - The full URL associated with this route (str)
-   - **_\_kv_get_data(target)_**
-     - Retrive the data associated with the `target` from the key-value store.
-     - Returns:
-       - A JSONable dict that holds extra info about the route (dict)
-   - **_\_kv_get_route_parts(kv_entry)_**
-     - Retrive all the parts that make up a route (i.e. routespec, target, data) from the key-value store given a `kv_entry`.
-     - A `kv_entry` is a key-value store entry where the key starts with `proxy.jupyterhub_prefix`. It is expected that only the routespecs
-       will be prefixed with `proxy.jupyterhub_prefix` when added to the kv store.
-     - Returns:
-       - routespec: The normalized route specification passed in to add_route ([host]/path/)
-       - target: The target host for this route (proto://host)
-       - data: The arbitrary data dict that was passed in by JupyterHub when adding this route.
-   - **_\_kv_get_jupyterhub_prefixed_entries()_**
-     - Retrive from the kv store all the key-value pairs where the key starts with `proxy.jupyterhub_prefix`.
-     - It is expected that only the routespecs will be prefixed with `proxy.jupyterhub_prefix` when added to the kv store.
-     - Returns:
-       - routes: A list of key-value store entries where the keys start with `proxy.jupyterhub_prefix`.
-
-## Testing jupyterhub-traefik-proxy
-
-There are some tests that use _etcdctl_ command line client for etcd.
-Make sure to set environment variable ETCDCTL_API=3 before running the tests, so that the v3 API to be used, e.g.:
-
-```
-$ export ETCDCTL_API=3
-```
-
-You can then run the all the test suite from the _traefik-proxy_ directory with:
-
-```
-$ pytest -v ./tests
-```
-
-Or you can run a specific test with:
-
-```
-$ pytest -v ./tests/<test-file-name>
 ```