Allow configuring from environment, add docker entrypoint (#229)

* Allow setting most config variables through environment

* Add docker entrypoint

* Update config docs

* Update deployment docs

* Change unit test after setting TLS to true by default

* Get media base dir from env

* Bump Grampsjs version in docs

Author: DavidMStraub

Dockerfile

@@ -20,6 +20,10 @@ RUN mkdir /app/static && touch /app/static/index.html
 RUN mkdir /app/db && mkdir /app/media && mkdir /app/indexdir && mkdir /app/users
 RUN mkdir /app/thumbnail_cache
 RUN mkdir /app/tmp
+ENV USER_DB_URI=sqlite:////app/users/users.sqlite
+ENV MEDIA_BASE_DIR=/app/media
+ENV SEARCH_INDEX_DIR=/app/indexdir
+ENV STATIC_PATH=/app/static
 
 # install gunicorn
 RUN python3 -m pip install --no-cache-dir --extra-index-url https://www.piwheels.org/simple \
@@ -32,4 +36,6 @@ RUN python3 -m pip install --no-cache-dir --extra-index-url https://www.piwheels
 
 EXPOSE 5000
 
+COPY docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]
 CMD gunicorn -w 8 -b 0.0.0.0:5000 gramps_webapi.wsgi:app --timeout 120 --limit-request-line 8190

docker-entrypoint.sh

@@ -0,0 +1,23 @@
+#!/bin/sh
+set -eu
+
+# Use Gramps.js frontend
+if [ -n "$GRAMPSJS_VERSION" ]
+then
+    # Download if necessary
+    if ! [ -d "/app/static/grampsjs-${GRAMPSJS_VERSION}" ]
+    then
+        wget "https://github.com/DavidMStraub/Gramps.js/releases/download/${GRAMPSJS_VERSION}/grampsjs-${GRAMPSJS_VERSION}.tar.gz"
+        tar -xzf "grampsjs-${GRAMPSJS_VERSION}.tar.gz"
+        mv "grampsjs-${GRAMPSJS_VERSION}" static/
+    fi
+    export STATIC_PATH="/app/static/grampsjs-${GRAMPSJS_VERSION}"
+fi
+
+# Recreate search index if not exists
+if [ -z "$(ls -A /app/indexdir)" ]
+then
+    python3 -m gramps_webapi --config /app/config/config.cfg search index-full
+fi
+
+exec "$@"

docs/Configuration.md

@@ -1,16 +1,16 @@
 # Configuration
 
-A configuration file is necessary to run the Gramps Web API. Its path can either be provided on the command line when running the module as a script (`python3 -m gramps_webapi --config path/to/config.cfg ...`), or as an environment variable, `GRAMPS_API_CONFIG=path/to/config.cfg`.
+Some configuration is necessary to run the Gramps Web API. When using a configuration file, its path can either be provided on the command line when running the module as a script (`python3 -m gramps_webapi --config path/to/config.cfg ...`), or as an environment variable, `GRAMPS_API_CONFIG=path/to/config.cfg`. Some of the configuration options can also be set without a configuration file, using environment variables (e.g. `TREE='My Tree' python3 -m gramps_webapi ...` ).
 
-The following settings can be specified in the configuration file.
+The following configuration options exist. The last column indicates whether the option can be set from an environment variable.
 
 
 ## Required settings
 
-Key | Description
-----|------------
-`TREE` | The name of the family tree database to use. Show available trees with `gramps -l`
-`SECRET_KEY` | The secret key for flask. This must be set for use in production. The secret must not be shared publicly. Changing it will invalidate all access tokens.`
+Key | Description | Set from environment
+----|-------------|---------------------
+`TREE` | The name of the family tree database to use. Show available trees with `gramps -l` | yes
+`SECRET_KEY` | The secret key for flask. This must be set for use in production. The secret must not be shared publicly. Changing it will invalidate all access tokens | yes
 
 !!! info
     You can generate a secure secret key e.g. with the command
@@ -21,27 +21,27 @@ Key | Description
 
 ## Optional settings
 
-Key | Description
-----|------------
-`MEDIA_BASE_DIR` | Path to use as base directory for media files, overriding the media base directory set in Gramps
-`SEARCH_INDEX_DIR` | Path for the full-text search index. Defaults to `indexdir` relative to the path where the script is run
-`STATIC_PATH` | Path to serve static files from (e.g. a static web frontend)
-`BASE_URL` | Base URL where the API can be reached (e.g. `https://mygramps.mydomain.com/`). This is necessary e.g. to build correct passwort reset links
-`CORS_ORIGINS` | Origins where CORS requests are allowed from. By default, all are disallowed. Use `"*"` to allow requests from any domain.
-`EMAIL_HOST` | SMTP server host (e.g. for sending password reset e-mails)
-`EMAIL_PORT` | SMTP server port
-`EMAIL_HOST_USER` | SMTP server username
-`EMAIL_HOST_PASSWORD` | SMTP server password
-`EMAIL_USE_TLS` | Boolean, whether to use TLS for sending e-mails. Defaults to false
-`DEFAULT_FROM_EMAIL` | "From" address for automated e-mails
-`THUMBNAIL_CACHE_CONFIG` | Dictionary with settings for the thumbnail cache. See [Flask-Caching](https://flask-caching.readthedocs.io/en/latest/) for possible settings.
+Key | Description | Set from environment
+----|-------------|---------------------
+`MEDIA_BASE_DIR` | Path to use as base directory for media files, overriding the media base directory set in Gramps | yes
+`SEARCH_INDEX_DIR` | Path for the full-text search index. Defaults to `indexdir` relative to the path where the script is run | yes
+`STATIC_PATH` | Path to serve static files from (e.g. a static web frontend) | yes
+`BASE_URL` | Base URL where the API can be reached (e.g. `https://mygramps.mydomain.com/`). This is necessary e.g. to build correct passwort reset links | yes
+`CORS_ORIGINS` | Origins where CORS requests are allowed from. By default, all are disallowed. Use `"*"` to allow requests from any domain. | no
+`EMAIL_HOST` | SMTP server host (e.g. for sending password reset e-mails) | yes
+`EMAIL_PORT` | SMTP server port. defaults to 465 | yes
+`EMAIL_HOST_USER` | SMTP server username | yes
+`EMAIL_HOST_PASSWORD` | SMTP server password | yes
+`EMAIL_USE_TLS` | Boolean, whether to use TLS for sending e-mails. Defaults to true  | no
+`DEFAULT_FROM_EMAIL` | "From" address for automated e-mails | yes
+`THUMBNAIL_CACHE_CONFIG` | Dictionary with settings for the thumbnail cache. See [Flask-Caching](https://flask-caching.readthedocs.io/en/latest/) for possible settings. | no
 
 
 ## Settings only during development
 
-Key | Description
-----|------------
-`DISABLE_AUTH` | If `True`, disable the authentication system. **Warning: never** use this in a production environment, as it will allow read and write access from the public!
+Key | Description | Set from environment
+----|-------------|---------------------
+`DISABLE_AUTH` | If `True`, disable the authentication system. **Warning: never** use this in a production environment, as it will allow read and write access from the public! | no
 
 ## Example configuration file
 

docs/Deployment.md

@@ -23,24 +23,7 @@ Upload your media files to the server. Make sure your Gramps database uses relat
     Use the "Convert media paths from absolute to relative" option in the "Media Manager" tool in Gramps' "Tools" menu if necessary.
 
 
-
-## Step 3: Create a configuration file
-
-Create a configuration and upload it to your server (we will assume the path is `~/config.cfg`). It should contain at least the following lines:
-
-```python
-TREE="..."  # set the name of your family tree
-SECRET_KEY="..." # set your secret key
-# do not change the following lines as they refer to paths within the container
-USER_DB_URI="sqlite:////app/users/users.sqlite"
-MEDIA_BASE_DIR="/app/media"
-SEARCH_INDEX_DIR="/app/indexdir"
-STATIC_PATH="/app/static"
-```
-
-For other configuration options, see [Configuration](Configuration.md)
-
-## Step 4: Docker configuration
+## Step 3: Docker configuration
 
 Create a new file on the server named `docker-compose.yml` and insert the following contents:
 
@@ -53,11 +36,14 @@ services:
     restart: always
     ports:
       - "80:5000"
+    environment:
+      TREE: "..." # set the name of your family tree
+      SECRET_KEY: "..." # set your secret key
+      BASE_URL: "http://localhost:5554"
     volumes:
       - gramps_users:/app/users
       - gramps_index:/app/indexdir
       - gramps_thumb_cache:/app/thumbnail_cache
-      - ~/config.cfg:/app/config/config.cfg
       - ~/gramps_db:/root/.gramps/grampsdb
       - ~/gramps_media:/app/media
 
@@ -67,14 +53,14 @@ volumes:
   gramps_thumb_cache:
 ```
 
-This will generate three named volumes to make sure that the user database, search index, and thumbnail cache will persist, and will mount the configuration file, Gramps database directory, and the directory holding your media files into the container (the paths on the left-hand side of the colon `:` refer to your host machine, the ones on the right hand side to the container and should not be changed).
+This will generate three named volumes to make sure that the user database, search index, and thumbnail cache will persist, and will mount the Gramps database directory and the directory holding your media files into the container (the paths on the left-hand side of the colon `:` refer to your host machine, the ones on the right hand side to the container and should not be changed). For more configuration options, see [Configuration](Configuration.md).
 
 !!! warning
     The above will make the API available on port 80 of the host machine **without SSL/TLS protection**. You can use this for local testing, but do not expose this directly to the internet, it is completely insecure!
 
 
 
-## Step 5: Secure access with SSL/TLS
+## Step 4: Secure access with SSL/TLS
 
 The web API **must** be served to the public internet over HTTPS. There are several options, e.g.
 
@@ -156,7 +142,7 @@ networks:
 
 Please see the [acme-companion](https://github.com/nginx-proxy/acme-companion) docs for how to set up your domain.
 
-## Step 6: Import database
+## Step 5: Import database
 
 If you have copied the Gramps database directory directly in Step 1, you can skip this step.
 
@@ -172,7 +158,7 @@ docker-compose run gramps_webapi \
 where `My family tree` is the name of the new family tree (must match the setting in the config file) and `my_tree.gramps` the file name of the Gramps XML export.
 
 
-## Step 7: Add users
+## Step 6: Add users
 
 Initialize the user system by creating an administrator account with the command
 
@@ -188,19 +174,20 @@ docker-compose run gramps_webapi \
 See [User system](Users.md) for more details and how to add additional users.
 
 
-## Step 8: Build the search index
+## Step 7: Start the server
 
-Build the initial search index by running
+Run
 
-```bash
-docker-compose run gramps_webapi \
-    python3 -m gramps_webapi search index-full
+```
+docker-compose up -d
 ```
 
-## Step 9: Start the server
+On first run, it will build the full-text search index of the API.
 
-Run
+## Optional step: use Gramps.js web frontend
 
+If you want to add the [Gramps.js](https://github.com/DavidMStraub/Gramps.js) web frontend to your installation, simply add
+```yaml
+      - GRAMPSJS_VERSION=v0.1.1
 ```
-docker-compose up -d
-```
+(or any other released version) to the `environment` block in the `gramps_webapi` service section of `docker-compose.yml`.

gramps_webapi/__main__.py

@@ -34,11 +34,12 @@
 
 
 @click.group("cli")
-@click.option("--config", help="Set the path to the config file", required=True)
+@click.option("--config", help="Set the path to the config file")
 @click.pass_context
 def cli(ctx, config):
     """Gramps web API command line interface."""
-    os.environ[ENV_CONFIG_FILE] = os.path.abspath(config)
+    if config:
+        os.environ[ENV_CONFIG_FILE] = os.path.abspath(config)
     ctx.obj = create_app()
 
 

gramps_webapi/app.py

@@ -51,7 +51,23 @@ def create_app(db_manager=None):
     app.config.from_object(DefaultConfig)
 
     # overwrite with user config file
-    app.config.from_envvar(ENV_CONFIG_FILE)
+    if os.getenv(ENV_CONFIG_FILE):
+        app.config.from_envvar(ENV_CONFIG_FILE)
+
+    # if tree is missing, try to get it from the env or fail
+    app.config["TREE"] = app.config.get("TREE") or os.getenv("TREE")
+    if not app.config.get("TREE"):
+        raise ValueError("TREE must be specified")
+
+    # if secret key is missing, try to get it from the env or fail
+    app.config["SECRET_KEY"] = app.config["SECRET_KEY"] or os.getenv("SECRET_KEY")
+    if not app.config.get("SECRET_KEY"):
+        raise ValueError("SECRET_KEY must be specified")
+
+    # try setting media basedir from environment
+    app.config["MEDIA_BASE_DIR"] = app.config.get("MEDIA_BASE_DIR") or os.getenv(
+        "MEDIA_BASE_DIR"
+    )
 
     # instantiate DB manager
     if db_manager is None:
@@ -69,6 +85,10 @@ def create_app(db_manager=None):
         JWTManager(app)
 
         # instantiate and store auth provider
+        # if DB URI is missing, try to get it from the env or fail
+        app.config["USER_DB_URI"] = app.config.get("USER_DB_URI") or os.getenv(
+            "USER_DB_URI"
+        )
         if not app.config.get("USER_DB_URI"):
             raise ValueError("USER_DB_URI must be specified")
         app.config["AUTH_PROVIDER"] = SQLAuth(db_uri=app.config["USER_DB_URI"])

gramps_webapi/config.py

@@ -20,22 +20,23 @@
 """Default configuration settings."""
 
 import datetime
+import os
 
 
 class DefaultConfig(object):
     """Default configuration object."""
 
     PROPAGATE_EXCEPTIONS = True
-    SEARCH_INDEX_DIR = "indexdir"
-    EMAIL_HOST = "localhost"
-    EMAIL_PORT = 0
-    EMAIL_HOST_USER = ""
-    EMAIL_HOST_PASSWORD = ""
-    EMAIL_USE_TLS = False
-    DEFAULT_FROM_EMAIL = ""
-    BASE_URL = "http://localhost/"
+    SEARCH_INDEX_DIR = os.getenv("SEARCH_INDEX_DIR", "indexdir")
+    EMAIL_HOST = os.getenv("EMAIL_HOST", "localhost")
+    EMAIL_PORT = int(os.getenv("EMAIL_PORT", "465"))
+    EMAIL_HOST_USER = os.getenv("EMAIL_HOST_USER", "")
+    EMAIL_HOST_PASSWORD = os.getenv("EMAIL_HOST_PASSWORD", "")
+    EMAIL_USE_TLS = True
+    DEFAULT_FROM_EMAIL = os.getenv("DEFAULT_FROM_EMAIL", "")
+    BASE_URL = os.getenv("BASE_URL", "http://localhost/")
     CORS_EXPOSE_HEADERS = ["X-Total-Count"]
-    STATIC_PATH = "static"
+    STATIC_PATH = os.getenv("STATIC_PATH", "static")
     THUMBNAIL_CACHE_CONFIG = {
         "CACHE_TYPE": "filesystem",
         "CACHE_DIR": "thumbnail_cache",

tests/test_endpoints/test_user.py

@@ -172,12 +172,12 @@ def test_reset_password_trigger_invalid_user(self):
         assert rv.status_code == 404
 
     def test_reset_password_trigger_status(self):
-        with patch("smtplib.SMTP") as mock_smtp:
+        with patch("smtplib.SMTP_SSL") as mock_smtp:
             rv = self.client.post(BASE_URL + "/users/user/password/reset/trigger/")
             assert rv.status_code == 201
 
     def test_reset_password(self):
-        with patch("smtplib.SMTP") as mock_smtp:
+        with patch("smtplib.SMTP_SSL") as mock_smtp:
             rv = self.client.post(BASE_URL + "/users/user/password/reset/trigger/")
             context = mock_smtp.return_value
             context.send_message.assert_called()
@@ -444,7 +444,7 @@ def test_add_user(self):
         assert rv.status_code == 200
 
     def test_register_user(self):
-        with patch("smtplib.SMTP") as mock_smtp:
+        with patch("smtplib.SMTP_SSL") as mock_smtp:
             # role is not allowed
             rv = self.client.post(
                 BASE_URL + "/users/new_user_2/register/",
@@ -509,7 +509,7 @@ def test_register_user(self):
             assert rv.status_code == 403
 
     def test_confirm_email(self):
-        with patch("smtplib.SMTP") as mock_smtp:
+        with patch("smtplib.SMTP_SSL") as mock_smtp:
             rv = self.client.post(
                 BASE_URL + "/users/new_user_3/register/",
                 json={