Running in production

Kinto is a standard Python application.

Recommended settings for production are listed below. Some :ref:`general insights about deployment strategies <deployment>` are also provided.

Because we use it for most of our deploys, PostgreSQL is the recommended backend for production.

Install and setup PostgreSQL

(requires PostgreSQL 9.5 or higher).

Kinto dependencies do not include PostgreSQL tooling and drivers by default, which should be installed and configured before proceeding to the next steps. More information is available at the PostgreSQL Documentation.

PostgreSQL client

Before installing the Python libraries for PostgreSQL we need to first install the header files for the PostgreSQL database backend.

On Debian / Ubuntu based systems:

$ sudo apt-get install libpq-dev

On RedHat / Fedora based systems:

$ dnf install postgresql-devel

On MacOS:

$ brew install postgresql

Install PostgreSQL Python Dependencies

Kinto PostgreSQL backends rely on specific Python packages (like SQLAlchemy and psycopg2), which can be installed with:

$ uv pip install kinto[postgresql]

Run a PostgreSQL server

The instructions to run a local PostgreSQL database are out of scope here.

A detailed guide is available on the Kinto Wiki.

Database setup

Kinto default database name is set to postgres on localhost:5432. To change it refer to Creating a configuration file.

To create a specific database:

CREATE DATABASE dbname WITH ENCODING UTF8;

By default Kinto uses UTC timezone, so make sure that the assigned database for kinto is set to UTC with:

ALTER DATABASE dbname SET TIMEZONE TO UTC;

Privileges basics

Kinto default user is set to postgres with password postgres. To change that refer to Creating a configuration file.

In order to initialize the database tables and objects, the specified user must have some privileges. For example, to create a user from scratch:

CREATE USER dbuser WITH PASSWORD 'dbpassword';
GRANT ALL PRIVILEGES ON DATABASE dbname TO dbuser;

For a read-only setup, it is possible to define a user that only has the privilege to read the tables:

CREATE USER dbuser WITH PASSWORD 'dbpassword';
GRANT USAGE ON SCHEMA public TO dbuser;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO dbuser;

In this case, you should use the kinto.readonly setting to tell Kinto that the database is read-only.

Even if the stack is read-only, some internal values like authentication tokens may still be to be stored in cache. If the cache backend is configured to use PostgreSQL, then write operations still must be granted on the cache table:

GRANT UPDATE, INSERT, DELETE ON cache TO dbuser;

Also, in future versions of Kinto, some new tables may be created. It is possible to change the default privileges to allow reading the future tables:

ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO dbuser;

Creating a configuration file

Once a PostgreSQL is up and running somewhere, you should edit your configuration file or create a new one with the init command and select the PostgreSQL option. The file is created by default as config/kinto.ini, to specify another filename use the --ini parameter.

$ kinto init --ini production.ini

Select your database server address, name and user by editing the configuration file. Also make sure that the PostgreSQL :ref:`backend settings<configuration-backends>` are selected. For our example, the backend configuration would be:

kinto.storage_backend = kinto.core.storage.postgresql
kinto.storage_url = postgresql://dbuser:dbpassword@localhost/dbname
kinto.cache_backend = kinto.core.cache.postgresql
kinto.cache_url = postgresql://dbuser:dbpassword@localhost/dbname
kinto.permission_backend = kinto.core.permission.postgresql
kinto.permission_url = postgresql://dbuser:dbpassword@localhost/dbname

Creating tables and indices

The last step consists in creating the necessary tables and indices, run the migrate command:

$ kinto migrate --ini production.ini

Important

You should run migrate every time you change the configuration file or kinto is upgraded.

Note

Alternatively the SQL initialization files can be found in the Kinto source.

Production checklist

Recommended settings

Most default setting values in the application code base are suitable for production.

Also, the set of settings mentioned below might deserve some review or adjustments:

kinto.http_scheme = https
kinto.paginate_by = 100
kinto.batch_max_requests = 25
kinto.storage_pool_size = 50
kinto.cache_pool_size = 50
kinto.permission_pool_size = 50
fxa-oauth.cache_ttl_seconds = 3600

Note

For an exhaustive list of available settings and their default values, refer to the Kinto source code.

By default, nobody can read buckets list. You can change that using:

kinto.bucket_read_principals = system.Authenticated

Beware that if you do so, everyone will be able to list bucket information (including user's personal buckets).

Handling CDN

If you want to put your Kinto behind a CDN you must make sure to define the right host or you will leak the main server host.

kinto.http_host = cdn.firefox.com

You can make sure your service is correctly configured by looking at the service URL returned on the service home page. It should be your CDN service URL.

It might also be relevant to set your main server :ref:`as readonly <configuration-features>`.

In the configuration of the CDN service, you should also:

Allow OPTIONS requests (CORS)
Pass through cache and concurrency control headers: ETag, Last-Modified, Expire
Pass through pagination header: Next-Page
Cached responses should depend on querystring parameters (e.g. try with different ?_limit= values)

Monitoring

In order to enable monitoring features (eg. Prometheus or StatsD), install extra requirements:

make install-monitoring

See :ref:`settings section <monitoring-with-prometheus>` for the configuration aspects.

Counters

Name	Description
`users`	Number of unique user IDs.
`authn_type.basicauth`	Number of basic authentication requests
`authn_type.fxa`	Number of FxA authentications

Timers

Name	Description
`authentication.permits`	Time needed by the permissions backend to allow or reject a request
`view.hello.GET`	Time needed to return the hello view
`view.heartbeat.GET`	Time needed to return the heartbeat page
`view.batch.POST`	Time needed to process a batch request
`view.{resource}-{type}.{method}`	Time needed to process the specified {method} on a {resource} (e.g. bucket, collection or record). Different timers exists for the different type of resources (record or collection)
`cache.{method}`	Time needed to execute a method of the cache backend. Methods are `ping`, `ttl`, `expire`, `set`, `get` and `delete`
`storage.{method}`	Time needed to execute a method of the storage backend. Methods are `ping`, `collection_timestamp`, `create`, `get`, `update`, `delete`, `delete_all`, `get_all`
`permission.{method}`	Time needed to execute a method of the permission backend. Methods are `add_user_principal`, `remove_user_principal`, `get_user_principals`, `add_principal_to_ace`, `remove_principal_from_ace`, `get_object_permission_principals`, `check_permission`

JSON Logging

At Mozilla, applications' log files follow a specific JSON schema, that is processed through Kibana or Heka.

With the following configuration, all logs are structured in JSON and redirected to standard output (See 12factor app).

Note

You must install the mozilla-cloud-services-logger package.

[loggers]
keys = root

[handlers]
keys = console

[formatters]
keys = generic, json

[logger_root]
level = INFO
handlers = console

[handler_console]
class = StreamHandler
args = (sys.stdout,)
level = NOTSET
formatter = json

[formatter_json]
class = kinto.core.JsonLogFormatter

[formatter_generic]
format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

Run the Kinto application

Using Apache mod wsgi

This is probably the easiest way to setup a production server.

With the following configuration for the site, Apache should be able to run the Kinto application:

WSGIScriptAlias /         /path/to/kinto/app.wsgi
WSGIPythonPath            /path/to/kinto
SetEnv          KINTO_INI /path/to/kinto.ini

<Directory /path/to/kinto>
  <Files app.wsgi>
    Require all granted
  </Files>
</Directory>

Using nginx

nginx can act as a reverse proxy in front of Granian (or any other wsgi server like Gunicorn or uWSGI).

Configure nginx to listen to Granian (or other wsgi server) and to forward requests to it. For example, if Granian is listening on port 8000, the nginx configuration file should look like so:

upstream kinto {
    server 127.0.0.1:8000;
}

server {
    listen      8000;
    server_name .my-kinto.org; # substitute your machine's IP address or FQDN
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;   # adjust to taste

    location / {
        proxy_pass http://kinto;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

It is also wise to restrict the private URLs (like for __heartbeat__):

location ~ /v1/__(.+)__ {
    allow 127.0.0.1;
    allow 172.31.17.16;
    deny all;
}

Running with Granian

uv pip install granian

To run the application using Granian, an app.py file must be picked up. It is available in the Kinto Python package or can be downloaded from GitHub:

wget https://raw.githubusercontent.com/Kinto/kinto/main/app.py

Then, run the application with:

KINTO_INI=/path/to/production.ini granian --interface wsgi --port 8000 app:application

Nginx as cache server

If Nginx is used as a reverse proxy, it can also act as a cache server by taking advantage of Kinto optional cache control response headers (forced :ref:`in settings <configuration-client-caching>` or set :ref:`on collections <collection-caching>`).

The sample Nginx configuration file shown above will look like so:

proxy_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:100m inactive=200m;
proxy_cache_key "$scheme$request_method$host$request_uri$";

server {
    ...

    location / {
        proxy_cache my_zone;

        proxy_pass http://kinto;
    }
}

Upgrading Kinto

Important

We follow semver for version numbers.

Before upgrading, read the release notes about potential breaking changes.

See also :ref:`API versioning <api-versioning>`.

First, make the potential changes to the configuration file, as described in the release notes.

If installed as Python package, upgrade Kinto using:

uv pip install --upgrade kinto

Since there might be some database schema changes, do not forget to run the migration with:

kinto migrate

Once done, restart the server.

Using backoff

The :ref:`backoff feature <backoff-indicators>` of the HTTP API allows to reduce the hits of clients during a period of time.

In order to leverage this, change the kinto.backoff setting to a number of seconds (e.g. 3600) and reload/restart the server some time before starting the upgrade process.

Do not forget to revert it once the upgrade is done ;)

FilesExpand file tree

production.rst

Latest commit

History

production.rst

File metadata and controls

Running in production

Install and setup PostgreSQL

PostgreSQL client

Install PostgreSQL Python Dependencies

Run a PostgreSQL server

Database setup

Privileges basics

Creating a configuration file

Creating tables and indices

Production checklist

Recommended settings

Handling CDN

Monitoring

Counters

Timers

JSON Logging

Run the Kinto application

Using Apache mod wsgi

Using nginx

Running with Granian

Nginx as cache server

Upgrading Kinto

Using backoff