Merge pull request #8 from MatchmakerExchange/exchange_server

Authentication support and refactoring

Author: buske

.travis.yml

@@ -1,25 +1,44 @@
 # After changing this file, check it on:
 #   http://lint.travis-ci.org/
 language: python
+
 python:
   - "2.7"
   - "3.3"
   - "3.4"
   - "3.5"
+
+sudo: false
+
+cache:
+  apt: true
+
+addons:
+  apt:
+    sources:
+      - elasticsearch-2.x
+    packages:
+      - elasticsearch
+
+services:
+  - elasticsearch
+
 install:
   - pip install -e .
   - mme-server quickstart
+
 script:
   # Test via module runner
   - python -m mme_server test
   # Test via setup.py (with environment variable to also test quickstart)
   - MME_TEST_QUICKSTART=1 coverage run --source=mme_server setup.py test
-services:
-  - elasticsearch
+
 before_script:
   # Delay for elasticsearch to start
   - sleep 10
+
 before_install:
   - pip install coveralls
+
 after_success:
-  - coveralls
+  - coveralls

MANIFEST.in

@@ -2,4 +2,5 @@
 include LICENSE.txt
 include README.md
 include MANIFEST.in
-recursive-include mme_server/schemas *.json
+recursive-include mme_server/schemas *.json
+recursive-include mme_server/templates *.html

README.md

@@ -11,8 +11,9 @@ This code is intended to be illustrative and is **not** guaranteed to perform we
 
 
 ## Dependencies
+
 - Python 2.7 or 3.3+
-- ElasticSearch
+- ElasticSearch 2.x
 
 
 ## Quickstart
@@ -42,12 +43,20 @@ This code is intended to be illustrative and is **not** guaranteed to perform we
     mme-server quickstart
     ```
 
-1. Run tests:
+1. Run tests (must run quickstart first):
 
     ```sh
     mme-server test
     ```
 
+1. Authorize an incoming server:
+
+    ```sh
+    mme-server clients add myclient --label "My Client" --key "<CLIENT_AUTH_TOKEN>"
+    ```
+
+    Leave off the `--key` option to have a secure key randomly generated for you.
+
 1. Start up MME reference server:
 
     ```sh
@@ -59,14 +68,17 @@ This code is intended to be illustrative and is **not** guaranteed to perform we
 1. Try it out:
 
     ```sh
-    curl -XPOST -H 'Content-Type: application/vnd.ga4gh.matchmaker.v1.0+json' \
-         -H 'Accept: application/vnd.ga4gh.matchmaker.v1.0+json' \
-         -d '{"patient":{
+    curl -XPOST \
+      -H 'X-Auth-Token: <CLIENT_AUTH_TOKEN>' \
+      -H 'Content-Type: application/vnd.ga4gh.matchmaker.v1.0+json' \
+      -H 'Accept: application/vnd.ga4gh.matchmaker.v1.0+json' \
+      -d '{"patient":{
         "id":"1",
         "contact": {"name":"Jane Doe", "href":"mailto:[email protected]"},
         "features":[{"id":"HP:0000522"}],
-        "genomicFeatures":[{"gene":{"id":"NGLY1"}}]
-      }}' localhost:8000/match
+        "genomicFeatures":[{"gene":{"id":"NGLY1"}}],
+        "test": true
+      }}' localhost:8000/v1/match
     ```
 
 ## Installation
@@ -93,14 +105,14 @@ source .virtualenv/bin/activate
 First, download elasticsearch:
 
 ```sh
-wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz
-tar -xzf elasticsearch-2.1.1.tar.gz
+wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.6.zip
+unzip elasticsearch-1.7.6.zip
 ```
 
 Then, start up a local elasticsearch cluster to serve as our database (`-Des.path.data=data` puts the elasticsearch indices in a subdirectory called `data`):
 
 ```sh
-./elasticsearch-2.1.1/bin/elasticsearch -Des.path.data=data
+./elasticsearch-1.7.6/bin/elasticsearch -Des.path.data=data
 ```
 
 
@@ -117,18 +129,21 @@ Custom patient data can be indexed by the server in two ways (if a patient 'id'
 1. Batch index from the Python interface:
 
     ```py
-    >>> from mme_server.models import DatastoreConnection
-    >>> db = DatastoreConnection()
-    >>> db.patients.index('/path/to/patients.json')
+    >>> from mme_server.backend import get_backend
+    >>> db = get_backend()
+    >>> patients = db.get_manager('patients')
+    >>> patients.index('/path/to/patients.json')
     ```
 
 1. Single patient index the Python interface:
 
     ```py
-    >>> from mme_server.models import Patient, DatastoreConnection
-    >>> db = DatastoreConnection()
+    >>> from mme_server.backend import get_backend
+    >>> db = get_backend()
+    >>> patients = db.get_manager('patients')
+    >>> from mme_server.models import Patient
     >>> patient = Patient.from_api({...})
-    >>> db.patients.index_patient(patient)
+    >>> patients.index_patient(patient)
     ```
 
 
@@ -142,8 +157,3 @@ If you have any questions, feel free to post an issue on GitHub.
 This repository is managed by the Matchmaker Exchange technical team. You can reach us via GitHub or by [email](mailto:[email protected]).
 
 Contributions are most welcome! Post an issue, submit a bugfix, or just try it out. We hope you find it useful.
-
-
-## Implementations
-
-We don't know of any organizations using this code in a production setting just yet. If you are, please let us know! We'd love to list you here.

mme_server/__init__.py

@@ -1 +1,2 @@
 from .cli import main
+from .server import app

mme_server/auth.py

@@ -0,0 +1,40 @@
+"""
+Module hanlding the authentication of incoming and outgoing server requests.
+
+Stores:
+* Authenticated servers (`servers` index)
+"""
+from __future__ import with_statement, division, unicode_literals
+
+import logging
+import flask
+
+from functools import wraps
+
+from flask import request, jsonify
+
+from .backend import get_backend
+
+
+logger = logging.getLogger(__name__)
+
+
+def auth_token_required():
+    def decorator(f):
+        @wraps(f)
+        def decorated_function(*args, **kwargs):
+            logger.info("Authenticating request")
+            token = request.headers.get('X-Auth-Token')
+            backend = get_backend()
+            servers = backend.get_manager('servers')
+            server = servers.verify(token)
+            if not server:
+                error = jsonify(message='X-Auth-Token not authorized')
+                error.status_code = 401
+                return error
+
+            # Set authenticated server as flask global for request
+            flask.g.server = server
+            return f(*args, **kwargs)
+        return decorated_function
+    return decorator

mme_server/backend.py

@@ -0,0 +1,23 @@
+"""
+Module for accessing the backend connection
+"""
+
+from __future__ import with_statement, division, unicode_literals
+
+import logging
+import flask
+
+from elasticsearch import Elasticsearch
+
+from .managers import Managers
+
+logger = logging.getLogger(__name__)
+
+
+
+def get_backend():
+    backend = getattr(flask.g, '_mme_backend', None)
+    if backend is None:
+        backend = flask.g._mme_backend = Managers(Elasticsearch())
+
+    return backend

mme_server/cli.py

@@ -8,8 +8,10 @@
 import logging
 import unittest
 
+from binascii import hexlify
+
+from .backend import get_backend
 from .compat import urlretrieve
-from .models import get_backend
 from .server import app
 
 
@@ -56,10 +58,12 @@ def index_file(index, filename, url):
 
     with app.app_context():
         backend = get_backend()
+        patients = backend.get_manager('patients')
+        vocabularies = backend.get_manager('vocabularies')
         index_funcs = {
-            'hpo': backend.vocabularies.index_hpo,
-            'genes': backend.vocabularies.index_genes,
-            'patients': backend.patients.index,
+            'hpo': vocabularies.index_hpo,
+            'genes': vocabularies.index_genes,
+            'patients': patients.index_file,
         }
         index_funcs[index](filename=filename)
 
@@ -73,11 +77,87 @@ def fetch_resource(filename, url):
         logger.info('Saved file to: {}'.format(filename))
 
 
+def list_servers(direction='out'):
+    with app.app_context():
+        backend = get_backend()
+        servers = backend.get_manager('servers')
+        response = servers.list(direction=direction)
+        # print header
+        fields = response['fields']
+        print('\t'.join(fields))
+
+        for server in response.get('rows', []):
+            print('\t'.join([repr(server[field]) for field in fields]))
+
+def list_clients():
+    return list_servers(direction='in')
+
+def add_server(id, direction='out', key=None, label=None, base_url=None):
+    if not label:
+        label = id
+
+    if direction == 'out' and not base_url:
+        raise Exception('base-url must be specified for outgoing servers')
+
+    with app.app_context():
+        backend = get_backend()
+        servers = backend.get_manager('servers')
+        # Generate a random key if one was not provided
+        if key is None:
+            key = hexlify(os.urandom(30)).decode()
+        servers.add(server_id=id, server_key=key, direction=direction, server_label=label, base_url=base_url)
+
+def add_client(id, key=None, label=None):
+    add_server(id, 'in', key=key, label=label)
+
+def remove_server(id, direction='out'):
+    with app.app_context():
+        backend = get_backend()
+        servers = backend.get_manager('servers')
+        servers.remove(server_id=id, direction=direction)
+
+def remove_client(id):
+    remove_server(id, direction='in')
+
+
 def run_tests():
     suite = unittest.TestLoader().discover('.'.join([__package__, 'tests']))
     unittest.TextTestRunner().run(suite)
 
 
+def add_server_subcommands(parser, direction):
+    """Add subparser for incoming or outgoing servers
+
+    direction - 'in': incoming servers, 'out': outgoing servers
+    """
+    server_type = 'client' if direction == 'in' else 'server'
+    subparsers = parser.add_subparsers(title='subcommands')
+    subparser = subparsers.add_parser('add', description="Add {} authorization".format(server_type))
+    subparser.add_argument("id", help="A unique {} identifier".format(server_type))
+    if server_type == 'server':
+        subparser.add_argument("base_url", help="The base HTTPS URL for sending API requests to the server (e.g., <base-url>/match should be a valid endpoint).")
+
+    subparser.add_argument("--key", help="The secret key used to authenticate requests to/from the {} (default: randomly generate a secure key)".format(server_type))
+    subparser.add_argument("--label", help="The display name for the {}".format(server_type))
+    if server_type == 'server':
+        subparser.set_defaults(function=add_server)
+    else:
+        subparser.set_defaults(function=add_client)
+
+    subparser = subparsers.add_parser('rm', description="Remove {} authorization".format(server_type))
+    subparser.add_argument("id", help="The {} identifier".format(server_type))
+    if server_type == 'server':
+        subparser.set_defaults(function=remove_server)
+    else:
+        subparser.set_defaults(function=remove_client)
+
+    subparser = subparsers.add_parser('list', description="List {} authorizations".format(server_type))
+    if server_type == 'server':
+        subparser.set_defaults(function=list_servers)
+    else:
+        subparser.set_defaults(function=list_clients)
+
+
 def parse_args(args):
     from argparse import ArgumentParser
 
@@ -116,6 +196,12 @@ def parse_args(args):
                            help="The host the server will listen to (0.0.0.0 to listen globally; 127.0.0.1 to listen locally; default: %(default)s)")
     subparser.set_defaults(function=app.run)
 
+    subparser = subparsers.add_parser('servers', description="Server authorization sub-commands")
+    add_server_subcommands(subparser, direction='out')
+
+    subparser = subparsers.add_parser('clients', description="Client authorization sub-commands")
+    add_server_subcommands(subparser, direction='in')
+
     subparser = subparsers.add_parser('test', description="Run tests")
     subparser.set_defaults(function=run_tests)
 

mme_server/compat.py

@@ -6,3 +6,13 @@
     from urllib import urlretrieve
 except ImportError:
     from urllib.request import urlretrieve
+
+try:
+    from urllib2 import urlopen, Request
+except ImportError:
+    from urllib.request import urlopen, Request
+
+try:
+    from urlparse import urlsplit
+except ImportError:
+    from urllib.parse import urlsplit