Merge pull request #116 from Lukasa/abstraction

HTTP/1.1 and HTTP/2 abstraction layer

Author: Lukasa

README.rst

@@ -13,11 +13,11 @@ improved speed, lower bandwidth usage, better connection management, and more.
 
 ``hyper`` provides these benefits to your Python code. How? Like this::
 
-    from hyper import HTTP20Connection
+    from hyper import HTTPConnection
 
-    conn = HTTP20Connection('http2bin.org:443')
+    conn = HTTPConnection('http2bin.org:443')
     conn.request('GET', '/get')
-    resp = conn.getresponse()
+    resp = conn.get_response()
 
     print(resp.read())
 

docs/source/advanced.rst

@@ -13,12 +13,11 @@ may want to keep your connections alive only as long as you know you'll need
 them. In HTTP/2 this is generally not something you should do unless you're
 very confident you won't need the connection again anytime soon. However, if
 you decide you want to avoid keeping the connection open, you can use the
-:class:`HTTP20Connection <hyper.HTTP20Connection>` and
-:class:`HTTP11Connection <hyper.HTTP11Connection>` as context managers::
+:class:`HTTPConnection <hyper.HTTPConnection>` as a context manager::
 
-    with HTTP20Connection('http2bin.org') as conn:
+    with HTTPConnection('http2bin.org') as conn:
         conn.request('GET', '/get')
-        data = conn.getresponse().read()
+        data = conn.get_response().read()
 
     analyse(data)
 
@@ -57,10 +56,9 @@ Very easy!
 Multithreading
 --------------
 
-Currently, ``hyper``'s :class:`HTTP20Connection <hyper.HTTP20Connection>` and
-:class:`HTTP11Connection <hyper.HTTP11Connection>` classes are **not**
-thread-safe. Thread-safety is planned for ``hyper``'s core objects, but in this
-early alpha it is not a high priority.
+Currently, ``hyper``'s :class:`HTTPConnection <hyper.HTTPConnection>` class
+is **not** thread-safe. Thread-safety is planned for ``hyper``'s core objects,
+but in this early alpha it is not a high priority.
 
 To use ``hyper`` in a multithreaded context the recommended thing to do is to
 place each connection in its own thread. Each thread should then have a request
@@ -155,8 +153,8 @@ headers for the original request, after, or in the middle of sending the
 response body.
 
 In order to receive pushed resources, the
-:class:`HTTP20Connection <hyper.HTTP20Connection>` object must be constructed
-with ``enable_push=True``.
+:class:`HTTPConnection <hyper.HTTPConnection>` object must be constructed with
+``enable_push=True``.
 
 You may retrieve the push promises that the server has sent *so far* by calling
 :meth:`get_pushes() <hyper.HTTP20Connection.get_pushes>`, which returns a

docs/source/index.rst

@@ -14,9 +14,9 @@ improved speed, lower bandwidth usage, better connection management, and more.
 
 ``hyper`` provides these benefits to your Python code. How? Like this::
 
-    from hyper import HTTP20Connection
+    from hyper import HTTPConnection
 
-    conn = HTTP20Connection('http2bin.org:443')
+    conn = HTTPConnection('http2bin.org:443')
     conn.request('GET', '/get')
     resp = conn.get_response()
 

docs/source/quickstart.rst

@@ -59,20 +59,20 @@ http2bin.org, a HTTP/1.1 and HTTP/2 testing service.
 
 Begin by getting the homepage::
 
-    >>> from hyper import HTTP20Connection
-    >>> c = HTTP20Connection('http2bin.org')
+    >>> from hyper import HTTPConnection
+    >>> c = HTTPConnection('http2bin.org')
     >>> c.request('GET', '/')
     1
     >>> resp = c.get_response()
 
 Used in this way, ``hyper`` behaves exactly like ``http.client``. You can make
 sequential requests using the exact same API you're accustomed to. The only
 difference is that
-:meth:`HTTP20Connection.request() <hyper.HTTP20Connection.request>` returns a
-value, unlike the equivalent ``http.client`` function. The return value is the
-HTTP/2 *stream identifier*. If you're planning to use ``hyper`` in this very
-simple way, you can choose to ignore it, but it's potentially useful. We'll
-come back to it.
+:meth:`HTTPConnection.request() <hyper.HTTPConnection.request>` may return a
+value, unlike the equivalent ``http.client`` function. If present, the return
+value is the HTTP/2 *stream identifier*. If you're planning to use ``hyper``
+in this very simple way, you can choose to ignore it, but it's potentially
+useful. We'll come back to it.
 
 Once you've got the data, things diverge a little bit::
 
@@ -101,8 +101,8 @@ the response from any of them, and switch between them using their stream IDs.
 
 For example::
 
-    >>> from hyper import HTTP20Connection
-    >>> c = HTTP20Connection('http2bin.org')
+    >>> from hyper import HTTPConnection
+    >>> c = HTTPConnection('http2bin.org')
     >>> first = c.request('GET', '/get')
     >>> second = c.request('POST', '/post', body='key=value')
     >>> third = c.request('GET', '/ip')
@@ -112,40 +112,18 @@ For example::
 
 ``hyper`` will ensure that each response is matched to the correct request.
 
-Making Your First HTTP/1.1 Request
------------------------------------
+Abstraction
+-----------
 
-With ``hyper`` installed, you can start making HTTP/2 requests. At this
-stage, ``hyper`` can only be used with services that *definitely* support
-HTTP/2. Before you begin, ensure that whichever service you're contacting
-definitely supports HTTP/2. For the rest of these examples, we'll use
-Twitter.
-
-You can also use ``hyper`` to make HTTP/1.1 requests. The code is very similar.
-For example, to get the Twitter homepage::
-
-    >>> from hyper import HTTP11Connection
-    >>> c = HTTP11Connection('twitter.com:443')
-    >>> c.request('GET', '/')
-    >>> resp = c.get_response()
-
-The key difference between HTTP/1.1 and HTTP/2 is that when you make HTTP/1.1
-requests you do not get a stream ID. This is, of course, because HTTP/1.1 does
-not have streams.
+When you use the :class:`HTTPConnection <hyper.HTTPConnection>` object, you
+don't have to know in advance whether your service supports HTTP/2 or not. If
+it doesn't, ``hyper`` will transparently fall back to HTTP/1.1.
 
-Things behave exactly like they do in the HTTP/2 case, right down to the data
-reading::
+You can tell the difference: if :meth:`request <hyper.HTTPConnection.request>`
+returns a stream ID, then the connection is using HTTP/2: if it returns
+``None``, then HTTP/1.1 is being used.
 
-    >>> resp.headers['content-encoding']
-    [b'deflate']
-    >>> resp.headers
-    HTTPHeaderMap([(b'x-xss-protection', b'1; mode=block')...
-    >>> resp.status
-    200
-    >>> body = resp.read()
-    b'<!DOCTYPE html>\n<!--[if IE 8]><html clas ....
-
-That's all it takes.
+Generally, though, you don't need to care.
 
 Requests Integration
 --------------------
@@ -155,8 +133,7 @@ requests doesn't support HTTP/2 though. To rectify that oversight, ``hyper``
 provides a transport adapter that can be plugged directly into Requests, giving
 it instant HTTP/2 support.
 
-All you have to do is identify a host that you'd like to communicate with over
-HTTP/2. Once you've worked that out, you can get started straight away::
+Using ``hyper`` with requests is super simple::
 
     >>> import requests
     >>> from hyper.contrib import HTTP20Adapter
@@ -169,14 +146,6 @@ HTTP/2. Once you've worked that out, you can get started straight away::
 This transport adapter is subject to all of the limitations that apply to
 ``hyper``, and provides all of the goodness of requests.
 
-A quick warning: some hosts will redirect to new hostnames, which may redirect
-you away from HTTP/2. Make sure you install the adapter for all the hostnames
-you're interested in::
-
-    >>> a = HTTP20Adapter()
-    >>> s.mount('https://http2bin.org', a)
-    >>> s.mount('https://www.http2bin.org', a)
-
 .. _requests: http://python-requests.org/
 
 HTTPie Integration

hyper/common/connection.py

@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+"""
+hyper/common/connection
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Hyper's HTTP/1.1 and HTTP/2 abstraction layer.
+"""
+from .exceptions import TLSUpgrade
+from ..http11.connection import HTTP11Connection
+from ..http20.connection import HTTP20Connection
+from ..tls import H2_NPN_PROTOCOLS
+
+
+class HTTPConnection(object):
+    """
+    An object representing a single HTTP connection to a server.
+
+    This object behaves similarly to the Python standard library's
+    ``HTTPConnection`` object, with a few critical differences.
+
+    Most of the standard library's arguments to the constructor are not
+    supported by hyper. Most optional parameters apply to *either* HTTP/1.1 or
+    HTTP/2.
+
+    :param host: The host to connect to. This may be an IP address or a
+        hostname, and optionally may include a port: for example,
+        ``'http2bin.org'``, ``'http2bin.org:443'`` or ``'127.0.0.1'``.
+    :param port: (optional) The port to connect to. If not provided and one also
+        isn't provided in the ``host`` parameter, defaults to 443.
+    :param secure: (optional, HTTP/1.1 only) Whether the request should use
+        TLS. Defaults to ``False`` for most requests, but to ``True`` for any
+        request issued to port 443.
+    :param window_manager: (optional) The class to use to manage flow control
+        windows. This needs to be a subclass of the
+        :class:`BaseFlowControlManager <hyper.http20.window.BaseFlowControlManager>`.
+        If not provided,
+        :class:`FlowControlManager <hyper.http20.window.FlowControlManager>`
+        will be used.
+    :param enable_push: (optional) Whether the server is allowed to push
+        resources to the client (see
+        :meth:`get_pushes() <hyper.HTTP20Connection.get_pushes>`).
+    """
+    def __init__(self,
+                 host,
+                 port=None,
+                 secure=None,
+                 window_manager=None,
+                 enable_push=False,
+                 **kwargs):
+
+        self._host = host
+        self._port = port
+        self._h1_kwargs = {'secure': secure}
+        self._h2_kwargs = {
+            'window_manager': window_manager, 'enable_push': enable_push
+        }
+
+        # Add any unexpected kwargs to both dictionaries.
+        self._h1_kwargs.update(kwargs)
+        self._h2_kwargs.update(kwargs)
+
+        self._conn = HTTP11Connection(
+            self._host, self._port, **self._h1_kwargs
+        )
+
+    def request(self, method, url, body=None, headers={}):
+        """
+        This will send a request to the server using the HTTP request method
+        ``method`` and the selector ``url``. If the ``body`` argument is
+        present, it should be string or bytes object of data to send after the
+        headers are finished. Strings are encoded as UTF-8. To use other
+        encodings, pass a bytes object. The Content-Length header is set to the
+        length of the body field.
+
+        :param method: The request method, e.g. ``'GET'``.
+        :param url: The URL to contact, e.g. ``'/path/segment'``.
+        :param body: (optional) The request body to send. Must be a bytestring
+            or a file-like object.
+        :param headers: (optional) The headers to send on the request.
+        :returns: A stream ID for the request, or ``None`` if the request is
+            made over HTTP/1.1.
+        """
+        try:
+            return self._conn.request(
+                method=method, url=url, body=body, headers=headers
+            )
+        except TLSUpgrade as e:
+            # We upgraded in the NPN/ALPN handshake. We can just go straight to
+            # the world of HTTP/2. Replace the backing object and insert the
+            # socket into it.
+            assert e.negotiated in H2_NPN_PROTOCOLS
+
+            self._conn = HTTP20Connection(
+                self._host, self._port, **self._h2_kwargs
+            )
+            self._conn._sock = e.sock
+
+            # Because we skipped the connecting logic, we need to send the
+            # HTTP/2 preamble.
+            self._conn._send_preamble()
+
+            return self._conn.request(
+                method=method, url=url, body=body, headers=headers
+            )
+
+    # Can anyone say 'proxy object pattern'?
+    def __getattr__(self, name):
+        return getattr(self._conn, name)

hyper/common/exceptions.py

@@ -42,3 +42,12 @@ class ConnectionResetError(Exception):
         """
         A HTTP connection was unexpectedly reset.
         """
+
+class TLSUpgrade(Exception):
+    """
+    We upgraded to a new protocol in the NPN/ALPN handshake.
+    """
+    def __init__(self, negotiated, sock):
+        super(TLSUpgrade, self).__init__()
+        self.negotiated = negotiated
+        self.sock = sock

hyper/contrib.py

@@ -14,9 +14,10 @@
 except ImportError:  # pragma: no cover
     HTTPAdapter = object
 
-from hyper import HTTP20Connection
+from hyper.common.connection import HTTPConnection
 from hyper.compat import urlparse
 
+
 class HTTP20Adapter(HTTPAdapter):
     """
     A Requests Transport Adapter that uses hyper to send requests over
@@ -27,15 +28,21 @@ def __init__(self, *args, **kwargs):
         #: A mapping between HTTP netlocs and ``HTTP20Connection`` objects.
         self.connections = {}
 
-    def get_connection(self, netloc):
+    def get_connection(self, host, port, scheme):
         """
-        Gets an appropriate HTTP/2 connection object based on netloc.
+        Gets an appropriate HTTP/2 connection object based on host/port/scheme
+        tuples.
         """
+        secure = (scheme == 'https')
+
+        if port is None:  # pragma: no cover
+            port = 80 if not secure else 443
+
         try:
-            conn = self.connections[netloc]
+            conn = self.connections[(host, port, scheme)]
         except KeyError:
-            conn = HTTP20Connection(netloc)
-            self.connections[netloc] = conn
+            conn = HTTPConnection(host, port, secure=secure)
+            self.connections[(host, port, scheme)] = conn
 
         return conn
 
@@ -45,20 +52,20 @@ def send(self, request, stream=False, **kwargs):
         """
         parsed = urlparse(request.url)
 
-        conn = self.get_connection(parsed.netloc)
+        conn = self.get_connection(parsed.hostname, parsed.port, parsed.scheme)
 
         # Build the selector.
         selector = parsed.path
         selector += '?' + parsed.query if parsed.query else ''
         selector += '#' + parsed.fragment if parsed.fragment else ''
 
-        stream_id = conn.request(
+        conn.request(
             request.method,
             selector,
             request.body,
             request.headers
         )
-        resp = conn.get_response(stream_id)
+        resp = conn.get_response()
 
         r = self.build_response(request, resp)