|
| 1 | +# http2-client |
| 2 | + |
| 3 | +An native-Haskell HTTP2 client library based on `http2` and `tls` packages. |
| 4 | + |
| 5 | +Hackage: https://hackage.haskell.org/package/http2-client . |
| 6 | + |
| 7 | +## General design |
| 8 | + |
| 9 | +HTTP2 is a heavy protocol. HTTP2 features pipelining, query and responses |
| 10 | +interleaving, server-pushes, pings, stateful compression and flow-control, |
| 11 | +priorization etc. This library aims at exposing these features so that library |
| 12 | +users can integrate `http2-client` in a variety of applications. In short, we'd |
| 13 | +like to expose as many HTTP2 features as possible. Hence, the `http2-client` |
| 14 | +programming interface can feel low-level for users with expectations to get an |
| 15 | +API as simple as in HTTP1.x. |
| 16 | + |
| 17 | +Exposing most HTTP2 primitives as a drawback: the library allows a client to |
| 18 | +behave abnormally with-respect to the HTTP2 spec. That said, we try to prevent |
| 19 | +notoriously-difficult errors such as concurrency bugs by coercing users with |
| 20 | +the programming API following Haskell's philosophy to factor-out errors at |
| 21 | +compile time. For instance, a client can send DATA frames on a stream after |
| 22 | +closing it with a RST (easy to spot). However, a multi-threaded client will not |
| 23 | +be able to interleave DATA frames with HEADERS and their CONTINUATIONs and the |
| 24 | +locking required to achieve this invariant is hidden (hard to implement). |
| 25 | + |
| 26 | +Following this philosophy, we prefer to offer a somewhat low-level API in |
| 27 | +`Network.HTTP2.Client` and higher-level APIs (with a different performance |
| 28 | +trade-off) in `Network.HTTP2.Client.Helpers`. For instance, |
| 29 | +`Network.HTTP2.Client.Helpers.waitStream` will consume a whole stream in memory |
| 30 | +before returning whereas `Network.HTTP2.Client` users will have to take chunks |
| 31 | +one at a time. We look forward to the linear arrows extension for improving the |
| 32 | +library design. |
| 33 | + |
| 34 | +### Versioning and GHC support |
| 35 | + |
| 36 | +We try to follow https://pvp.haskell.org/ as `a.b.c.d` with the caveat that if |
| 37 | +`a=0` then we are still slightly unhappy with some APIs and we'll break things |
| 38 | +arbitrarily. |
| 39 | + |
| 40 | +We aim at supporting GHC-8.x, contributions to support GHC-7.x are welcome. |
| 41 | + |
| 42 | +### Installation |
| 43 | + |
| 44 | +This package is a standard `Stack` project, please also refer to Stack's |
| 45 | +documentation if you have trouble installing or using this package. Please |
| 46 | +also have a look at the Hackage Matrix CI: |
| 47 | +https://matrix.hackage.haskell.org/package/http2-client . |
| 48 | + |
| 49 | +## Usage |
| 50 | + |
| 51 | +First, make sure you are somewhat familiar with HTTP and HTTP2 standards by |
| 52 | +reading RFCs or Wikipedia pages. If you use the library, feel free to shoot me |
| 53 | +an e-mail (cf. commits) or a tweet @lucasdicioccio . |
| 54 | + |
| 55 | +### Help and examples |
| 56 | + |
| 57 | +Please see some literate Haskell documents in the `examples/` directory. For |
| 58 | +a more involved usage, we currently provide a command-line example client: |
| 59 | +`http2-client-exe` which I use as a test client and you could use to test |
| 60 | +various flow-control parameters. This binary lives in a separate package |
| 61 | +at https://github.com/lucasdicioccio/http2-client-exe . |
| 62 | + |
| 63 | +The Haddocks, at https://hackage.haskell.org/package/http2-client, should have |
| 64 | +plenty implementation details, so please have a look. Otherwise, you can ask |
| 65 | +help by creating an Issue on the bug-tracker. |
| 66 | + |
| 67 | +### Opening a stream |
| 68 | + |
| 69 | +First, you open a (TLS-protected) connection to a server and configure the |
| 70 | +initial SETTINGS to advertise. Then you can open and consume streams. Opening |
| 71 | +streams takes a stream-definition and expresses two sequential parts. First, |
| 72 | +sending the HTTP headers, which reserves an increasing stream-ID with the |
| 73 | +server. Second, you consume a stream by sending DATA chunk or receiving DATA |
| 74 | +chunks. One thing that can prevent concurrency is if you have too many opened |
| 75 | +streams for the server. The `http2-client` library tracks server's max |
| 76 | +concurrency preference and will prevent you from opening too many streams. |
| 77 | + |
| 78 | +### Sending chunked data |
| 79 | + |
| 80 | +Sent data must be chunked according to server's preferences. A function named |
| 81 | +`sendData` performs the chunking but this chunking could have some suboptimal |
| 82 | +overhead if you want to repeatedly call sendData with a buffer size that is not |
| 83 | +a multiple of the server's preferred chunk size. |
| 84 | + |
| 85 | +### Flow control |
| 86 | + |
| 87 | +HTTP2 mandates a flow-control system that cannot be disabled. DATA chunks |
| 88 | +consume credit from the flow-control system. The standard defines a |
| 89 | +flow-control context per stream plus one global per-connection. |
| 90 | + |
| 91 | +** Received DATA flow control ** In order to keep receiving data you need to |
| 92 | +periodically transfer credit to the server. One transfers credit to server by |
| 93 | +calling `_updateWindow`, which transfers locally-accumulated credit (you |
| 94 | +accumulate credit with `addCredit`). The current implementation already follows |
| 95 | +a "zero-sum" credit where received DATA is immediately consumed and |
| 96 | +re-credited. That is, if you only keep calling `_updateWindow` at some |
| 97 | +frequency the stream will progress. You can also `_addCredit` to permit |
| 98 | +receiving more DATA on a stream/connection (e.g., if you want to implement |
| 99 | +something like TCP slow-start). |
| 100 | + |
| 101 | +** Sent DATA flow control ** A server following the HTTP2 specification |
| 102 | +strictly will kick you for sending too much data. The `http2-client` library |
| 103 | +allows you to be more aggressive than the server allows and you have to care |
| 104 | +for your streams. We provide an incoming flow-control context that will allow |
| 105 | +you to call `_withdrawCredit` to wait until some credit is available. At the |
| 106 | +time of this writing, the `sendData` function does not call `_withdrawCredit` |
| 107 | +and we provide no equivalent. Note that the chunking and flow-control |
| 108 | +mechanisms have interesting interactions in HTTP2 in a multi-threaded context. |
| 109 | +Pay attention to always take credit in the per-stream flow-control context |
| 110 | +before taking it from the global per-connection flow-control context. |
| 111 | +Otherwise, you risk starving the global per-connection flow-control with no |
| 112 | +guarantee that you'll be allowed to send a DATA frame. |
| 113 | + |
| 114 | +## Settings changes |
| 115 | + |
| 116 | +The HTTP2 RFC acknowledges the inherent race conditions that may occur when |
| 117 | +changing SETTINGS. The `http2-client` library should be rather permissive and |
| 118 | +accept rather than reject frames caught violating inconsistent settings once |
| 119 | +client settings are made stricter. Conversely, the `http2-client` library tries |
| 120 | +to enforce server-SETTINGS strictly before ACKnowledging the setting changes. |
| 121 | +This configuration can lead to problems if the server send more-permissive |
| 122 | +SETTINGS (e.g., allowing a large default window size -> which recredits all |
| 123 | +streams) but if the server applies this change locally only after receiving the |
| 124 | +client ACK. One way to be double-sure the `http2-client` library is always |
| 125 | +strict would be to apply settings changes in two steps: settings that move in |
| 126 | +the "stricter direction" (e.g., fewer concurrency, smaller initial window) |
| 127 | +should be applied _before_ ACK-ing the SETTING frame. Meanwhile settings that |
| 128 | +move in the "looser direction" (e.g., more concurrency) should be applied |
| 129 | +_after_ ACK-ing the SETTINGS frame. |
| 130 | + |
| 131 | +The current design apply SETTINGS: |
| 132 | +- (client prefs) after receiving a ACK for sent SETTINGS, you get the choice to |
| 133 | + wait for an ACK or wait in a thread, but you must wait for an ACK to apply |
| 134 | + changed settings (the `_settings` function will return an IO to wait for the |
| 135 | + ACK and apply settings). Note that the initial SETTINGS change frame is |
| 136 | + waited for in a thread without library's user intervention (if you feel |
| 137 | + strongly against this choice, please open a bug). |
| 138 | +- (server prefs) immediately after receiving and hence before sending ACK-SETTINGS |
| 139 | + |
| 140 | +Fortunately, changing settings mid-stream is probably a rare behavior and the |
| 141 | +default SETTINGS are large enough to avoid creating fatal errors before |
| 142 | +sending/receiving the initial SETTINGS frames. |
| 143 | + |
| 144 | +## Things that are hardcoded |
| 145 | + |
| 146 | +A number of HTTP2 features are currently hardcoded: |
| 147 | +- PINGs are replied-to immediately (i.e., a server could hog a connection with PINGs) |
| 148 | +- the initial SETTINGS frame sent to the server is waited-for in a separate |
| 149 | + thread, settings are applied to the connection when the server ACKs the frame |
| 150 | +- flow-control from DATA frames is decremented immediately when received (in a |
| 151 | + separate thread) rather than when consumed from the client |
| 152 | +- similarly, flow-control re-increment every DATA received as soon as it is |
| 153 | + received |
| 154 | + |
| 155 | +## Contributing |
| 156 | + |
| 157 | +Contributions are welcome. As I start integrating external contribution I plan |
| 158 | +to follow the following procedure: |
| 159 | +- stop pushing directly into master |
| 160 | +- develop any patch in a new branch, branched from master |
| 161 | +- merge requests target master |
| 162 | + |
| 163 | +Please pay attention to the following: |
| 164 | +- avoid introducing external dependencies, especially if dependencies are not in stackage |
| 165 | +- avoid reformatting-only merge requests |
| 166 | +- please verify that you can `stack clean` and `stack build --pedantic` |
| 167 | + |
| 168 | +General mindset to have during code-reviews: |
| 169 | +- be kind |
| 170 | +- be patient |
| 171 | +- surpass egos and bring data if there is a disagreement |
| 172 | + |
| 173 | +## Bugtracker |
| 174 | + |
| 175 | +Most of the following points have their own issues on the issue tracker at |
| 176 | +GitHub: https://github.com/lucasdicioccio/http2-client/issues . |
| 177 | + |
| 178 | +### Things that will likely change the API |
| 179 | + |
| 180 | +I think the fundamentals are right but the following needs tweaking: |
| 181 | + |
| 182 | +- function to reset a stream will likely be blocking until a RST/EndStream is |
| 183 | + received so that all DATA frames are accounted for in the flow-control system |
| 184 | +- need a way to hook custom flow-control algorithms |
| 185 | + |
| 186 | +### Support of the HTTP2 standard |
| 187 | + |
| 188 | +The current implementation follows the HTTP2 standard except for the following: |
| 189 | +- does not handle `PRIORITY` |
| 190 | +- does not expose padding |
| 191 | +- does not handle `SETTINGS_MAX_HEADER_LIST_SIZE` |
| 192 | + * it's unclear to me whether this limitation is applied per frame or in total |
| 193 | + * the accounting is done before compression with 32 extra bytes per header |
| 194 | +- does not implement most of the checks that should trigger protocol errors |
| 195 | + * unwanted frames on idle/closed streams |
| 196 | + * increasing IDs only |
| 197 | + * invalid settings |
| 198 | + * invalid window in flow-control |
| 199 | + * invalid frame sizes |
| 200 | + * data-consumed out of flow-control limits |
| 201 | + * authority of push promise https://tools.ietf.org/html/draft-ietf-httpbis-http2-17#section-10.1 |
| 202 | + * ... |
0 commit comments