HTTP server support with full history #32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original code developed as a GSoC 2023 project by Emna Rekik. Code refactored in order to provide better bisectability as the origical commits were not bisectable. Fixes zephyrproject-rtos#59685 Fixes zephyrproject-rtos#59686 Fixes zephyrproject-rtos#59688 Fixes zephyrproject-rtos#59690 Fixes zephyrproject-rtos#59670 Fixes zephyrproject-rtos#59700 Fixes zephyrproject-rtos#59684 Fixes zephyrproject-rtos#59693 Fixes zephyrproject-rtos#59693 Fixes zephyrproject-rtos#59694 Fixes zephyrproject-rtos#59699 Fixes zephyrproject-rtos#59696 Fixes zephyrproject-rtos#59688 Fixes zephyrproject-rtos#59690 Fixes zephyrproject-rtos#59670 Fixes zephyrproject-rtos#59700 Fixes zephyrproject-rtos#59685 Fixes zephyrproject-rtos#59686 Fixes zephyrproject-rtos#59688 Fixes zephyrproject-rtos#59691 Signed-off-by: Emna Rekik <[email protected]> Signed-off-by: Jukka Rissanen <[email protected]>
Tests for HTTP server support. Signed-off-by: Emna Rekik <[email protected]> Signed-off-by: Jukka Rissanen <[email protected]>
A simple HTTP server sample application. Signed-off-by: Emna Rekik <[email protected]> Signed-off-by: Jukka Rissanen <[email protected]>
As picolib is now the default libc, use that instead of newlibc. Also use native_sim for testing instead of native_posix so that zephyr based libc is used instead of host one. Signed-off-by: Jukka Rissanen <[email protected]>
Fix the config options so that the sample app works properly with native_sim board which makes testing much easier. Signed-off-by: Jukka Rissanen <[email protected]>
Do not force debug log level but allow user to set it via prj.conf file. Signed-off-by: Jukka Rissanen <[email protected]>
No need to run as a Linux process as we can now run it in native_sim board which is in fact run as a Linux process but with Zephyr based libc. Signed-off-by: Jukka Rissanen <[email protected]>
We should not use socket API names inside network stack as it can cause issues with general Posix API support. Instead we prefix socket API calls with zsock_ so that the socket API is properly namespaced. Signed-off-by: Jukka Rissanen <[email protected]>
Place TLS setup behind IS_ENABLED() so it gets compile tested. Signed-off-by: Jukka Rissanen <[email protected]>
Have separate macros to setup a HTTPS service. Signed-off-by: Jukka Rissanen <[email protected]>
Use the http_method values for creating the bitmask of supported methods. Signed-off-by: Jukka Rissanen <[email protected]>
Move global receive buffer to each client struct so that multiple clients can connect at the same time. Also all other client specific global variables are placed to the client context struct. Signed-off-by: Jukka Rissanen <[email protected]>
The sec tag values are suppose to be supplied by the application so remove it from the library. Signed-off-by: Jukka Rissanen <[email protected]>
Save few bytes from service struct if TLS is not enabled. Signed-off-by: Jukka Rissanen <[email protected]>
The example certs are copied from echo-server sample application. Signed-off-by: Jukka Rissanen <[email protected]>
This adds support for dynamic GET requests. Signed-off-by: Jukka Rissanen <[email protected]>
Add a way to receive dynmic HTTP request messages. Signed-off-by: Jukka Rissanen <[email protected]>
Restart the http server application if start call fails. Signed-off-by: Jukka Rissanen <[email protected]>
This adds support for dynamic POST requests. Signed-off-by: Jukka Rissanen <[email protected]>
The REST support needs more thinking and is probably best placed in the application, so remove it for now. Signed-off-by: Jukka Rissanen <[email protected]>
Use one TEMP_BUF_LEN define for small buffers. Signed-off-by: Jukka Rissanen <[email protected]>
Use snake_case instead of camelCase for member variables of struct http_client_ctx. Signed-off-by: Robert Lubos <[email protected]>
When parsing HTTP/2 headers frame it does not really matter whether we've upgraded from HTTP/1 or not, we're dealing HTTP/2 now so just handle the frame. Signed-off-by: Robert Lubos <[email protected]>
Fix the HTTP/1 to HTTP/2 upgrade procedure - immediately after replying with switching protocols HTTP/1 reply, we should initiate HTTP/2 connection (by sending settings frame) and send proper reply in HTTP/2 frames. Signed-off-by: Robert Lubos <[email protected]>
Preface (settings frame) should be sent when connection starts, not as a response to the settings frame from the client. Signed-off-by: Robert Lubos <[email protected]>
Window update handling has nothing to do with HTTP/1 upgrade process, hence remove incorrect code from the handler. Signed-off-by: Robert Lubos <[email protected]>
In case of HTTP/1 to HTTP/2 upgrade, the server preface is sent directly after replying with 101 Switching Protocols response (it has to be the first frame on the stream). Therefore, it's no longer needed to send the preface again, after receiving the client preface. Signed-off-by: Robert Lubos <[email protected]>
The revents field is cleared within poll(), but in the iteration when the client is accepted, we should not process any events for that client. Signed-off-by: Robert Lubos <[email protected]>
POLLERR shouldn't be ignored, but rather treated as a regular socket error. Therefore, upon POLLERR on a client socket, close the connection, and for listening sockets, abort the server operation (ideally, we'd need to restart the server in such case). Also, when looping sockets, start with index 1, as we don't need to monitor eventfd socket in the main loop, it is verified earlier. Signed-off-by: Robert Lubos <[email protected]>
The parameters of memset call were swapped. Signed-off-by: Jukka Rissanen <[email protected]>
Add Huffman code encoder. Signed-off-by: Robert Lubos <[email protected]>
Instead of hardcoding header fields, use HPACK encoder. Signed-off-by: Robert Lubos <[email protected]>
Encode respective server settings into the settings frame, instead of replying with empty settings frame header: * SETTINGS_HEADER_TABLE_SIZE set to 0, to indicate no dynamic table support * SETTINGS_MAX_CONCURRENT_STREAMS to reflect Kconfig setting. Signed-off-by: Robert Lubos <[email protected]>
END_HEADERS and END_STREAM flags have nothing to do with settings, hence update the corresponding function names to avoid confusion. Signed-off-by: Robert Lubos <[email protected]>
As the server source file grows it's becoming harder to navigate, therefore split the source into logical subcomponents (server core, HTTP1 processing, HTTP2 processing). Signed-off-by: Robert Lubos <[email protected]>
Initialize HTTP parser during state transition so that we do it only once / request and not for every packet received. Signed-off-by: Jukka Rissanen <[email protected]>
We need to make sure that all the headers have been received before the HTTP/1 request processing can continue. Signed-off-by: Jukka Rissanen <[email protected]>
There is no need to depend on CONFIG_POSIX_API because the HTTP server is a network internal library. Cleanup remaining POSIX configs left from earlier commits. Signed-off-by: Jukka Rissanen <[email protected]>
Fix HTTP1 request processing when payload exceeds the servers buffer size. We need to be able to process the payload in fragments. Make sure we report length == 0 to the application (message end), only when the whole message has been received, and not at the end of each processed fragment. As URL/header fields can also be fragmented, take this into account when parsing. URL parsing now keeps track of the currently received part. We only parsed Upgrade header so far, but this information can also be read from the parser context, so rely on the HTTP1 parser instead. Signed-off-by: Robert Lubos <[email protected]>
Keep track of bytes received in total for current request and print that information in the end. Signed-off-by: Robert Lubos <[email protected]>
According to HTTP2 specification, Data frames are not allowed to open new streams. Upon Data frame reception, stream should already be allocated by the preceding Headers frame. Signed-off-by: Robert Lubos <[email protected]>
Fix HTTP2 data frame processing, so that it can process fragmented data. Fix HTTP2 headers frame processing, it should also be able to process fragmented headers frame. Only individual header fields are non-fragmentable (buffer has to be large enough to include entire header field). Signed-off-by: Robert Lubos <[email protected]>
Keeping track of open sockets doesn't really work well with poll(), as there may be gaps in the pollfd array (for instance, when two clients connected, and the first one closed the connection). In effect, some sockets may not really be monitored resulting in server malfunction. This could be addressed by shifting sockets withing pollfd array when sockets are closed, but since poll() ignores invalid FD anyway, we can just feed it with the total array size, as the closed socket is overwritten with a invalid FD in the pollfd array when closed. Signed-off-by: Robert Lubos <[email protected]>
Make HTTP upgrade work with dynamic resources and large payloads. Signed-off-by: Robert Lubos <[email protected]>
Instead of sending a dummy Data frame with END_STREAM flag, we can include the flag in the last Data frame with the actual data we send (this can be determined, as we won't call resource callback anymore, if there's no more data left to report) or include it in the Headers frame, if the resource handler did not report any data to send at all. Signed-off-by: Robert Lubos <[email protected]>
Use zsock_pollfd instead of pollfd. Signed-off-by: Robert Lubos <[email protected]>
Add initial implementation of flow control. For now, just send Window Updates when done processing current data frame. Signed-off-by: Robert Lubos <[email protected]>
Call it server_internal.h to be more consistent with the rest of the networking codebase. Signed-off-by: Robert Lubos <[email protected]>
It's not used and according to its content, it should not really be a part of the library. Signed-off-by: Robert Lubos <[email protected]>
Add client inactivity timer responsible for closing stale connections. Signed-off-by: Robert Lubos <[email protected]>
There's really no point allowing to spawn multiple servers from different threads with current design, as they'd all monitor the same services. Therefore, modify the server API to allow to start/stop it only, and make the server run from a background thread (single instance only). The server thread will monitor all registered HTTP services. Signed-off-by: Robert Lubos <[email protected]>
State handler should not really need to access HTTP server context, which contains low-level socket information. Signed-off-by: Robert Lubos <[email protected]>
A few symbols were missed when transitioning to ZSOCK_ prefixed APIs. Signed-off-by: Robert Lubos <[email protected]>
HPACK is mandatory for HTTP2, therefore there's no point having a Kconfig to enable it explicitly. Signed-off-by: Robert Lubos <[email protected]>
When replying with a static resource content, we can include END_STREAM flag rightaway, no need to send dummy frame. Signed-off-by: Robert Lubos <[email protected]>
Fix tests after the server API update. Remove tests that no longer make sense. Signed-off-by: Robert Lubos <[email protected]>
Whenever a resource is being accessed, it needs to be locked to avoid concurrent access from different client. We can't allow to modify the resource from two different clients at the same time for instance. Signed-off-by: Robert Lubos <[email protected]>
In case of communication erros, or in case the client simply aborts the upload, report the error to the application, so that it know no more data is expected for the resource and that the upload was incomplete. Signed-off-by: Robert Lubos <[email protected]>
The application is now able to detect when transaction was aborted and react accordingly. Signed-off-by: Robert Lubos <[email protected]>
Remove Kconfigs that are no longer in use. Align the naming to use HTTP_SERVER prefix for all related configs. Signed-off-by: Robert Lubos <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backup of the upstream HTTP server support PR