REST request where server is not closing the socket

[espruino-discuss3]

Posted at 2015-10-28 by Kolban

I am making a REST request to an HTTP server and sending a GET request to it. This is sending a response but at the end of the response, I am not seeing a response.on("close", ...) being called. My suspicion is that the server is not closing its end of the HTTP socket. Does the Espruino implementation look at the HTTP response and its Content-Length and then close the client socket or does it expect the server to close the socket?

[espruino-discuss3]

Posted at 2015-10-28 by @gfwilliams

I think Espruino expects the server to close the socket.

My guess is that because you've added your own headers, you haven't put a Connection: close header on the REST request?

---

Posted at 2015-10-28 by Kolban

I'll do a low level sniff of the HTTP protocols and see what is being sent back wards and forwards and that will prime the conversation better.

However, do we have the correct semantics just now? Here is my thinking ...

We send a HTTP GET request ...
We get back an HTTP response ...
The response is composed of:
[Header]
[Body]

The [Header] contains "Content-Length" which is the size in bytes of the body.
We call on("data") with each piece of the body received.

However ... if the server should NOT close the socket connection, then the JS app will not know that it has received a complete response. There is no "bracket" callback that says that the data received up to this point constitutes a single record response.

We might wish to capture this somewhere in documentation if these are the semantics we want to stick with.

---

Posted at 2015-10-28 by @allObjects

@Kolban, are you talking from A) an application - or B) firm/middle ware point of view? I assume the latter, because the callback given to the http (request) is a complete response - and I perceive that callback as the "bracket" callback you are talking about. Whether the 'connection' and with it the socket has to be closed right away or later, is that not defined somewhere else? ...version of the http protocol first, and then seccond with timeout?

Cuttrnyly I'm 'walking' through the AT and ESP8266_0v25-wifi modules to have a better undertanding how things work. NetworkJS and http modules are obviously built-in boudles and source cannot be found under http://www.espruino.com/modules/.

Have to find out how built in-code is pulled and where the source of it resides... Do you know?

---

Posted at 2015-10-28 by Kolban

@gfwilliams I am also working the story from the perspective of the recipient of the HTTP Request ... in this case the AutoRemote application for Android. It is worth reviewing the logs and story that I have pasted there:

http://forum.joaoapps.com/index.php?threads/tcp-or-rest-protocol-for-receiving-commands.1569/

My best guess right now is that there are two errors in play ... and again ... this is a guess. One side comes from Espruino and one side comes from AutoRemote. Here is my thinking.

1. Espruino sends a GET HTTP request to AutoRemote ... which responds with an HTTP Response but does NOT close the HTTP connection. Even though the response message has returned "Content-Length" bytes, Espruino has not closed the connection and hence no 'on(close)' callback. This may be working as implemented but it might not be working as fully desired. As a JS programmer, I now don't have a nice mechanism to know when I have received a "unit" of HTTP response even though the knowledge is present. I.e. the header of the response says the body is "XYZ" bytes long and after having received "XYZ" bytes ... there is no callback to say that the "unit" of response is present.

2. AutoRemote is not honoring the "Connection: close" request sent by Espruino which "should" close the connection after sending a single response back. However ... even though that is likely wrong of AutoRemote and I'm taking that up with the AutoRemote team, I'm not convinced that Espruino shouldn't handle this story as part of a "keep-alive" strategy.

---

Posted at 2015-10-28 by Kolban

Howdy @allObjects, I'm afraid I'm not 100% with you. This is a rather subtle area and requires a deep context for conversation. It juggles HTTP protocol, TCP/IP parlance (sockets), and the design of the 'http' module in Espruino and specifically subtleties in HTTP headers related to "Connection".

I'll be delighted to have a conversation with you on these topics but am not sure a forum post back and forth is the best way to go. If you want to go that route, keep posting and I'll keep responding. If you prefer, we can chat publicly or privately on the gitter stream. If you wish, we can also chat actually via skype ... I am always keen to find folks who wish to understand the internals of areas that I am tinkering with ... so that we may share our thinking and considerations. I find that a puzzle shared is usually a puzzle halved.

The source for the Espruino networking areas are:

libs/network/jswrap_net.c
libs/network/network.c
libs/network/socketserver.c
libs/network/http/jswrap_http.c

and then the board specific networking files.

---

Posted at 2015-10-28 by tve

For reference, here's what the HTTP/1.0 protocol spec says about the cloding of connections (http://www.w3.org/Protocols/HTTP/1.0/spec.html#Operation):

Except for experimental applications, current practice requires that the connection be established by the client prior to each request and closed by the server after sending the response. Both clients and servers should be aware that either party may close the connection prematurely, due to user action, automated time-out, or program failure, and should handle such closing in a predictable fashion. In any case, the closing of the connection by either or both parties always terminates the current request, regardless of its status.

HTTP/1.1 added a whole section on this topic: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1 the most relevant part is perhaps:

Clients and servers SHOULD NOT assume that a persistent connection is maintained for HTTP versions less than 1.1 unless it is explicitly signaled. See section 19.6.2 for more information on backward compatibility with HTTP/1.0 clients.

Also, remember that each direction of a TCP connection can be closed independently. For example, the client may close its output, which has zero effect on the other (server->client) direction. I did not see a way to do this or detect this using the Espressif SDK, however. Sigh...

---

Posted at 2015-10-28 by tve

IMHO, the implementation of the http client in espruino should:

• use HTTP/1.0
• parse the Content-Length header (I believe Chunked-Encoding can be avoided with HTTP/1.0)
• close the connection once the number of payload bytes indicated in the Content-Length header have been read

Implementing HTTP/1.1 can be a future enhancement once the above is working...

---

Posted at 2015-10-29 by @gfwilliams

Yes, sounds to me like Espruino should count bytes and close the connection when that's reached (maybe only if there's a content length header?).

However, I'd say that people have been using HTTP with Espruino for a long time, and it seems that they don't generally have problems with this. Espruino's HTTP client does seem to work fine in pretty much all cases, or there would be a lot more complaints on the forum.

As far as I can tell:

• Espruino does send HTTP/1.0 in the header
• It also always adds a Connection: close header

From the spec @tve noted above, it looks like in this case the HTTP server should definitely close the request, and AutoRemote itself is not doing what it should?

So yes, we should add content length counting, but I don't think it's actually that high priority?

---

Posted at 2015-10-29 by Kolban

Issue #684 created for tracking.