http res.on('close') never called for some GET requests

[espruino-discuss3]

Posted at 2015-03-11 by net-tobi

Dear community,

sorry for writing again about problems I experienced with Espruino. I used the latest build of 1.75.

I experienced that the "close" event of the http response is not raised for the same get request. No matter how often I try. On the other side I am using the Express JS framework. With other HTTP clients it works well.

    var request = function() {
        var body = "";
        var options = {
            method: 'GET',
            host: '192.168.0.11',
            port: 3000,
            path: "http://localhost:3000/api/Entities/54fabe55ef581d7d5f8bcac1/tasks?filter[where][active]=true&filter[fields][id]=true&access_token=I9sbI5cy6m1gHONeNmCHULGX4CwSGBzC0RDdnPDCfMW4or2UmDyxWg0ZbnQAT1yN",
            headers: {
                "Content-Type": "application/json",
                "Content-Length": body.length
            }
        };
        var contents = "";
        var req = require('http').request(options, function(res) {
            console.log(res);
            res.on('data', function(data) {
                contents += data;
            });
            res.on('close', function() {
                console.log('CLOSE');
                if(res.statusCode === "200" || res.statusCode === "204") {
                    console.log('OK');
                } else {
                    console.log('FAIL');
                }
                contents = "";
            });

        });
        req.on('error', function(e) {
            console.log('Error');
        });
        req.write(body);
        req.end();
    };

The response looks ok:

{
  "headers": {
    "X-Powered-By": "Express",
    "Vary": "Origin, Accept-Encoding",
    "Access-Control-Allow-Credentials": "true",
    "Content-Type": "application/json; charset=utf-8",
    "Content-Length": "2",
    "ETag": "W/\"2-d4cbb29\"",
    "Date": "Wed, 11 Mar 2015 13:30:53 GMT",
    "Connection": "close"
   },
  "httpVersion": "1.1",
  "statusCode": "200",
  "statusMessage": "OK",
  "#ondata": function (b) {l+=b}
 }

How can I handle such problems with the http class of Espruino? What is the problem, that the "close"-event handler is not getting called?

Best,

Tobias

[espruino-discuss3]

Posted at 2015-03-11 by @gfwilliams

Can you try requesting a bigger file? With 2 characters I guess it's possible that the connection would have already closed by the time the main function was called. I know I deal with that for the data handler, but maybe not the close handler.

To work around it, you could do:

var req = require('http').request(options, function(res) {
            console.log(res);
            var l = res.headers["Content-Length"];
            res.on('data', function(data) {
                contents += data;
                if (contents.length>=l) console.log("Do stuff now");
            });
        });

---

Posted at 2015-03-11 by net-tobi

Hi Gordon,

with a bigger file the close handler is getting called and it works as expected.

Best,

Tobias

---

Posted at 2015-03-11 by @gfwilliams

Ok, thanks - I'll make a bug for it.

The workaround I suggested should fix it for now though

---

Posted at 2015-03-11 by net-tobi

Thank you!

---

Posted at 2015-03-11 by @gfwilliams

bug: espruino/Espruino#503

---

Posted at 2015-03-11 by @gfwilliams

Just to add that:

        var req = require('http').request(options, function(res) {
            // ...
        });
         req.on('close', function() { ... });

Should also work for you - and might be better.

---

Posted at 2015-03-11 by net-tobi

Both workarounds don't work for me. The response is closed too fast, so the on-handlers for "close" and "data" are not getting called, because it is too late.

---

Posted at 2015-03-11 by @gfwilliams

That's odd - the data handler should work. What should happen in that case is that when the on('data' handler is added, the internal stream buffer is checked for data and if there is any, the function is queued up with it immediately.

---

Posted at 2015-03-12 by net-tobi

It seems to be a timing problem of the asynchronous calls. I experienced situations where it worked with the same function and the same data without any changes being made. But for now it is not working for me.

---

Posted at 2015-03-20 by net-tobi

Do you have another idea how I can realize a workaround since both suggestions doesn't work for me?

---

Posted at 2015-03-20 by @gfwilliams

Ok, I just tried what was basically your code, on a WIZnet device (with the current firmware), connecting to an Apache server:

var eth = require("WIZnet").connect();

var request = function() {
  var body = "";
  var options = {
    method: 'GET',
    host: '192.168.1.50',
    port: 80,
    path: "/ok.txt",
    headers: {
      "Content-Type": "application/json",
      "Content-Length": body.length
    }
  };
  var contents = "";
  var req = require('http').request(options, function(res) {
    console.log(res);
    res.on('data', function(data) {    
      console.log('DATA>'+data);
      contents += data;
    });
    res.on('close', function() {
      console.log('CLOSE');
      if(res.statusCode === "200" || res.statusCode === "204") {
        console.log('OK');
      } else {
        console.log('FAIL');
      }
      contents = "";
    });
  });
  req.on('error', function(e) {
    console.log('Error');
  });
  req.write(body);
  req.end();
};

request();

Result is:

{
  "headers": {
    "Date": "Fri, 20 Mar 2015 10:46:45 GMT",
    "Server": "Apache/2.2.22 (Ubuntu)",
    "Last-Modified": "Fri, 20 Mar 2015 10:23:31 GMT",
    "ETag": "\"d44028-3-511b5b54ad422\"",
    "Accept-Ranges": "bytes",
    "Content-Length": "2",
    "Vary": "Accept-Encoding",
    "Connection": "close",
    "Content-Type": "text/plain"
   },
  "httpVersion": "1.1",
  "statusCode": "200",
  "statusMessage": "OK"
 }
DATA>X
CLOSE
OK

And it works every time. Any thoughts about what could be different?

What happens when you use Espruino with Apache, rather than Express? Maybe Express is somehow doing something different?

As far as workarounds, if you control the server and we can't figure this our, why not just send a few more bytes?

---

Posted at 2015-03-20 by net-tobi

Thank you for your response again. I'll try that out on the weekend with Apache and try to deeper analyze the problem in my special case. Currently my workaround for the problem is to set a timeout before I send the request and call the callback if nothing is getting returned.

I'll provide you with updates soon.

---

Posted at 2015-03-20 by @gfwilliams

Are you using WIZnet, or some other module for accessing the net? I guess that could have some affect (although I tried on Linux as well).

Also I wonder whether it is because there is a lot of code being called after request() - maybe try:
setTimeout(request,10);

---

Posted at 2015-03-20 by net-tobi

I am using the CC3000 module. I'll try it with setTimeout as well at the weekend

---

Posted at 2015-03-22 by net-tobi

I spent some time to analyze the problem... It seems to be a problem with my CC3000, because sometimes the same code worked and sometimes not...

I get such errors very often:

ERROR: Timeout in CC3000 driver (4103)
ERROR: Connect failed (err -1)
CC3000 WiFi is not responding. Power cycling...

and

ERROR: Socket error -1 while sending

Maybe it is a problem with the firmware. Do you know exactly which USB cable I need for an update?

What is the state of the ESP8266 problems? Maybe I can use that, if these are solved.

---

Posted at 2015-03-23 by net-tobi

I found a way to update my CC3000, I now have less error messages, but the problem with the close Handler persists.

---

Posted at 2015-03-23 by @gfwilliams

Yes, the CC3000 isn't desperately reliable. When I get a moment I'll see if I can try a similar 2-character request with it - but honestly it wouldn't surprise me if it was some problem with the module itself. I seem to recall that if you tried to receive data on a closed socket it'd just crash, so it's possible that there is no way to get the small response out of it.

Is this is for your uni project? Can you not just modify your server slightly to either send more data or to hold the connection open for a few more ms after sending?

The ESP8266 support is getting there, but it still needs some work as different versions of firmware need slightly different commands.

Did you try connecting to an Apache server?

---

Posted at 2015-03-23 by net-tobi

Thanks gordon. Yes it is for the project I explained to you via email.

Sure I can send more data from the server side, that's a workaround which is ok for the moment, but the final solution shall be able to process the data in a correct way (wether it is less or more data).

I performed some tests with Apache which came to the same result.

I have no problem switching to the ESP8266 when it is working in a proper way with the latest firmware.

In this project I use WLAN for the presentation. The final version of the system shall use GPRS which I will adapt as soon it is officially supported.

Best,

Tobias

---

Posted at 2015-05-01 by @gfwilliams

Just to add, I recently made a change (it'll auto-build to http://www.espruino.com/binaries/git/commits/5ae24206fb924569f14265fa57372132356a7a75 soon, or will be in 1v78) that may well fix the problem you're having with close not being called.