No callback on http request

[espruino-discuss3]

Posted at 2015-05-10 by Schweigi

It seems that the callback of require("http").request is not executed. When using require("http").get then the callback is executed. I use the Pico with 1v78. Originally I wanted to make a post request but never got a response so I tried to reproduce it with the get command.

Remark: It looks like the forum engine injects a http:// in front of the host name.

Callback is executed for:

var req = require("http").get("www.google.com", function(res)  {
        console.log("status: " + res.statusCode);
});

Callback (print status code) is not executed for:

var options = { host: "www.google.com" };
var req = require("http").request(options, function(res)  {
        console.log("status: " + res.statusCode);
});

Does somebody know how to solve this?

[espruino-discuss3]

Posted at 2015-05-10 by DrAzzy

First, try a site that doesn't do anything weird (like google.com does)

Failing that, post your code (be sure to snip out your wifi password).

What network adapter are you using? ESP or Wiznet?

I know it works on v78 with the ESP8266

---

Posted at 2015-05-10 by Schweigi

Ups. I forgot to call req.end() when I created the test code for the GET request. I'm sorry for the confusion. So the following is working too now:

var options = { host: "www.google.com" };
var req = require("http").request(options, function(res)  {
        console.log("status: " + res.statusCode);
});
req.end(); // <=== was missing

But I still have some issues with the post call not ending. I think its because of the Mandrill API. The following sample code (POST req. to Requestbin) is working. The console prints the status code (200) after the request:

Serial1.setup(9600, { rx: B7, tx : B6 });
var wifi = require("ESP8266WiFi").connect(Serial1, function(err) {
  if (err) throw err;

  wifi.reset(function(err) {
    if (err) throw err;

    wifi.connect("<WIFI_SSID>", "<WIFI_PW>", function(err) {
      if (err) throw err;

      var content = JSON.stringify({ "msg": "Hello World!"});
      var options = { 
        host: "requestb.in", 
        port: 80, 
        path: "/1lxvhjs1", 
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "Content-Length": content.length
        }
      };
      var req = require("http").request(options, function(res)  {
          console.log("status: " + res.statusCode);
      });
      req.end(content);
      console.log('Request sent');
    });
  });
});

But when I do the same for the Mandrill API then the post request is not ending. Mandrill gets the request successfully but its seems that the Espruino is still waiting for a response:

Serial1.setup(9600, { rx: B7, tx : B6 });
var wifi = require("ESP8266WiFi").connect(Serial1, function(err) {
  if (err) throw err;

  wifi.reset(function(err) {
    if (err) throw err;

    wifi.connect("<WIFI_SSID>", "<WIFI_PW>", function(err) {
      if (err) throw err;
      
      var content = JSON.stringify({
        "key": "<MANDRILL_API_TOKEN>",
        "text": "This is a demo",
        "message": {
          "from_email": "[email protected]",
          "to": [
            {
              "email": "<YOUR_EMAIL>",
              "type": "to"
            }
          ]
        }   
      });
      
      var options = { 
        host: "mandrillapp.com", 
        port: 80, 
        path: "/api/1.0/messages/send.json", 
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "Content-Length": content.length
        }
      };
      var req = require("http").request(options, function(res)  {
          console.log("status: " + res.statusCode);
      });
      req.end(content);
      console.log('Request sent');
    });
  });
});

I will try to investigate further. The same request (http & https) using a rest-client (Postman) is working correctly.

---

Posted at 2015-05-10 by Schweigi

I re-routed the http request form the Pico trough a proxy on my computer. The request itself is working but the Pico doesn't seem to close the connection. Also sometimes (10% of the time) not all the payload data is sent (it gets stuck at 272 bytes). I added a wifi.at.debug and after the data is sent the following is printed by at.debug:

] "> \r" <--- "\r"
] "\nER" <--- "\nER"
] "ERRO" <--- "RO"
] "ERROR" <--- "R"
] "ERROR\r\n" <--- "\r\n"
] "Unl" <--- "Unl"
] "Unlin" <--- "in"
] "Unlink" <--- "k"
] "Unlink\r\n" <--- "\r\n"
["AT+CIPCLOSE=0\r\n"
] "l" <--- "l"
] "li" <--- "i"
] "link" <--- "nk"
] "link i" <--- " i"
] "link is" <--- "s"
] "link is n" <--- " n"
] "link is not" <--- "ot"
] "link is not\r\n" <--- "\r\n"

I guess this error is the problem. But I don't know where its coming from.

---

Posted at 2015-05-10 by @gfwilliams

Hm, not sure about the link error you're getting (Espruino seems to be finding the error message from the Esp8266 and trying to close the connection).

However I think usually Espruino waits for the server to close the connection rather than counting bytes from content length - it usually sends the Connection: close header. While it does for a normal get I'm not sure if it does when you call request. Maybe you could try adding it?

---

Posted at 2015-05-10 by Schweigi

I checked. The Connection: close header is also added on the .request call by default. I think the issue is related to the content length. If I send too much data then I get the same behavior as with the Mandrill API also with the Requestbin API example.

var CONTENT_LENGTH = 10; //<== always ok
//var CONTENT_LENGTH = 250-300; //<== request is sent but no answer received
//var CONTENT_LENGTH = 800; //<== request is not sent

Serial1.setup(9600, { rx: B7, tx : B6 });
var wifi = require("ESP8266WiFi").connect(Serial1, function(err) {
  if (err) throw err;
  wifi.reset(function(err) {
    if (err) throw err;
    wifi.connect("<WIFI_SSID>", "<WIFI_PW>", function(err) {
      if (err) throw err;
      
      var content = "";
      for(var i=0; i < CONTENT_LENGTH; i++) {
        content += "x";
      }
      
      var options = { 
        host: "requestb.in", 
        port: 80, 
        path: "/1lxvhjs1", 
        method: "POST",
        headers: {
          "Content-Type": "text/plain",
          "Content-Length": content.length
        }
      };
      
      wifi.at.debug();
      var req = require("http").request(options, function(res)  {
          console.log("status: " + res.statusCode);
      });
      req.end(content);
      console.log("Request sent (length: " + content.length + ")");
    });
  });
});

While debugging I also found three other issues. I can reproduce those with a simple GET call. Just attach a ESP8266WiFi to your Pico and press the build-in button to trigger a request.

1. The first issue is, that when I do multiple requests all but the first produce an error (see AT debug msgs). I event tried to give it a couple of minutes between the button pushes to be sure the prior connection is really dead but it does't help. I never managed to make more than one successful http request without restarting the Pico in-between.
2. The second issue is, that if there is no res.on('data') event handler, then res.on('close') isn't called either (also CIPCLOSE is never called).
3. The third issue is, that for some hosts (or probably servers) the connection isn't closed after the GET is done but only after the server closes it. Thus it can't take a long time until the data is available to read - see BBC example.

var URL = "http://www.pur3.co.uk/hello.txt";

/*
  ** ISSUE 3 **:
  Call to the BBC page closes only 60s after request has been started.
  The AT debug messages show that the response is received immediately but 
  the call is only closed after a timeout.
  
  wifi.at.debug() output after 60s:
  ] "\r" <--- "\r"
  ] "\nE" <--- "\nE"
  ] "ERR" <--- "RR"
  ] "ERRO" <--- "O"
  ] "ERROR\r" <--- "R\r"
  ] "\nUn" <--- "\nUn"
  ] "Unli" <--- "li"
  ] "Unlink" <--- "nk"
  ] "Unlink\r" <--- "\r"
  ] "\n" <--- "\n"
  ["AT+CIPCLOSE=0\r\n"
  
var URL = "http://open.live.bbc.co.uk/weather/feeds/en/2647937/observations.rss";
*/

Serial1.setup(9600, { rx: B7, tx : B6 });
var wifi = require("ESP8266WiFi").connect(Serial1, function(err) {
  if (err) throw err;
  wifi.reset(function(err) {
    if (err) throw err;
    wifi.connect("<WIFI_SSID>", "<WIFI_PW>", function(err) {
      if (err) throw err;

      console.log("Press button to trigger a request");
      wifi.at.debug();
      
      setWatch(function(e) {
        var req = require("http").get(URL, function(res)  {
          console.log("Connection status: " + res.statusCode);
          
          var content = "";
          /* 
            ** ISSUE 2 **
            If the res.on('data') event handler is removed 
            then "AT+CIPCLOSE=0" is never executed and 
            res.on('close') is never called.
          */  
          res.on('data', function(data) {
            content += data;
          });
          
          res.on('close', function() {
            console.log("Connection data: " + content);
            console.log("Connection closed");
          });
        });
        
        console.log('Request sent');
      }, BTN, { repeat: true, debounce : 50, edge: "rising" });
    });
  });
});

Btw my Pico firmware has 1v78 and the ESP8266 has 0018000902-AI03.

---

Posted at 2015-05-10 by DrAzzy

espruino/EspruinoDocs#138 <-multiple requests don't work because the Esp8266WiFi module doesn't manage the callbacks correctly for multiple connections.

---

Posted at 2015-05-10 by Schweigi

My issue (1) is if the requests are serial, not parallel. So only one request at a time.

---

Posted at 2015-05-10 by Schweigi

I finally found it.

I used a breadboard for all my experiments with the ESP8266 on it. Apparently this was the issue. After soldering the ESP8266 to a shim all communication errors are gone.

One small issue I mentioned in my previous post (issue (2)) is still left. When there is no res.on('data') event handler then the res.on('close') handler is never called.I'm not sure if this is by design or a bug.

---

Posted at 2015-05-11 by @gfwilliams

When there is no res.on('data') event handler then the res.on('close') handler is never called.

No, I reckon that's a bug - thanks for letting me know! I'll get a fix in for that.

---

Posted at 2015-05-11 by @gfwilliams

Ok, fix now in. It'll be in the 1v79 release, or you can get it from the GitHub builds