MQTT client and HTTP module don't work together?

[espruino-discuss]

Posted at 2018-01-25 by sureshkm

Hi @gfwilliams and others,
I've used Espruino Pico with ESP8266 on it. My device is trying to connect to the internet the similar way that the Amazon Echo or Google Home connects.

I collect the WiFi credentials using a web form that is served from the HTTP server that runs on Espruino Pico,

apServer = require("http").createServer(hsRouter).listen(80);

After collecting the WiFi credentials and storing them in the ROM, i close the server, connect to the internet and then finally connect to my MQTT broker that runs on Digital Ocean.

var client = require("MQTT").create(server, options /*optional*/);
....  // some code
apServer.close()
....  // some code
client.connect();

I get the error message from MQTT client saying that the connection is closed.

in function called from system
Uncaught Error: This socket is closed.
 at line 1 col 56
...ite(b(a.DISCONNECT<<4)+'\0'),this.client.end(),this.client=!...
                              ^
in function "disconnect" called from line 1 col 52
c.ctimo=void 0,c.emit('disconnected'),c.disconnect();
                                                   ^

But the MQTT broker is running fine and i get to see that MQTT client could reach the broker server. Instead of MQTT client call, if i make the following call, it works,

require("http").get("http://pur3.co.uk/hello.txt", function(res) {
  res.on('data', function(data) {
    console.log("HTTP> "+data);
  });
  res.on('close', function(data) {
    console.log("Connection closed");
  });
});

I believe that there is something happening only after i run the HTTP server and close. Because if i retrieve the WiFi credentials from the memory instead of running the HTTP server and I'm able to connect to the MQTT broker successfully.

Please help me.. thanks.

[espruino-discuss]

Posted at 2018-01-26 by PaddeK

Hmm looks like the close() for the apServer closes the MQTT connection too.

---

Posted at 2018-01-26 by sureshkm

@PaddeK But the MQTT makes a connection during

client.connect();

and not during

var client = require("MQTT").create(server, options /*optional*/);

Right? In that case, client connection is happening after the close() call.

---

Posted at 2018-01-26 by @gfwilliams

So you're saying that if you just run require("MQTT").create on its own, without the AP, it works?

Can you try doing something like:

apServer.close()
setTimeout(function() {
  var client = require("MQTT").create(server, options /*optional*/);
  client.connect();
}, 2000);

And see if that works ok?

The calls to the ESP8266 will be asynchronous, so I guess it's possible that the driver isn't correctly delaying the creation of the new MQTT connection, and so is trying to re-use the apServer socket that hasn't yet closed?

---

Posted at 2018-01-26 by sureshkm

Sure, i will try.

But if i make HTTP request in the place of MQTT connect, I'm getting the HTTP response back from the server. The following code works

apServer.close()
......
require("http").get("http://pur3.co.uk/h­ello.txt", function(res) {
  res.on('data', function(data) {
    console.log("HTTP> "+data);
  });
  res.on('close', function(data) {
    console.log("Connection closed");
  });
});

---

Posted at 2018-01-26 by sureshkm

@gfwilliams

var   client = require("MQTT").create('IP ADDRESS', options /*optional*/);
....
function setup () {
         wifi = require("ESP8266WiFi_0v25").connect(Serial2, function(err) {

        // reset the n/w before the connection
        wifi.reset(function(err) {
            throwError(err);
            // SSID, Access Point PWD, Email id, Token Details, Session Token
            var apSSID = typeof f.read(0) !== 'undefined' ? E.toString(f.read(0)) : '',
                apPWD = typeof f.read(1) !== 'undefined' ? E.toString(f.read(1)) : '';
        
            if (apSSID) {
                wifi.connect(apSSID, apPWD, function(err) {
                    console.log('Device is successfully connected to the WiFi.');

                    // MQTT starts here
                    client.connect();
                    ....
           } else {
                    // creating the AP, running the HTTP server code
                    // after getting the WiFi creds and saving that to the ROM, 
                    apServer.close();
                    setup();
           }
}

If you notice, whenever there is a cold start, i instantiate the ESP8266WiFi_0v25 twice in the flow and that is the case where it does not work.

Every time after the cold start, it instantiates the ESP8266WiFi_0v25 once and it works fine.
Does it give any clue?
Does it need to do anything with the AP or Station mode of the Wifi module?

---

Posted at 2018-01-26 by @gfwilliams

Ok, yes - I think one problem might be that you're calling require("ESP8266WiFi_0v25").connect twice. In that case you now have two ESP8266 bits of code trying to run concurrently, which'll really mess things up!

Try only initialising it once. Or if you feel you really have to do it twice, use clearTimeout() and Serial2.removeAllListeners() first.

Also, you're trying to shut a socket and then immediately restart the ESP8266, before the command has even finished sending.

Try tweaking your code to add a timeout as well:

apServer.close();
setTimeout(setup, 1000);

---

Posted at 2018-01-26 by PaddeK

If the http module is the one from nodejs the close function does have a callback parameter.
It gets called when the connection is actually closed.

-- Edit --
Just for completeness sake.. you could also listen to the "close" event.

---

Posted at 2018-01-26 by sureshkm

Thanks @PaddeK, i just tried to use the callback, but unfortunately there is no callback it seems.

---

Posted at 2018-01-26 by sureshkm

I tried adding Serial2.removeAllListeners() and tested, it does not work.
I tried initializing the require("ESP8266WiFi_0v25").connect only once and tested, it does not work.

But all the scenarios, if i make a new HTTP connection instead of client.connect() (not making any MQTT calls), i'm getting the response and the socket is not closed.

I strongly believe that there is something to do with client.connect() , especially it happens only when i run/close the HTTP server first and try to connect to the MQTT.

---

Posted at 2018-01-27 by sureshkm

Ok, i figured out that this happens intermittently, so i'm trying to at least suppress the uncaught error, it does not work, but why? Isn't the try..catch supported?

try {
     client.connect();
} catch(e) {
    // do nothing ...
}

---

Posted at 2018-01-29 by @gfwilliams

There's no close callback, but you could probably hook onto the event if needed.

try...catch works fine, but the error isn't happening at that point in your code. Because it's async the error happens at some point when the rest of your code is idling.

Is it possible to post up a complete set of code that gives you the problem? Ideally as minimal as possible.

I'm a bit confused since the error that's being reported in MQTT points to lines that don't actually exist in the MQTT module on Espruino.com: http://www.espruino.com/modules/MQTT.min.js

Are you using your own version of the modules?

---

Posted at 2018-01-31 by sureshkm

Hi @gfwilliams, Thanks for looking into it.

I've unchecked 'Modules uploaded as functions (BETA)' and 'Javascript compiler' from the config. I'm not sure if they make any difference. But i get the following error and that line is part of MQTT.min.js file.

Uncaught Error: This socket is closed.
 at line 1 col 58
...e(g(c.DISCONNECT<<4)+"\x00"),this.client.end(),this.client=!...
                              ^
in function "disconnect" called from line 1 col 52
a.ctimo=void 0;a.emit("disconnected");a.disconnect()

I will try to post up the code soon.

---

Posted at 2018-02-01 by @gfwilliams

Ahh, ok - that looks more like the right file now - but still the same error :(

If you could post up a complete file that has the problem then it'd really help to track it down though.

---

Posted at 2018-02-01 by sureshkm

@gfwilliams I've posted some main parts of my code base. Hope this helps!

var wifi,
    wifiAPs = {},
    f = new (require("FlashEEPROM"))(),
    apServer,
    isProduction = true; // use this for debugging

function clog(s) {
  console.log(s);
}

/**
 * the MQTT broker server configurations
 */
var server = '192.168.1.14', // the ip of MQTT broker
    options = {
        client_id : UUID,
        //keep_alive: 60, // keep alive time in seconds
        port: 1883, // port number
        clean_session: true,
        username: "username", // name for auth that happens in the broker server
        password: "password",  // pwd for auth that happens in the broker server
        protocol_name: "MQTT", // or MQIsdp, etc..
        protocol_level: 4, // protocol level
    },
    client = require("MQTT").create(server, options /*optional*/);


/**
 * throwing error exceptions
 */
function throwError(err) {
    if (err) {
        progress = false;
        throw err;
    }
}

/**
 * Temporary server to collect the WiFi credentials like SSID, pwd etc
 * @method hsInit
 */
function hsInit() {
    apServer = require("http").createServer(hsRouter).listen(80);
}

/**
 * all the requests will be routed thro this
 * @method hsRouter
 */
function hsRouter(req, res) {
    getIndexHTML(req, res);
}

/**
 * index/home page of the Hula Server where we collect the WiFi credentials
 * @method getIndexHTML
 */
function getIndexHTML(req, res) {
    var pURL = url.parse(req.url, true);
    if (req.method === 'POST') {
        var postData = '',
            parsedData;

        req.on('data', function(d) {
            postData += d;
        });

        req.on('end', function(d) {
            parsedData = parseQuery(postData);
            clog(parsedData);

            // clear absolutely everything out of memory
            f.erase();

            // if there is a WiFi SSID given (required)
            if (parsedData.ap) {
                f.write(0, parsedData.ap);   
                clog(E.toString(f.read(0)));
            }

             // if there is a WiFi PWD given (required)

            // lets setup as we got the WiFi creds
            apServer.close();
            sTHandlerSetup = setTimeout(setup, 5000);
        });
    }

    // default/index route to display the web page
    if (pURL.pathname == '/') {

        // the HTML page that collects the WiFi creds
        res.writeHead(200, {'Content-Type': 'text/html'});
        // HTML Form to collect the required info 
        res.end('</body></html>'); 
    }
}


/**
 * initiate the required functions while booting the pico
 * @method onInit
 */
function onInit() {
    var setWatchId,
        pTime;

    // the external button for setting up 
    pinMode(B3, "input_pulldown");
    setWatchId = setWatch(function(e) {
        pTime = e.time - e.lastTime;

        // do not accept if it happens within 2 secs
        if(!progress && (isNaN(pTime) || pTime > 2)) {
            clearWatch(setWatchId);
            clog('Got you! Hang on! Im booting up.');
            progress = true;
            setup();
        }
    }, B3, { repeat: true, debounce : 150, edge: "rising" });
}

function setup() {

    if (sTHandlerSetup) {
      clog('clearing the setup timeout ...');
      clearTimeout(sTHandlerSetup);
    }
    Serial2.setup(115200, { rx: A3, tx : A2 });
    Serial2.removeAllListeners();
    wifi = require("ESP8266WiFi_0v25").connect(Serial2, function(err) {
        throwError(err);

        // reset the n/w before the connection
        wifi.reset(function(err) {
            throwError(err);
            // SSID, Access Point PWD, Email id, Token Details, Session Token
            var apSSID = typeof f.read(0) !== 'undefined' ? E.toString(f.read(0)) : '',
                apPWD = typeof f.read(1) !== 'undefined' ? E.toString(f.read(1)) : '';

            if (apSSID) {
                wifi.connect(apSSID, apPWD, function(err) {
                    throwError(err);
                    clog('Device is successfully connected to the WiFi.');
                    progress = false;
                    var sTHandlerMQTT;

                    sTHandlerMQTT = setTimeout(function() {
                        clog('clearing the MQTT timeout ...');
                        clearTimeout(sTHandlerMQTT);

                        // MQTT starts here
                        client.connect();

                        // MQTT: if there is any error 
                        client.on('error', function () {
                            clog('connection to the MQTT broker server failed');
                        });

                        // MQTT: while connecting
                        client.on('connected', function () {
                            clog('MQTT connected ...');
                        });

                        // MQTT: watching out for the messages that come in from the MQTT broker
                        client.on('message', function (topic, message) {
                            // logic for handling the messages
                        });

                        // in case, the MQTT broker server is down, reconnect
                        client.on('disconnected', function() {
                            clog("MQTT disconnected... reconnecting.");

                            // reconnect 3 times in every 5s
                            // 4 times in every 15s and 10 times in every 60s
                            client.connect();
                        });
                    }, 3000);
                });
            } else {

                // only if there is any WiFi SSID (Service Set IDentifier) in the EEPROM memory
                new Promise(function(resolve, reject) {

                    wifi.getAPs(function(err, aps) { 
                        for (var i in aps) {
                            for (var j in aps[i]) {
                                wifiAPs[aps[i][j]] = aps[i];
                                break;
                            }
                        }
                        resolve();
                    });
                }).then(function () {

                    // setup the WiFi access point
                    var SSID = 'sureshkm',
                        pwd = 'password';

                    wifi.createAP(SSID, pwd, 5, 'wpa_wpa2_psk', function(err) {
                        throwError(err);
                        clog('WiFi access point is successfully created.'); 
                    });

                    hsInit();
                }, function(err) {
                    throwError(new Error('Failed to collect the WiFi APs'));
                });
            }
        });
    });
}

// save the program into the micro controller 
save();

---

Posted at 2018-02-02 by @gfwilliams

I just tried this, and it works for me (after some messing around). I did have to manually use f.write and trigger it though since there was no HTML form - and it wouldn't have worked anyway because the password wasn't set.

I do get the disconnect error if I specify an incorrect IP address for the MQTT server though, so I've committed a fix for that which should go live in a few hours.

Looking at your code it seems that if you POSTed to anything that wasn't / you'd have trouble since you never write a response and close the connection unless it was. I guess that might trip you up if you shut down/reset the ESP8266 while that socket is still open.