Using Espruino Wifi in softAP+Station mode

[espruino-discuss]

Posted at 2017-04-27 by dave_irvine

Not sure if anyone else has tried this or had any luck, but the ESP8266 in the Espruino Wifi has built in functionality to simultaneously be an AP as well as a station/client.

The functionality isn't exposed in the Espruino libraries, so a little work is required to get it into the right mode, and I feel like I'm almost there, but something isn't quite right and I get the feeling the Espruino http library is to blame.

Try out the following code, and what should be able to happen is that if you connect to the AP that the Espruino makes, and visit http://192.168.4.1 (assuming it gets this IP, which in my testing it usually does), the Espruino should fetch a http resource. However, I seem to get stuck in some sort of loop

got http request
fetching page
got http request
fetching page
got http request
fetching page
got http request
fetching page
got http request
fetching page
Uncaught Error: No free sockets
at line 1 col 278
...row Error("No free sockets");

which to me indicates something odd is going on in the http lib.

Does anyone have any ideas?

const wifi = require("EspruinoWiFi");

const startDual = (callback) => {
  const at = wifi.at;

  at.cmd('AT+CWMODE=3\r\n', 1000, (cwm) => {
    if (cwm!="no change" && cwm!="OK" && cwm!="WIFI DISCONNECT")
      callback("CWMODE failed: "+(cwm?cwm:"Timeout"));
    else
      callback(null);
  });
};

const connect = (_apName, _apKey, callback) => {
  const apName = JSON.stringify(_apName);
  const apKey = JSON.stringify(_apKey);
  const at = wifi.at;
  
  at.cmd("AT+CWJAP="+apName+","+apKey+"\r\n", 20000, function cb(d) {
    if (["WIFI DISCONNECT","WIFI CONNECTED","WIFI GOT IP","+CWJAP:1"].indexOf(d)>=0) {
      return cb;
    }
    if (d!="OK") {
      setTimeout(callback,0,"WiFi connect failed: "+(d?d:"Timeout"));
    } else {
      setTimeout(callback,0,null);
    }
  });
};

const becomeAP = (ssid, options, callback) => {
  const at = wifi.at;
  options = options||{};
  
  if (!options.password || options.password.length<8) throw new Error("Password must be at least 8 characters");
  var enc = options.password?"3":"0"; // wpa2 or open
  if (options.authMode) {
    enc={
      "open":0,
      "wpa":2,
      "wpa2":3,
      "wpa_wpa2":4
    }[options.authMode];
    if (enc===undefined) throw new Error("Unknown authMode "+options.authMode);
  }
  if (options.channel===undefined) options.channel=5;

  at.cmd("AT+CWSAP="+JSON.stringify(ssid)+","+JSON.stringify(options.password)+","+options.channel+","+enc+"\r\n", 5000, function(cwm) {
      if (cwm!="OK") callback("CWSAP failed: "+(cwm?cwm:"Timeout"));
      else callback(null);        
    });
};

const startServer = () => {
  var http = require("http");
  
  //start a server
  http.createServer(function (req, res) {
    console.log('got http request');
    
    // break context in case something is weird.
    setTimeout(() => {
      console.log('fetching page');
      
      // fetch a resource
      http.get("http://www.pur3.co.uk/hello.txt", function(clientRes) {
        console.log("Response: ",clientRes);
        clientRes.on('data', function(d) {
          console.log("--->"+d);
        });
      });
    }, 1000);
  }).listen(80);
};

// Force a scan so that 'at' gets assigned
wifi.scan((err, result) => {
  if (err) { throw err; }
  
  startDual((err) => {
    if (err) { throw err; }
    
    // Connect to a real AP
    connect('apSSID', 'apPass', (err) => {
      if (err) { throw err; }
      
      console.log('wifi connected');
      
      // Become an AP
      becomeAP('newSSID', {
        password: 'newPass',
        authMode: 'wpa2',
      }, (err) => {
        if (err) { throw err; }
        
        console.log('ap started');
        
        // Fetch some resource to prove we can
        require("http").get("http://www.pur3.co.uk/hello.txt", function(res) {
          console.log("Response: ",res);
          res.on('data', function(d) {
            console.log("--->"+d);
          });
          res.on('end', () => {
            // start a http server
            startServer();
          });
        });
      });
    });
  });
});

[espruino-discuss]

Posted at 2017-04-27 by dave_irvine

Actually I'm sure its a bug in the http library. If I log the HTTP requests when they arrive, the first one is normal, and then the second one has this to say:

"method": "ntent-Type:",
"url": "text/plain",

so it looks like there is some corruption going on where the outgoing http request is getting intercepted by the http server and becoming an incoming http request!

---

Posted at 2017-04-27 by @gfwilliams

Is it possible that the library or some part of it could have been initialised twice somehow? That could explain things being messed up.

An HTTP bug is unlikely given it works fine in for all the other connection modes and devices. It's possible, but I'd try and rule out some other things first :)

Also, did you try just calling wifi.connect and wifi.startAP? The way the EspruinoWiFi library works, the wifiMode variable should be ORed such that if you call both functions then Espruino is put into CWMODE 3 - the softap/station mode you're after?

---

Posted at 2017-04-27 by dave_irvine

Ah I didn't realise that calling the two as per usual would do the ORing. Might be worth adding to the docs.

I'll move the http require to the top of the file tomorrow and simplify everything down using connect and startAP rather than my hacky workaround.

---

Posted at 2017-04-28 by @gfwilliams

Yes, updating the docs to make it more explicit would be handy!

I'd try a really minimal example and see if that works, and then work up to what you have. Hopefully it'll be easier to figure out whether it is an Espruino issue or something in your code

---

Posted at 2017-04-28 by dave_irvine

Alright so I simplified down and the same issue persists.

const wifi = require("EspruinoWiFi");
const http = require('http');

const startServer = () => {
  http.createServer(function (req, res) {
    console.log('got http request');
    console.log(req);
    
    getPage((data) => {
      console.log('got http response');
      console.log(data);
    });
  }).listen(80);
  console.log('http server started');
};

const getPage = (callback) => {
  http.get("http://www.pur3.co.uk/hello.txt", function(res) {
    let data = '';
    console.log("Response: ",res);

    res.on('data', function(d) {
      data += d;
    });

    res.on('end', () => {
      callback(data);
    });
  });
};

wifi.connect('realAP', {
  password: 'realPW'
}, (err) => {
  if (err) { throw err; }

  console.log('wifi connected');

  wifi.startAP('ourAP', {
    password: 'ourPW',
    authMode: 'wpa2',
  }, (err) => {
    if (err) { throw err; }

    console.log('ap started');

    getPage((data) => {
      console.log(data);
      startServer();
    });
  });
});

When I log the incoming http requests I'm definitely seeing corrupt data so I'm fairly sure the http.get is somehow looping back onto the softAP rather than going out via the station.

"method": "b390b887c80"",
"url": "\nAccept-Ranges:",

---

Posted at 2017-04-28 by dave_irvine

Not to complicate matters but I was also seeing similar behaviour when I wired up a second ESP8266 module to my Espruino WiFi and tried to do the same thing by setting one ESP module to be an AP and the other one to be a station. It seems like the http module has no way of knowing which "network" to use, and I couldn't find a way of telling it explicitly.

---

Posted at 2017-04-28 by @gfwilliams

In the case of two WiFi modules together, the http lib definitely doesn't know which one to use - it's not designed to handle that at all.

However when you have AP and client on the same ESP8266 it's different, since the ESP8266 doesn't differentiate between connections.

For starters, you're never sending any HTTP response for the server, so every request gets left open and it's hardly surprising you have issues.

The simple HTTP server example from the Internet page works fine:

const startServer = () => {
  http.createServer(function (req, res) {
    res.writeHead(200);
    res.end("Hello World");
  }).listen(80);
  console.log('http server started');
};

And even after sending a response, your code appears to work for me:

const startServer = () => {
  http.createServer(function (req, res) {
    console.log('got http request');
    console.log(req);
    res.end("Hello world");
    getPage((data) => {
      console.log('got http response');
      console.log(data);
    });
  }).listen(80);
  console.log('http server started');
};

Although occasionally I don't get a response, but then I do have WiFi that keeps dropping out here.

Also worth noting that const and let act just like var in Espruino, so may not work as you expect. They're not part of ES5, but I've had to put placeholders in because everyone kept using them :)

---

Posted at 2017-04-28 by @gfwilliams

Also, since you don't differentiate between pages, when you access 192.168.4.1 you're going to get two requests in quick succession - for / and /favicon.ico - which will fire off two HTTP requests and once and might get confusing.

---

Posted at 2017-04-28 by dave_irvine

OK I was trying to keep the example super simple which is why I wasn't responding to the incoming http requests.

Here is a more in-depth example that I can't get to work.

Flow should be like this:

1. Connect Espruino to real AP
2. Espruino becomes an AP
3. Espruino starts HTTP server
4. Connect device to Espruino, browse to 192.168.4.1
5. Espruino gets http request, then fetches an internet resource via the real AP.
6. Espruino takes the response from the internet resource and sends it back to the device that made the http request.

const wifi = require("EspruinoWiFi");
const http = require('http');

const startServer = () => {
  http.createServer(function (req, res) {
    console.log('got http request');
    console.log(req);
    
    if (req.url === '/') {
      res.writeHead(200);

      getPage((data) => {
        console.log('got http response');
        console.log(data);
        console.log('send http response back to requester');
        res.end(data);
      });
    }
    
  }).listen(80);
  console.log('http server started');
};

const getPage = (callback) => {
  console.log('get page');
  
  http.get("http://www.pur3.co.uk/hello.txt", function(res) {
    let data = '';
    console.log("Response: ", res);
    
    res.on('data', function(d) {
      data += d;
    });
    
    res.on('end', () => {
      console.log('get request ended');
      callback(data);
    });
  });
};

wifi.connect('realAP', {
  password: 'realPW'
}, (err) => {
  if (err) { throw err; }
  console.log('wifi connected');
  wifi.startAP('ourAP', {
    password: 'ourPW',
    authMode: 'wpa2',
  }, (err) => {
    if (err) { throw err; }
    console.log('ap started');
    startServer();
  });
});

---

Posted at 2017-04-28 by @gfwilliams

But you can get it to work just fine in AP+Client mode if you use the code I gave?

Your code is still buggy. When the browser requests /favicon.ico then the if (req.url === '/') { check fails and the HTTP response is left open.

---

Posted at 2017-04-28 by dave_irvine

Yes with your example its fine, the problem lies when you try to hold the incoming http request open while you fetch a resource.

---

Posted at 2017-04-28 by @gfwilliams

Ok, so it doesn't have anything to do with the AP/Client thing, but is literally just a client request when a server is open?

It's taken me a while to find it, but it looks like it's to do with the ESP8266 AT command driver. Basically it does the GET request and everything's fine, but then it gets a socket closed command. At that point the socket is closed but there's still data, and the server thinks that means that a socket has opened - so part of the GET gets treated as another request to the server.

So yeah, it is something broken in Espruino. I'll see if I can find a nice fix for it

---

Posted at 2017-04-28 by dave_irvine

Ah I suppose I had never thought of the simpler case where the Espruino is just in station mode and runs a http server, I was too caught up in my own example. Lets hope that it is just the problem that you have found and there isn't another layered issue when I get back to using the dual-mode.

Thanks very much for looking into the issue. Hopefully you can find a fix soon!

---

Posted at 2017-04-28 by @gfwilliams

Ok, just fixed it, there was another problem too which required a more heavy workaround.

Note that your code above will probably still break because you're not calling res.end all the time, but something like this works fine now:

function startServer() {
  http.createServer(function (sreq, sres) {
    console.log('got http request',sreq.url);
    sres.writeHead(200);
    http.get("http://www.pur3.co.uk/hello.txt", function(res) {
      var data = '';
      console.log("Response");
      res.on('data', function(d) { data += d; });
      res.on('end', function() {
        console.log('get request ended');
        sres.end(data);
      });
    });
  }).listen(80);
  console.log('http server started');
}

(I pulled out all the ES6 stuff just in case that happened to be a potential issue, but I'm pretty sure it's not)

---

Posted at 2017-04-28 by dave_irvine

Does this mean I need to flash a different firmware build to my Espruino or do you mean that changing the JS code around so that the http.get happens directly inside the request handler has made a difference (very odd if that is the case!)

---

Posted at 2017-04-28 by @gfwilliams

Ahh, no - it was all in the EspruinoWiFi JS module, which has now been updated online.

Just re-upload your code, the new module will get pulled in automatically, and it should work a lot better.

---

Posted at 2017-04-28 by dave_irvine

Oh cool, I'll give it a go. Can you show me what you changed? (just interested). I had a look on Github but it doesn't seem like the modules are stored there.

---

Posted at 2017-04-28 by @gfwilliams

They are on GitHub, just in a different repository to Espruino. The changes are here: https://github.com/espruino/EspruinoDocs/commits/master/devices/EspruinoWiFi.js

Two main ones you'd be interested in:

• Fix the unsent data -> accepted socket bug: espruino/EspruinoDocs@b0db655
• Fix the socket opening race condition by accepting a socket when it's reported as open, instead of when data arrives: espruino/EspruinoDocs@fd41d9b

At some point soon I plan to roll this all into a C file which'll get included in the firmware (which should save a bunch of RAM) but in the mean time the JS module is a bit of a mess because during development the ESP8266 module's protocol kept changing every few weeks! :)

---

Posted at 2017-04-28 by dave_irvine

Let me know if some sort of Patreon sponsorship would help get the WiFi stuff into C any faster, I'm short on RAM as it is as I need to support TLS, and also drive a SSD1306 based display, with images.

Thanks again for your help.

---

Posted at 2017-04-28 by @gfwilliams

Thanks! Yes, absolutely - I'm trying to prioritise things that come up from Patreon backers and those who bought proper Espruino boards, so if you're both that's awesome :)

If you need to save RAM I'd definitely suggest trying:

• Turn on simple Esprima minification in the IDE
• Turn 'Save on Send' to 'Yes'

Save on send will save your code into flash memory, and then any function that's defined there will have its source code executed straight from flash - so won't be using up any RAM. With minification it should save you quite a lot of memory.

---

Posted at 2017-04-29 by dave_irvine

Thanks for the tips Gordon. I've signed up on Patreon, hopefully this helps you find the time to port the WiFi code to C soon :)

---

Posted at 2017-05-01 by dave_irvine

Gordon, unfortunately Esprima minification is causing me some real issues, sometimes modules that I want it to load from disk just won't load, then the next time I try to flash they will load, and the time after they will be gone again.

For now I've turned off all minification, but now I've come up against a memory limit while trying to flash.

If I don't require the final module I'm trying to flash, I get the following output:

Erasing Flash...
Writing...................................
Compressed 114368 bytes to 33031
Checking...
Done!
Loading 33031 bytes from flash...

which indicates to me I'm using 33kb of flash, well below the 512kb flash or even the 128kb ram that the Espruino WiFi has.

When I require my final module, which has a size on disk of 16kb, I then get

ERROR: Out of Memory!
Uncaught SyntaxError: Got UNFINISHED STRING expected EOF
at line 1 col 15
E.setBootCode("Modules.removeAllCached();\nModules.addCached...

but unless my maths is wrong, the 33kb it reports before, + 16kb is only 49kb, still well below 128kb.

I have save-on-send switched on, and I've tried "Modules uploaded as Functions (BETA)" both on and off.

Is there something I'm missing?

---

Posted at 2017-05-02 by @gfwilliams

Ahh - it's not quite as simple as being able to use all of the 128kB of RAM. There's some info here that might help: http://www.espruino.com/Performance

Also, a lot of the flash is used up by the interpreter itself - I think there's around 64kB available for code on the WiFi board.

You could try reset() on the left-hand side before the upload - it might help.

The issue is that right now, to save to flash, you have to fit all that data into RAM first - not only that but it's uploaded as a quoted string, and then has to be interpreted into a normal string - so you need twice as much RAM to upload as you'd expect.

Hopefully at some point I'll be able to come up with a workaround (uploading code in chunks), but for now, with E.setBootCode you're a little limited by the volume of code you can upload. However, once that is uploaded (especially with 'Modules uploaded as Functions') you should get much better RAM usage.

Thanks for the Patreon support - I was looking at the compiled in ESP8266 support over the weekend and have made quite a lot of progress with it. Hopefully if things go well I should have a beta version in a few weeks.