WebSockets very, very slow when sending ~1800 characters

[espruino-discuss]

Posted at 2017-12-03 by MortenMoulder

I have about 150 LEDs in a strip. I want to change those colors, based on patterns I write externally (in a browser for example). I send the 2D array like this:

[[255, 255, 255], [255, 0, 0], [255, 255, 255]]

and so on from my NodeJS server running the "ws" module as a server. The ESP8266 NodeMCU then connects to the NodeJS server, and waits for it to send the RGB colors for all these LEDs.

The problem occurs when I want to send the RGB codes for all 150 LEDs at once. This takes 1000-1500 milliseconds and it simply queues them up. If I keep it running for 30-60 seconds, then disconnect my NodeJS server, it will keep changing the colors on the LED strip, because it keeps the colors in a queue (because it's too slow?) .

I am running this on a local network and here is my Espruino code:

var esp8266 = require("ESP8266");

function onInit() {
	var ssid = "SSID";
	var wifiOpts = {
		password: "PASSWORD"
	};
	var wifi = require("Wifi");
	wifi.connect(ssid, wifiOpts, function(err) {
		console.log("connected? err=", err, "info=", wifi.getIP());
		var ws = connect();

		ws.on('open', function() {
			console.log("Connected");
		});

		ws.on("message", function(msg) {
			msg = JSON.parse(msg);
			esp8266.neopixelWrite(NodeMCU.D2, msg);
		});
	});
}

function connect() {
	var WebSocket = require("ws");
	var host = "192.168.1.209";
	return new WebSocket(host, {
		path: '/',
		port: 8001,
		protocolVersion: 13,
		origin: 'Espruino',
		keepAlive: 60,
	});
}

save();

Any clue what could be wrong? Is 1800 characters simply too much?

[espruino-discuss]

Posted at 2017-12-03 by MortenMoulder

Okay, so I've opted for the "net" module instead as a socket server. It's much, much, much faster! I can easily do 50ms now without issues.

var esp8266 = require("ESP8266");

function onInit() {
  var ssid = "SSID";
  var wifiOpts = {
    password: "PASSWORD"
  };
  var wifi = require("Wifi");
  wifi.connect(ssid, wifiOpts, function(err) {
    console.log("connected? err=", err, "info=", wifi.getIP());
    require("net").connect({host: "192.168.1.209", port: 8001}, function(socket) {
      console.log("Connected");

      var arr = "";
      socket.on("data", function(data) {
        arr += data;
        if(arr.slice(-2) == "]]") {
          data = JSON.parse(arr);
          esp8266.neopixelWrite(NodeMCU.D2, data);
          arr = "";
        }
      });
    });
  });
}

save();

I had to hack it up a bit, because I send so many characters. I saw that the socket.on("data") got hit way too many times. Apparently it either receives or sends in chunks of ~260 characters, so I simply add those to a string, check if the last two characters are ]] (meaning my 2D array is complete) and then JSON.parse() it.

Works amazingly fast!

---

Posted at 2017-12-03 by Wilberforce

Each of your pixels is represented as 15 characters. You could make this shorter by:

a. Removing spaces
b. Converting to hex - e.g send FFEEDD, And send without a delimiter. You can send 40 or so of these in your 260 char packet.
c. Send as binary - perhaps use the Dataview classes

---

Posted at 2017-12-03 by MortenMoulder

The problem wasn't the data being sent, the problem was WebSockets being slow as hell. I can do this probably faster than 20ms now using raw TCP socket.

But yes, that's a good point.

---

Posted at 2017-12-04 by @gfwilliams

Yes, I'd just send as binary. I think most likely it's the masking/unmasking that happens for Websockets. That's done in JS with a repeated append at the moment, which is probably the reason for the slowness - but having to parse the JSON each time won't help either.

The masking could reasonably easily be rewritten using E.mapInPlace, a Uint32Array and finally E.toString - there was a forum post about this somewhere a while back but I think there was a similar result - the user just decided to use TCPIP instead - which is almost certainly the right answer if you control both ends of the system :)

---

Posted at 2017-12-04 by Wilberforce

This npm module seems to support turning off masking:

https://www.npmjs.com/package/ws-pure

---

Posted at 2017-12-04 by @gfwilliams

Neat - so if you used that on the server it'd probably be loads faster.