get SHA1 hash

[espruino-discuss]

Posted at 2015-10-19 by JumJum

Just playing around a bit with websockets.
I wonder if there is a way to create a websocket server on Espruino.
Right now my problem is to calculate SHA1-hash from a given Sec-WebSocket-Key, to initiate a connection between client and server.
There are some sources available, but it would take away a lot of available memory.

[gfwilliams]

Posted at 2015-10-20 by @gfwilliams

There's a bug open for it here - while you could hack something up in JS you'd basically be creating your own web server from scratch.

It's much better if it's built into the Espruino firmware itself. I pasted some code there that does most of what's needed to enable WebSockets, minus the SHA1.

However, there's also a branch (AES) on GitHub which contains mbedtls, which contains a SHA1 implementation. At some point I'll expose that and then WebSockets can be enabled easily - but it's probably easiest to wait until that happens.

[espruino-discuss]

Posted at 2015-10-22 by sameh.hady

I am actually working on adding it to the ws module, yes it is a 1 big function and still trying to figure out how to save memory.

[gfwilliams]

Posted at 2015-10-22 by @gfwilliams

Do you actually need it for on the client side?

[espruino-discuss]

Posted at 2015-10-22 by sameh.hady

No, I thought I would extend the ws module and make it cover both the client and server.

So we can use it like:

/// Client
var WebSocket = require("ws");
 var ws = new WebSocket("HOST",{
      port: 8080,
      protocolVersion: 13,
      origin: 'Espruino',
      keepAlive: 60  // Ping Interval in seconds.
    });enter code here

/// Server
var WebSocketServer = require("ws").Server;
 var wss = new WebSocketServer({port: 8080});
 wss.on('connection', function(ws) {
  ws.on('message', function(message) {
    bla bla
  });

And thats why I wanted a way to only require the sha1 function if I am creating a server but for the client I really don't need it.

[gfwilliams]

Posted at 2015-10-22 by @gfwilliams

As I'd said above, it'd be much better if someone could actually just build this into Espruino, in C. It'd be faster and would use much less memory.

While you can do a server in JS, if you're planning on making it serve up webpages too (which you'll have to if using Pico+ESP8266 because you can only have one webserver at a time), you'd have to implement your own HTTP server in JS and not use Espruino's own one.

[gfwilliams]

Posted at 2015-10-22 by @gfwilliams

At the very least I'd create a new module called wsserver or something - so people who just want the client don't have to pull in a SHA1 implementation.

[espruino-discuss]

Posted at 2015-10-22 by sameh.hady

yup i guess that will be my only choice, since I am not good in C i am trying my best to implement it with js.

I guess for now we can have 2 modules ws & wss

[espruino-discuss]

Posted at 2015-10-23 by JumJum

@sameh.hady, may be we can share our experience. Base is the same, since my knowledge of C is poor. May be waiting for somebody adding a server to Espruino, as suggested by @gordon, would be a better way. But at least I learned a lot around datahandling with sockets ;-)
I've found an SHA1 implementation and added it to my testing environment around Espruino on ESP8266. You can find my actual source here:

• http://jumspruino.jumware.com/sources/ESP8266Test.js, see function startSocket and SHA-functions.
• http://jumspruino.jumware.com/sources/WebSocket.htm
  This is far away from being perfect, but it includes SHA1 and decoding/encoding for socket frame. Its tested in my environment with firefox and chrome. For connection I use the http-module from Espruino, which makes it easy to handle headers.
  Todo list, please have in mind, very early status:
• keep connection alive
• socket frame for data > 200 bytes
• problem with IE, which sends uppercase chars in upgrade header.
• testing, testing, testing
• .....

[espruino-discuss]

Posted at 2015-10-23 by sameh.hady

Nice work @jumjum, in my case I've used a longer Sha1 function which I couldn't include in ESP8266 because of its memory limit. So I couldnt test it actually till now.

For the keep alive all what you need is sending the ping message to the clients on interval and also pong pack if the client sent a ping.

In my ws client I found that the server was not sending me any pings and thats why I made the client keep the connections alive by sending ping requests to the server and wait for it's pong.

Is there any difference between net.createServer & http.createServer? I thought it is better to use the net one.

I will try your code when I am back home and will let you know

[gfwilliams]

Posted at 2015-10-23 by @gfwilliams

http.createServer creates an HTTP server, which parses the headers of the HTTP request for you, net.createServer is a plain TCP/IP one.

To be honest, if @jumjum's code can use the built in HTTP server that's awesome, and I take back everything I said about the JS implementation :) it might actually be the best way to do it if you can use the existing HTTP server with it.

[espruino-discuss]

Posted at 2015-10-23 by JumJum

@gordon is right, I used the HTTP server to get headers parsed.
BTW, is there an easy way to encode/decode socket frames ? Socket itself delivers raw data and I had to seperate the data part oit of it.
I don't know how node.js handles this, but can imagine that code for this already exists somewhere around http handling. (sorry lack of C-handling, tried to find and failed.
May be we could have an option to switch on/off decoding/encoding ?
To be honest, even this short sha1 takes a lot of memory, at least at runtime.
Had to use some "optimization" to get it running. And it takes some time to calculate.
Therefore I would prefer to have sha1 in firmware, same way we have sha224 and sha256.
And if there is a nicer way to do encode64, please give me a hint.

[gfwilliams]

Posted at 2015-10-23 by @gfwilliams

Wow, encode64 is impressive! You actually just needed atoa though ;)

[espruino-discuss]

Posted at 2015-10-23 by sameh.hady

in websockets you need to convert string to sha1 then to hex then base64 this hex
I was doing something like:

function hexEncode(str) {
    var x = [];
    for (var i = 0; i < str.length; i += 2) {
        x.push(String.fromCharCode("0x" + str.substr(i, 2)));
    }
    return btoa(x.join(""));
}

[gfwilliams]

Posted at 2015-10-23 by @gfwilliams

Instead of "0x" + str.substr(i, 2), you could try parseInt(str.substr(i, 2),16). Not much difference, but I guess it feels a bit more reliable :)

[espruino-discuss]

Posted at 2015-10-23 by sameh.hady

make sense :D everyday learning something new. still the sha1 function is dang big and in both ways will consume lots of memory. Funny part is I had an idea to make an http request from Espruino to a php page to get the encoded string directly. I guess this would consume way less memory :)

[espruino-discuss]

Posted at 2015-10-25 by sameh.hady

Actually I do think this option is the best one so far, I am into doing a POC to see how will it work

[espruino-discuss]

Posted at 2015-10-25 by @allObjects

Cool...embedded objects are better than sepeate to install plug-ins / extension - and they are safe as long as they are signed... I guess that - and the perforance in the early days - made them rarely appear. Since applets have full access to all resources and therefore have to prompt the user for permission, it is still something that meddles with the user's trust.

[espruino-discuss]

Posted at 2015-10-29 by d0773d

@gordon would something like https://libwebsockets.org/index.html work?

[gfwilliams]

Posted at 2015-10-29 by @gfwilliams

@d0773d I'm not sure that would really help... The majority of the work would just be getting WebSockets to work with a JS API - as all the header parsing and connection stuff is already part of Espruino.

As I posted right at the top or this thread, there's a bug open for support where I pasted the code that basically sets up the server side of this. All it needs is the SHA1 hash - which is part of mbedtls in the AES branch on GitHub. The whole thing can also be developed and tested under Linux.

It shouldn't be a lot of work, what it needs is for someone to write some C code - but it looks like that person might end up being me.

[espruino-discuss]

Posted at 2016-02-11 by Wilberforce

I've managed to get web sockets working using @jumjum 's js implementation from here:
http://forum.espruino.com/comments/12723204/

It would be great to get SHA1 function into the core.

I've attempted to build the crypto module for the esp8266, however get build errors (sorry can't recall what they were specifically)

I wondering if the hashlib library would be a good place to add the sha1 function, so then the ws.js could use this rather than rely on crypto?

[gfwilliams]

Posted at 2016-02-11 by @gfwilliams

Which board are you doing this on? The Pico would 'just work'.

It would be great to get SHA1 function into the core.

It is in the core though. You just need the crypto library to be built in.

I wondering if the hashlib library would be a good place to add the sha1 function

Seriously, just figure out why mbedtls library isn't building (I guess this is for ESP8266?) rather than just duplicating something that already exists and works fine.

I'm planning to remove hashlib at some point. Crypto is trying to be similar to CryptoJS so should be easy for people to get to grips with. mbedtls is great, and will be an easy source of loads more crypto functions if/when people want them as well.

Perhaps one issue is that you only want SHA1 and not AES/etc - but that could be tweaked with some ifdefs pretty easily.

[espruino-discuss]

Posted at 2016-02-12 by Wilberforce

I'm trying to build crypto for esp8266.

CC libs/crypto/jswrap_crypto.o
libs/crypto/jswrap_crypto.c: In function 'jswrap_crypto_error':
libs/crypto/jswrap_crypto.c:91:3: error: invalid initializer
   if (e) jsError(e);
   ^
libs/crypto/jswrap_crypto.c: At top level:JsVar

*jswrap_crypto_error_to_jsvar(int err) {
  const char *e = jswrap_crypto_error_to_str(err);
  if (e) return jsvNewFromString(e);
  return jsvVarPrintf("-0x%x", -err);
}

void jswrap_crypto_error(int err) {
  const char *e = jswrap_crypto_error_to_str(err);
>>>  if (e) jsError(e);
  else jsError("Unknown error: -0x%x", -err);
}

The issue appears to be the flash_str #define:

#define jsError(fmt, ...) do { \
    FLASH_STR(flash_str, fmt); \
    jsError_flash(flash_str, ##__VA_ARGS__); \
  } while(0)

root@esp8266-VirtualBox:~/Espruino# grep '#define FLASH_STR' src/*.h
src/jsutils.h:#define FLASH_STR(name, x) static char name[] __attribute__((section(".irom.literal"))) __attribute__((aligned(4))) = x

static char name[] vs const char *

Is there a simple way to resolve this?

I'm not sure the protocol here - should I raise as a github issue?

[gfwilliams]

Posted at 2016-02-15 by @gfwilliams

It looks like you have? it's probably best to discuss build-related stuff in there...

[espruino-discuss]

Posted at 2016-02-15 by Wilberforce

Yes - I did raise it on github -I thought I would document what I had discovered while it was fresh.

[espruino-discuss]

Posted at 2016-02-25 by Wilberforce

I've written a crypto module to use on boards without the crypto module in firmware - it implements the SHA1 used by the web sockets module. The ws module does not need to be modified.

For testing if the crypto.js file is copied down to a web ide project it can be tested.

The following code assumes the ESP8266 has a wifi.save and is connected to a network.

var page = '<!DOCTYPE html><html lang=en><head><meta charset=utf-8>';
page += ' <meta http-equiv=X-UA-Compatible content="IE=edge">';
page += '<meta name=viewport content="width=device-width,initial-scale=1">';
page += '<link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">';
page += '</head><body>';
page += '<div class="well well-lgt">';
page += '<div class="panel-heading"><h3>Websockets</h3></div>';
page += '<button class="btn btn-warning" id="send" onClick="ws.send(count);count=count+1;this.innerHTML=\'Send \'+count">Send to Espruino</button>';
page += '<ul id="list" class="list-group"></ul>';
page += '</div>';
page += '<script>';
page += 'var ws,count=1;setTimeout(function(){';
page += 'ws = new WebSocket("ws://" + location.host + "/my_websocket", "protocolOne");';
page += 'ws.onmessage = function (event) { console.log({msg:event.data});';
page += 'var ul = document.getElementById("list");';
page += 'var li = document.createElement("li");';
page += 'li.className="list-group-item";';
page += 'li.appendChild(document.createTextNode(event.data));';
page += 'ul.appendChild(li);';
page += '};';
page += 'setTimeout(function() { ws.send("Hello to Espruino!");ws.send(JSON.stringify({browser:95})); }, 1000);';
page += '},1000);</script></body></html>';
function onPageRequest(req, res) {
  res.writeHead(200, {
    'Content-Type' : 'text/html'
  });
  res.end(page);
}
var server = require('ws').createServer(onPageRequest);
server.listen(80);
server.on("websocket", function (ws) {
  ws.on('message', function (msg) {
    console.log({
      ws : msg
    });
    if (msg % 2 === 0)
      ws.send(msg + ' is multiple of two');
  });
  ws.send("Hello from Espruino!");
  ws.send(JSON.stringify({
      date : Date.now()
    }));

});

Start a browser at your Eps8266 address and see:

If you click the button, you can see the event appear in the Web IDE console. Every 2nd click causes the server to send back to the browser.... two way socket!

https://github.com/wilberforce/EspruinoDocs/blob/master/modules/crypto.js

For clarity, this the code that gets sent to the browser, using a CDN to get the bootstrap css:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width,initial-scale=1"><link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
</head>
   <body>
   	<div class="well well-lgt">
   		<div class="panel-heading"><h3>Websockets</h3>
   		</div>
   	    <button class="btn btn-warning" id="send" onClick="ws.send(count);count=count+1;this.innerHTML='Send '+count">
Send to Espruino</button>
   		<ul id="list" class="list-group">
   		</ul>
   	</div>
<script>
var ws, count = 1;
setTimeout(function () {
  ws = new WebSocket("ws://" + location.host + "/my_websocket", "protocolOne");
  ws.onmessage = function (event) {
    console.log({
      msg : event.data
    });
    var ul = document.getElementById("list");
    var li = document.createElement("li");
    li.className = "list-group-item";
    li.appendChild(document.createTextNode(event.data));
    ul.appendChild(li);
  };
  setTimeout(function () {
    ws.send("Hello to Espruino!");
    ws.send(JSON.stringify({
        browser : 95
      }));
  }, 1000);
</script>
   </body>
</html>

---

Attachments:

• crypto.js
• web-socket-esp8266.js

[gfwilliams]

Posted at 2016-02-25 by @gfwilliams

Nice - thanks for posting it up!

We just need to find a nice way of adding it such that it doesn't interfere with the in-built crypto library on other boards.

[espruino-discuss]

Posted at 2016-02-25 by Wilberforce

Sure.

I was hoping that a require("crypto") would look for a native install first, and then look in modules on espruino site.

Does it not work this way?

The other thing I tried and could quite pull off was using a require("path to github raw") and then attempted to add that to the modules cache, so that a later require("crypto") would find the module already loaded.

[gfwilliams]

Posted at 2016-02-25 by @gfwilliams

I was hoping that a require("crypto") would look for a native install first, and then look in modules on espruino site.

Yes - it should do this, but it needs a bit of tweaking on my side first as currently the list of internal modules reported by espruino.com isn't entirely accurate.

and then attempted to add that to the modules cache

You could probably do something really hacky by modifying the module cache directly:

global["\xFF"].modules["crypto"] =  global["\xFF"].modules["http://..."];

[espruino-discuss]

Posted at 2016-03-20 by user63214

Works great! Thanks!

[espruino-discuss]

Posted at 2016-03-28 by Wilberforce

I've managed to get the sha functions compiling on the esp8266 build, by disabling the aes functions. I'll get that up on github and do a pull request. Then this can be part of the esp8266 core, with little additional overhead on the current build...