Memory efficient way of taking in data from an http request that needs to be written to EEPROM?

[espruino-discuss]

Posted at 2017-01-11 by DrAzzy

I'm working on a project with Espruino on ESP8266 (I'm going to have like 7 of them - these are pingpong light controllers; I finally got back into that one. This is for importing/exporting patterns), where data is stored on an EEPROM.

I need some way to receive data via url parameter, then convert that into a Uint8 array or something else I can write to the EEPROM. The size of the data is 128 bytes.

My original plan was to use JSON.stringify and base64 encode that, then send that to the Esp8266-running-Espruino as a parameter. Then it would base64 decode it, eval it, and write it, eg eeprom.write(offset,eval(atob(data))) where data is the base64-encoded data. But this is where the wheels fall off - I don't have enough memory to eval it, because of how much space the array takes up. Even if i were to preface the data with 'new Uint8Array', it would still choke on making the intermediary array.

Am I forgetting about some more memory efficient way to get data into Espruino from an http request?

Is there a more efficient format I could use to store/send it?

[gfwilliams]

Posted at 2017-01-11 by @gfwilliams

Actually there's no need to eval/stringify it at all.

atob/btoa will do exactly what you want... Just convert your raw data into a string on the PC (String.fromCharCode.apply(null, data)), and stick it through btoa.

Then to decode, call atob - it'll give you a string, but you can convert that straight to a Uint8Array with: E.toUint8Array(atob(encoded_data))

You might be able to do a little better (avoiding the new allocation for toUint8Array) but hopefully that'll be good enough for you. If you do it right you can make sure that encoded_data gets freed after atob but before toUint8Array

[espruino-discuss]

Posted at 2017-01-13 by DrAzzy

Thanks! That's the kind of thing I was hoping for - I'll try it and see how I get on with it.

Also, I'm just staring at that String.fromCharCode.apply(null,data) line... I do not understand why this works the way it does, or frankly how apply is supposed to work with return values at all (none of the js docs I've looked at seem to describe how apply() deals with the fact that he's got N return values to return, but can only return one thing - it seems like at least in the case of strings, they get concatenated, but I'd love to find real docs that describe this).

[gfwilliams]

Posted at 2017-01-16 by @gfwilliams

.apply isn't really like .map - it just calls the function once, but with the arguments in the array. Calling String.fromCharCode(32,33,34,35,36,...) just returns one string with all those characters in it.

It's actually pretty horrible to do in Espruino since it ends up creating a sparse array for all the arguments, but on the desktop it's a really handy shortcut ;)

[espruino-discuss]

Posted at 2017-01-17 by DrAzzy

The code to generate the string is generated on the Espruino too - an import and an export function. But it does seem to work:

leds.export = function (type,index) {
  console.log(type);
	try {
		var eep=eepromtype[type][0];
		var off=eepromtype[type][1]+eepromtype[type][2]*index;
      var lenh=(eepromtype[type][2]/2); //one of the types has length 128 bytes, too long to handle with one call to .apply(), but all are even, so dividing in half is safe. 
      console.log(off);
		var rv= String.fromCharCode.apply(null,(eep.read(off,lenh)));
      rv+= String.fromCharCode.apply(null,(eep.read(off+lenh,lenh)));
      return btoa(rv);
	} catch (err) {
      console.log(err);
		return "";
	}
	
};

leds.import = function (type,index,data) {
	try {
		var eep=eepromtype[type][0];
		var off=eepromtype[type][1]+eepromtype[type][2]*index;
		eep.write(off,E.toUint8Array(atob(data)));
		return 200;
	} catch (err) {
		return 400;
	}
};

Thanks!

And yeah, I was getting it confused with .map(), and had forgotten you could pass multiple arguments to String.fromCharCode(). Been writing too much C lately.

[gfwilliams]

Posted at 2017-01-17 by @gfwilliams

:) There's a function called E.toString() that's a more efficient version of String.fromCharCode.apply for Espruino - if you decide you need it.