Puck.js Maximum Storage Space?

[espruino-discuss]

Posted at 2021-01-18 by MattB

I can't seem to store more than 40k of data in a file.

If I reset the Puck and do a.....
require("Storage").getFree();
.....I get.....
"=40960"

I'm trying to store movement and temperature data in a file. This works until the file gets to about 40k, then it fails.

Does this limit sound about right (using the latest firmware)? Am I missing something? Can I clear down the storage to get more space somehow?

[gfwilliams]

Posted at 2021-01-19 by @gfwilliams

Yes, that's right. There's 512k on the chip, but with the Bluetooth stack, bootloader, and then Espruino there's only 40k left.

Potentially it would be possible to do a different build of the Puck.js firmware that stripped out features that you're not using (graphics, crypto, networking) and then freed that up as space to allow you to save data.

How much space do you need?

It's worth saying that:

• If you store data as binary you'll be able to pack data in a lot more densely
• You can also compress with require("heatshrink").compress

What I'd suggest is you store the data in a Uint8Array/DataView in RAM until you get to a certain size (lets say 5k), then you just write it to Storage with require("Storage").write("data1", require("heatshrink").compress(data))

That should get you a pretty decent amount of data stored

[espruino-discuss]

Posted at 2021-01-19 by @allObjects

...stick something like that into your PuckJS: https://www.cypress.com/products/f-ram-nonvolatile-ferroelectric-ram --- in particular: https://www.cypress.com/search/psg/115241#/?_facetShow=ss_ppart_family,ss_pinterface,fs_pdensity_kb_,ss_porganization_x_x_y_,ss_ppackage,ss_pfrequency_mhz_,fs_pspeed_ns_,ss_ptemp_classification,fs_pmin_operating_temp_c_,fs_pmax_operating_temp_c_,fs_pmin_operating_voltage_v_,fs_pmax_operating_voltage_v_,fs_pmin_operating_vccq_v_,fs_pmax_operating_vccq_v_,ss_ptape_reel,ss_pautomotive_qualified,fs_part_price&ss_porganization_x_x_y_=512Kb%20x%208&ss_porganization_x_x_y_=1Mb%20x%208&ss_ppackage=SOP&ss_ppart_family=Serial%20FRAM&ss_pinterface=SPI (price may not be justifiable... but there are smaller ones...). You also can go with plain serial flash...

[espruino-discuss]

Posted at 2021-01-19 by @fanoush

It is because of this line https://github.com/espruino/Espruino/blob/master/boards/PUCKJS.py#L71 , when disabling all optional stuff you can probably double that. With just bluetooth and graphics libraries enabled (see few lines above) I have this set to 30 e.g. in build for P8 smartwatch.

If you still need much more maybe you can put SPI flash chip inside? They are cheap and small. Not sure how much space is there but thanks to banglejs the SPI flash is now well supported as main storage so if there are spare 4 i/o pins it could work . Bare chip would do too or some small module like https://www.aliexpress.com/item/4001174857221.html can give you up to 16MB for < $2

[espruino-discuss]

Posted at 2021-01-19 by MattB

Thanks all, I'll try those suggestions.

@gfwilliams The plan is to use the device for logging temp and motion. It's going in the post at the moment for testing and 2-4 days seems to use about 17k of the 40k, so I am going to need to get more space if it's going to go on longer trips. I will try the compression you suggested. I suspect that will give me enough.

@allObjects That looks interesting. I'll put that down for version 2 of my tests :)

@fanoush I don't need most of the other functions so removing the others would work. Is there a guide somewhere on how to build my own firmware (ie once I have changed the file)?

[espruino-discuss]

Posted at 2021-01-19 by @allObjects

@mattb, when doing my initial work going down the same path of storing (way point and other) data, Storage module was not available yet... nor flash access with all the bells and whistle as now,... and especially for easy change/re-write of storage, it was all with the Original (first) Espruino Board... anyway much shorter on memory. The attempt ended for the same reason as yours: the (affordable and market available) FRAM storage at that time was quite limited - 32KB in my case, and also 5V dependent... this has changed a lot... but flash has changed a lot as well: 256-Kbit (32 K × 8) Serial (SPI) F-RAM - Ferroelecric RAM - SPI challenges.

In the quest for having more RAM available to the runtime, running code from from EEPROM was discussed... and the Espruino JS Interpreter Implementation presents itself quite well for that, since it interprets from the source. You can read about that at Running code off of an EEPROM. In this conversation you also find a concept implementation of. memory manager (with garbage collection) that was intended to be use with memory not needing erase... (nowhere written down is the next step tht was planned with this memory manager: named objects (Strings for at 1st), to have String Space externalized... usually not needed for fast calcs... but UI and human i/o related... Almost like a file system... not the first I wrote in my 'low level coding career'). But a said: even though we fight for individual bytes sometimes, for real storage the fight has to be big or go home.

[espruino-discuss]

Posted at 2021-01-20 by @fanoush

I don't need most of the other functions so removing the others would work. Is there a guide somewhere on how to build my own firmware (ie once I have changed the file)?

Yes, there is https://github.com/espruino/Espruino/blob/master/README_Building.md in root of the repository, there is even (a bit outdated) example for PUCKJS. It should be enough to run provision script for board name (that could even download the compiler?) and then use make line from puckjs example there make clean && DFU_UPDATE_BUILD=1 BOARD=PUCKJS RELEASE=1 make
If compiler is not downloaded automatically I'd suggest getting version 8-2019-q3-update from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads

[gfwilliams]

Posted at 2021-01-20 by @gfwilliams

I just added a tweaked firmware build for you (config file is at https://github.com/espruino/Espruino/blob/master/boards/PUCKJS_MINIMAL.py)

I've attached the firmware update zip here - it gives you 90kB Storage instead of the normal 40kB. Hope that helps!

Attachments:

• espruino_2v08.158_puckjs_minimal.zip

[espruino-discuss]

Posted at 2021-01-20 by MattB

@gfwilliams Brilliant. Thanks so much for that. I have a lot more space now.

I'm struggling with heatshrink though. So what I have so far is.......

var fileName = "log";
var f = require("Storage").open(fileName,"a")
var records = new Uint8Array(32);
var records = [1,2,3,4,5,6,7,8,9];
f.write(require("heatshrink").compress(records));

This creates a file, and I can open it. It's not readable though. I assume that it's a binary file. How do I go about getting this off the device and decompressing it (assume it is correct in the first place). I'd like to do the decompress off the device so I don't have to load the file into memory etc.

[gfwilliams]

Posted at 2021-01-21 by @gfwilliams

So what I have so far is.......

The problem is actually twofold here... First, because of the way StorageFile works, you can't currently use character code 0xFF in it, which rules out compression - but even if you could, you can't just add blobs of compressed data to each other (it's like adding the bytes of two JPEG files together and hoping for an image twice as wide :) ).

In the example I gave above, I write to Storage as a single file, which should work fine. So for your example, do something like this:

var fileName = "log";
var fileNumber = 0;
// work out the next available file number
require("Storage").list(fileName).forEach(f=>{
  var n = parseInt(f.substr(fileName.length));
  if (n>=fileNumber) fileNumber=n+1;
});
// here are your records - ideally this is a big array
var records = new Uint8Array(32);
records.set([1,2,3,4,5,6,7,8,9]);
// write the data
require("Storage").write(fileName+fileNumber, require("heatshrink").compress(records))

Also worth noting that the compression gets better the more data there is, so you really need to write bigger chunks to get the best out of it (hence my suggestion of 5k). This might be helpful for some examples of packing data: http://www.espruino.com/Data+Collection#ram-dataview

To read the data, all you need to do is iterate through the files decompressing and printing them:

require("Storage").list("log").forEach(f=>{
  print(btoa(require("heatshrink").decompress(require("Storage").read(f))))
});

In the example above I convert the data to base64 with btoa since it's binary, and when reading the data on the PC you can convert it back with atob.

You could also just send the compressed data, and then decompress it on the PC with https://github.com/espruino/EspruinoWebTools#heatshrinkjs - which would obviously improve your download speeds.

[espruino-discuss]

Posted at 2021-01-24 by MattB

@gfwilliams Ok I see, thanks again for that. The new firmware is helping a lot with space, but the code seems to be failing after a number of hours. I need to work on the code a bit and add your suggestions.