Espruino JavaScript
strange behavior with RLE compression of flash new code #5130
Replies: 1 comment
-
|
Posted at 2015-10-31 by Sacha Hi Francesco, Gordon Same Problem here with a pico: Sacha Posted at 2015-10-31 by @gfwilliams @francesco which board was this on? @sacha did the code work before? Can you share the code that is giving the error? I'll try and look at this next week. There is a bit of a balancing act. Adding compression let me increase the amount of available RAM, but then if you have a lot of static data, you can't save it all. Probably what is needed is better compression Posted at 2015-11-01 by Sacha Hi Gordon I never tried the code below 1.80. Sacha Posted at 2015-11-02 by @gfwilliams Any thoughts on a compression algorithm that could be used? It would be nice to have something more or less standard that could be exposed for other bits of JS code, but things like gzip seem to have pretty use a lot of RAM/flash so don't seem suitable. Best contender right now seems to be https://github.com/atomicobject/heatshrink - but that's not even remotely standard. Posted at 2015-11-02 by tve Maybe this is a dumb idea, but it seems to me that being able to store code outside of the JsVars would be helpful. A reference/handle to the allocated memory can still be stored as a JsVar. It immediately saves 25%, if I understand the 12 chars per 16 bytes correctly. But beyond that, it would also allow code to be stored in flash, which would be a big help with larger modules from a library, in particular. Posted at 2015-11-02 by DrAzzy Don't we have something like that (storing data without the 25% overhead) with "flat strings"? This sounds really familiar... Maybe it wasn't ever applied to code though... I forget the details. Posted at 2015-11-03 by @gfwilliams Yes - flat strings are used for Typed Arrays, and store data with just a single 16 byte var at the start as overhead. They take a bit longer to allocate though as Espruino has to search through to find a contiguous block of variables - and you can't guarantee that it will. There's an issue open for using them to store function code - but we probably don't want to do it when we're creating functions inside functions as that might be too slow. I'd really like to avoid 're-packing' the data that's in RAM if at all possible - it just sounds like a recipe for disaster. Posted at 2015-11-03 by Francesco Currently the compression of the source code is done on ARM or with the IDE ? I think the compression of the source code during the flash operation should be optionally enabled/disabled (with a special instruction like echo/save) If you are looking for a good compromise between compression ratio and low memory usage you can consider the standard algorithm LZO which requires no memory buffer during the decompression process. Posted at 2015-11-03 by DrAzzy Are modules stored as flat strings? They probably should be... Maybe something like the "compiled"? function (x) { Posted at 2015-11-03 by @gfwilliams The compression is done on the ARM. The problem is that you have ~80kB of RAM and 48kB of flash. If you don't compress then what do you do? I'm not sure I really see what having the option to disable compression really gets you. LSO looks good, but the licence is GPL and so can't be included in the more liberal MPL licenced Espruino! I guess I could ask if he'd be willing to licence it differently. @drazzy modules are executed when they go into Espruino, so they're stored in the same way as everything else. Hopefully a tweak to store functions as flat strings would automatically include any that are in modules too. Posted at 2015-11-03 by Francesco If you are looking for a non GPLed algorithm you can also consider LZ4 as a candidate (more fast but using few memory during process) My perception about source code compression is that, on particular and huge scripts, this process renders more complex the search for a compromise between performance and memory management. I take as example a big script that I have completed recently that works without problems on an ARM STM32F103RCT6 and old firmware, but shows this warning (ERROR: Too big to save to flash) on latest version. Posted at 2015-11-03 by @gfwilliams So you're using something other than the Pico or Espruino board? You could potentially build your own firmware and allow more storage space in Flash memory for your code - then this wouldn't be an issue. The cause is probably not really the compression, but that I had to reduce the flash memory used to store code on the F103 boards (in order to give more to the Espruino firmware). Posted at 2015-11-03 by @gfwilliams Ok, please could you try this build: http://www.espruino.com/binaries/git/commits/a5688ca49ad5b0512fc513b4a7919b3b627486ec It's using LZ4 compression, and should hopefully manage to do a better job than simple RLE. edit: just wait a few minutes - it hasn't finished building yet Posted at 2015-11-03 by Francesco I have tryed this before 1v80 in this situation my script provide me about 236 vars as free memory this is very strange compression of the source code, in a binary blob, is done after calling save() function, right? I'm not able now to perform a complete test but I think is a proportional trend and this solution render useless my effort done to optimize the memory usage on my script build my own firmware it's a good idea (and my secret target :-) ) but for now I should to find more time for analyze and understand your software Posted at 2015-11-03 by @gfwilliams The compression happens after Maybe you could compare with a recent build that doesn't have RLE compression like this one Chances are it will be exactly the same as the LZ4 version. My guess is that the increased memory usage is because the new Espruino builds actually store line numbers for functions (which takes one extra variable per function) in order to give you proper debugging. If you use the latest Web IDE from GitHub I just added an option so you can turn that off if you want to, which should save you some memory.
That'd be great - and if you are using Espruino in your company's products, anything you can contribute back to help Espruino is appreciated. Posted at 2015-11-03 by Francesco As a company, we will publish all improvements made on your original source code, because we believe in open source philosophy and would work in compliance with it I'm happy that this problem isn't related to the compression features, unfortunately I'm using the Web IDE from chrome store. Posted at 2015-11-04 by Sacha Hi Gorden save(); ={ "free": 3040, "usage": 2060, "total": 5100, "history": 1270, There seems to be a lot of potential in the compressen method. Maybe there is even a better method in the future. Is there some documentation with hints how to reduce space ? I.e. use UintArray not strings etc. Many thanks Sacha Posted at 2015-11-04 by @gfwilliams Great! There's http://www.espruino.com/Performance that talks about memory. Also, newer builds should improve memory usage a lot more. Yesterday I made a few more changes that reduce memory usage in general. I may have to find a different compression library - LZ4 seems quite big, so I can no longer build Espruino for 128kB parts like the Olimexino :( Any other thoughts? There must be something out there... Speed is not an issue at all as it only happens once at startup. Posted at 2015-11-04 by Francesco @sacha a little Off Topic: Posted at 2015-11-04 by @gfwilliams @francesco on the Espruino Pico that's just the amount of variables that you get. There's more RAM, so more can be used for variables. Posted at 2015-11-04 by Francesco I'll explain, my question is: how do you calculate/estimate the amount of free space to use for variables? How changes this space when I choose what modules include/exclude from a build? Posted at 2015-11-04 by @gfwilliams You have to change the details in If you need help customising the build for your company's board I can always provide consultancy services. Posted at 2015-11-04 by Moray Is compression related to the odd experience I have of sending code once, executing save and being told too big for flash, then resending exactly the same code doing save and finding it is 5000 bytes slimmer and does fit on flash? Up till now I've assumed this is something wrong with the first time the IDE processes a local module, but it still seems very odd to me. Posted at 2015-11-05 by @gfwilliams I have no idea why there should be any difference between the times when code it sent - there really shouldn't be. If you have some code that you can reliably get that with, maybe you could send it over so I can check? Posted at 2015-11-05 by DrAzzy Maybe the issue Moray encountered could be some sort of minification failure? If it was sending the code unminified, that'd do it? If turning off minification reproduced the issue, that'd confirm it.... Posted at 2015-11-07 by Moray I've narrowed it down to the first time code is sent, and perhaps only when a local module is present. I suspect it's something to do with minification. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Posted at 2015-10-30 by Francesco
I see some strange behavior with a feature recently introduced: RLE compression
A complex and big (weight of js file: 19.6 KB) software that has been loaded and executed more times without problem on version 1v79, it returns an error with latest firmware (1v81):
ERROR: Too big to save to flash (32796 vs 30716 bytes)
however, after the flash, the script seems to be complete
Beta Was this translation helpful? Give feedback.
All reactions