Espruino JavaScript
Flat String creation failing where it worked reliably before #1286
Replies: 15 comments
-
|
Posted at 2019-01-30 by @gfwilliams
What did you change in that time? I didn't change the IDE and you haven't updated the firmware, so something else must have happened. Maybe you just added more code, which used up the available memory? You could try running |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-30 by Robin Is running I'll give it a try. . . |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-30 by Robin ----clarification for #2
I had re-flashed to 2.00, but that was ~ a month? ago and this worked say twenty times on that version until the start of this week. Thought, could the server changes made earlier this week:
Wed 2019.01.30 Well, that didn't work and I even tried adding that code line inside the constructor to run immediately before. From:
So, as I see it, I have 1491 blocks available, rounding down for easier math 1400 1400 x 16 = 22,400 bytes available I'm requesting flat strings of 256 bytes and five of 180 bytes. Even with some allocation overhead, there should be scads of memory remaining, isn't this the case? |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-30 by Robin Wed 2019.01.30 Now peppered with the actual address as seen by allocation: stackEndAddress: 536958216 - actual strg: 536955892 = 2324 Free: 1375 - 1361 = 14 14 blocks * 16 = 224 bytes used strg: 536955892 - strd: 536954404 = 1488 There should be 93 blocks of stack remaining 93 == 1488 / 16 bytes per JsVar String strr isn't created as it appears to run into/off the end of the stack. So, either; using |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-30 by Robin Wed 2019.01.30 after four hours of number crunching, I thought I had a possible explanation until:
Ripping out code functions and performing the minimum initialization did allow the flat strings to be created. But what doesn't make sense is the available free blocks needed above and beyond just creating an instance. Just connecting and Load modules only usage 1910 x 16 = 30560 bytes which seems right considering non-minimized ASCII file size is around 48k for modules only From:
But why is an instance of a class with just arrays of 1156 bytes total and a handful of number vars create a usage of 19776 bytes? 19776 == 1236 * 16 This implies the entire class is being loaded yet again, even after the initial module(s) is/are loaded during the initial 'send' Then I create an instance of a class after instance creation 1236 x 16 = 19776 But, when I create an instance, in the above case, no more free space is used. Which doesn't make sense.
then is size here the same as the number of blocks returned when depth == 0 ? No, doesn't look like it as, when I apply a depth of 1 the sum of the returned sizes doesn't equal the number of blocks returned at depth 0 But that doesn't add up either, as a number var size is 2, but the internals page indicates the smallest size is 16 bytes per block. Sum of sizes is ten times ~== 10940 None of these methods agree. These observations draw me to conclude that
|
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-31 by @gfwilliams
Are you uploading the exact same code you did with the exact same firmware when it worked? The compiler shouldn't be an issue - it was a config problem on the server but the compiler itself didn't change. If you were to 'save on send' to 'direct to flash' that would give you a bunch more available memory. If you'd had that on before, and had changed it back, that would have broken things. I should add that it's not just that saving to flash gives you more memory, but it also doesn't save the function declarations to the command history, which means there's far less fragmentation.
Espruino doesn't allocate variables on the stack - they're all in one chunk of statically allocated memory.
This scans the whole variable and everything it links (like if you ran When you go down a level to output data, you might have two functions that are defined in the same scope. Both of those functions will bring in the same scope so will be more or less the same (huge) size.
Yes - because the act of sending (which is creating loads of functions and even executing them) can fragment memory.
It's most likely fragmentation - especially as you're using over 2/3 of your memory. The quick explanation is: In both cases there's the same amount of free memory, and the same amount used. This is one of the reasons I suggest that you just leave Espruino to do its own thing. If you created arrays the normal way: Espruino would automatically allocate flat strings for all those arrays, except if it couldn't find a big enough block of memory - in which case it would just allocate a normal String and continue perfectly fine, just a little bit slower. Of course if you are trying to write C code to access the raw data as a char array then you'll hit problems - but if you're using compiled JS or just normal JS you'll be fine. |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-01-31 by @allObjects @robin, in an object-oriented / memory managed / garbage collection enabled system calculations aren't that straight forward... also while an expression is executed, reference (vs. old way allocated) amount of memory can be a multiple of what the statically calculated demand is implying. Depending on the object granularity, the meta data / organizing data may outweigh even all the time the amount of actual user / payload data. A VM interpreting on source is even more susceptive to these challenges than a VM executing byte code. Byte code - like compiled code - is much more machine oriented than the high-level programming languages that we use these days. At one time, COBOL was thought to be high-level, compared to today's oo JS, it is gold-plated assembler at best. Reading beyond the code lines you publish, I assume you want to keep multiple frames of neopixel display (string) data. If it is all dynamic and constantly changing, you have to have it in some fast/easy to access memory. Did you ever think to put it into some serial RAM / FRAM / MRAM? The question is also what you use to manipulate these frames/buffers... If you want to manipulate them with the Espruino Graphics object, it has to be the Espruino's 'internal' memory... otherwise, it can easily be 'external'... as mentioned before: serial xRAM. Flash may work too, but may be cause challenges because of the write times and wear. It all depends... is always the right / correct answer... but only helpful to get the questioning / thought / design process going. |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by Robin Thr 2019.01.31 Thank you @gfwilliams for the clarifications and graphic.
Yes while adding small ~ten line functions saved one at a time. Was is happening, as array sizes are increasing, the code file could increase if I'm not removing an equivalent amount of bytes from that code file. I realize I'm near the max capability (without adding additional memory, a stock Espruino) of memory usage and trying to find the balance of what the max array size is, while leaving enough code play area for the end user. As I use Notepad++.exe with multiple tabs, bp and color, editing is much faster. A plausible condition that most likely occurred might be that as I had ~1000 blocks free, creating an instance and adding new code blocks in the console left pane shouldn't have given me any grief, which it wasn't. As it was late, it is quite possible that I hadn't re-tested the saved Notpad++ file on disk, by re-loading and re-sending. That was Sunday evening. When I attempted again on Monday, I was then loading a new file into a fresh IDE launch. My guess is that the new saved file was a fraction larger (how Windows/Notepad++ saves in blocks not bytes) than just copying the code functions into the IDE. So, when I attempted to use what I believed was the last working worked on code, the slightly larger file along with now larger arrays caused the low memory condition, which condition wasn't present during the end of the day, when it was working fine.
Along with seeing the compiler error that I observed Monday at the same time and noticed in the other thread post, that there was an indication of a server configuration issue, I drew the wrong conclusion that the cause was on the server side as I hadn't changed the code source, as I had just loaded it. From #6 wasn't quite answered 164 * 16 = 2624 bytes calculated vs Q: Does Espruino make an entire copy of the class functions (and called functions) when creating an instance using a constructor? Trace shows 1 block but free bytes drops by 4 JsVars. Is there another way to reveal this that I'm missing? (an array of ten small ints should fit into one block or one JsVar, correct?) |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by Robin Topic: Flat Strings Big mis-understanding here. I read (four months ago) 'New Espruino IDE Posted on Wed 14th, February 2018': Which caused me to post back in Oct From #2
and #3
and #2
The third example in #1 at: that and comments in other posts led me to believe that the technique shown was to use What is happening is that when arrays are small <8 elements, code execution (Javascript only) is acceptable. When creating larger arrays, >32 elements, code execution gets progressively and perceptibly slower (when using setInterval()) as array size increases. In order to access the largest 5 meter Neopixel strips will require array sizes of 900 elements. [60 neo/m * 5m * 3rgb] I have a design/speed reason(s) to stay away from |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by Robin Thr 2019.01.31 Observation: Memory not freeing up after call to I performed the following with over 1500 blocks of free memory. Which resulted in the MEMORY error and the consequence of having only 600 blocks remaining. Memory doesn't free up under this condition. |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by Robin Thr 2019.01.31 Thank you @allObjects for reminding me of the available Graphics object. I did revisit the online docs, but found it more suitable for fonts, polygons and X-Y position manipulation. My need is more random array offset intensive and requires a slightly different technique. It has been quite the challenge to put ten pounds of sh*t into a five pound can as I am attempting to maximize the array sizes to handle the five meter strings while still allowing for code space for the end user play area. A constant trade off of removing desired code for that which is absolutely needed. Still may have to abandon this project as the desire is to not have the user acquire additional memory hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by @gfwilliams
No, it links to it - and E.getSizeOf follows those links. To get the size of just the instance, subtract the E.getSizeOf of the class. Q: For clarification, a Uint is saved as ?? 16 byte JsVar e.g. Uint8ClampedArray(10) == ?? 16 byte JsVars? A Uint uses one 16 byte JsVar unless it's in a typed array, when it'll use whatever the size of that array is. Uint8ClampedArray(10) will take 3 JsVars (i think?), but then the data is stored after it... So |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-01 by @allObjects @robin, I don't get what is messing with you...
When you provide a depth parameter > 0 for E.getSizeOf(), you get JSON string back of object cluster, such as in the example output below. If you keep that in a variable, it eats up space as well until you assign something else to this variable. So no surprise you run out of memory... In "Memory evolution..." section towards the end of the console output you see what each of the components cost to establish. As you notice there are hidden things because the numbers do not straight add up. Nevertheless the figures are close enough to explain where and on what the emory is spent on. Since the memory tracking costs also memory, you have to adjust by it (mo - memory overhead - and you notice that it varies dependent on the length of the cmt comment assigned; cmt describes the activity that just happened to then show the increment in memory usage that activity did cost / consume in memory). Console output: |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-03 by Robin Sun 2019.02.03 Thank you @gfwilliams for the clarification in #13 That and @allObjects eGetSizeOfText.js should make things clearer.
Over thinking perhaps? When In the case when there isn't sufficient memory to create the output, as when I attempted to recurse a large class instance and at depth level two, although some level data was displayed, the function crapped out mid-way, finishing with the low memory error and cluttering memory. (maybe garbage collection doesn't/isn't occur?) I'm modifing eGetSizeOfText.js to see what else I can glean from under the hood. Thanks for cranking that out. |
Beta Was this translation helpful? Give feedback.
-
|
Posted at 2019-02-04 by @gfwilliams
I guess it's possible there is a 'lock leak' - it was designed as more of a debug function so may not have been tested that extensively for how it copes with low memory. Having said that I just looked at the code and I can't really see how that would happen. It's worth noting that |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Posted at 2019-01-30 by Robin
Wed 2019.01.30
Until a week ago, for the last three months, I had the following block of code inside a constructor creating several flat strings without fail.
Now, not all the strings get created.
I peppered the source with console.log statements and see three strings are created before the inevitable
Re-Reading:
I see that creating them takes time. There doesn't appear to be a solution to try, should that time duration block subsequent code execution.
I have tried reset(true), closing and re-opening the WebIDE and even powering down the Pico.
Is there a preferred way to somehow add a delay between each creation request and only advance after that request was successful?
Beta Was this translation helpful? Give feedback.
All reactions