Help needed on trace() to locate user var reference and contents

[espruino-discuss]

Posted at 2020-04-19 by Robin

Sun 2020.04.19

I need to monitor the number of blocks a user var requires in memory. I need to extract or view
the contents of that var, using the reference that trace provides.

Using trace,

http://www.espruino.com/Reference#l__global_trace

I'm able to view it's ID and space utilized: #348 and #531

     #348[r1,l2] Name String [1 blocks] "\xFFcod"        #531[r1,l0] FlatString [137 blocks] "console.log( \"    \" );\n  console.log( \"JsVars  Last:\

What I am after is how to get the value #531 and it's contents. It takes up 137 blocks.

Pairing down, now that I know the ID \xFFcod of the var

>trace(["\xFFcod"]);
#990[r0,l1] Array(1) [
  #986[r1,l2] Name Integer 0    #989[r1,l0] String [1 blocks] "\xFFcod"
]

But a curious response emerges, the reference value changes from #348 to #990, but it is the same location in memory?

Q1: Why does the refrence value change and what is that value exactly? Q2: Wht is the meaning of 'r0' and 'l1' and compared to 'r1' and 'l2' ?

Using getSizeOf()

>E.getSizeOf(["\xFFcod"], 2)
=[
  {
    name: "0",
    size: 2 }
 ]
>

>E.getSizeOf(["\xFFcod"], 1)
=[
  {
    name: "0",
    size: 2 }
 ]

>E.getSizeOf(["\xFFcod"], 0)
=3
> 

Q3: What is the 'name' attribute refereing to? Q4: What are the return values here as they don't seem to be the actual blocks as returned in a trace request? Q5: How to get the value #531 and it's contents using trace() ?

Thank you in advance. . . .

[gfwilliams]

Posted at 2020-04-20 by @gfwilliams

Q1: Why does the refrence value change and what is that value exactly?

It's because Espruino uses fixed-size blocks - it's the block number: http://www.espruino.com/Internals#variable-storage

Q2: Wht is the meaning of 'r0' and 'l1' and compared to 'r1' and 'l2' ?

Espruino's vars are reference counted for quick garbage collection. R = number of references (from other JS vars), L = number of locks (eg. bits of Espruino internal code that have a pointer to it).

Q3: What is the 'name' attribute refereing to?

The name of the field inside the object you just inspected.

"\xFFcod" is the name of the attribute of a function that contains a function's code. You can probably just search in your code for console.log( \"JsVars Last:\ to figure out which function it is.

Q4: What are the return values here as they don't seem to be the actual blocks as returned in a trace request?

However ["\xFFcod"] is just an array containing the String "\xFFcod" which is why E.getSizeOf isn't returning what you expect. You need to do whaverFunctionYouTraced["\xFFcod"] instead

Q5: How to get the value #531 and it's contents using trace() ?

Look down the hierarchy of the trace (eg, the lines above) to figure out what the function was, then you can write the code to reference it. If you paste the whole trace up I could show you.

But the issue here is you've got a very large function that's been uploaded. As above just searching your code for console.log( \"JsVars Last:\ will likely tell you what function that is.

[espruino-discuss]

Posted at 2020-04-22 by Robin

Tue 2020.04.21

In three parts as each too large for one forum post.

Thank you @gfwilliams for the identification of References and Locks. After looking up several web pages, I thought I'd surprise myself, with a solution on successful traversing the trace object with code. Well it did indeed become a surprise, but also a big disappointment.

This code file is an attempt to generate the errors under my control and is not the code file that has issues. It reproduces close enough, that I am able to create working functions that I hope are small enough to paste into the other module to locate it's issues.

'"\xFFcod" is the name of the attribute of a function that contains a function's code'

I would expect that each name be different, should the attribute be specific to a function. In the attached trace, there is #110 #74 #437 #358 #328 etc. which so far point to both strings and flat strings, so as I see it, isn't specific as in one to one with a particular function, specific to Strings or to FlatStrings or an object type.

While I understand the gist of your response indicated to use the information gathered and then peruse the source code file to see the original source. I get that, but what is needed is to inspect the block as referenced by the block number, such as #531 while in memory. I'm attempting to build small functions to perform that task inside the WebIDE during run time, and not by human visual means.

re: Q1: Why does the refrence value change and what is that value exactly?
A: It's because Espruino uses fixed-size blocks - it's the block number

I understood that, what I should have clarified is how do I get to #531 when I am able to locate in the trace block #348 using text: '\xFFcod' which I have now identified isn't a good technique either as there are many instances of '\xFFcod'

   #348[r1,l2] Name String [1 blocks] "\xFFcod"        #531

But that only works on completion of first upload, as performing a subsequent trace on that block by name results in a unique new block number.

Such as:

>trace("history")
  #949[r0,l1] String [1 blocks] "history"
=undefined
>trace("#949")
  #1272[r0,l1] String [1 blocks] "#949"
=undefined
> 

My guess is that the act of typing in the command is assigned a new block as it is uploaded such that Espruino is then knows what to do, e.g. modify the linked list after parsing and deciding if in fact it is a valid block.

Still searching for a better example, but couldn't find one in the haste to post this.

[espruino-discuss]

Posted at 2020-04-22 by Robin

Part 2

But therein lies the problem. How to get at the block (#438) that contains what we are after when we only have reference to the String name (#428) that is the actual text identifying it's function.

trace("dl") will locate it's referenced block #438, but I need what #438 points to, that is it's content, in this case block #428 the function itself, during runtime.

In our cleaned up version, it is block #438 after locating #428

  #428[r1,l2] Name String [1 blocks] "dl"    #438[r1,l0] Function {
      #437[r1,l2] Name String [1 blocks] "\xFFcod"        #637[r1,l0] FlatString [150 blocks] "console.log( \"    \" );\n  console.log( \"JsVars  Last:\" );\n  console.log( \"  Free:   lastFree \" + lastFree );\n  console.log( \"  Usage:  lastUsage \" + lastUsage );\n  console.log( \"  History: 

*** very large chunk removed for brevity ***

lastHistory - gmh() ) + \"    last: \" + lastHistory + \"    cur: \" + gmh() );\n  console.log( \"    \" );\n  console.log( \"    \" );\n  console.log( process.memory() );\n\n\n  lastFree = gmf();\n  lastUsage = gmu();\n  lastHistory = gmh();"
      #434[r1,l2] Name String [1 blocks] "\xFFlin"= int 114
    }

Using:

http://www.espruino.com/Reference#l__global_trace
http://www.espruino.com/Reference#l__global_global

I learned that the global variable is just a Javascript Object after:

>typeof global
="object"
> 

then creating an array that I can manipulate and assigning the contents of the global variable to the array object just created:

>var aaa=[]
=[  ]
>aaa=global
={
  Graphics: function () { [native code] },
  g: Graphics: {
    flip: function () { [native code] }
   },
  testmem: {  },
  tm: function () { ... },

. . .

etc

Using properties of that object, I am able to extract the data for a var I defined:

>var aaa=[]
=[  ]
>aaa=global

>aaa.num_str_vars
=2

That checks out and as expected.

> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/entries

Attempting to iterate using the entries method is fraught with errors:

>for (let [key, value] of Object.entries(aaa)) {
:  console.log(`${key}: ${value}`);
:}
Uncaught SyntaxError: Got '[' expected ';'
 at line 1 col 10
for (let [key, value] of Object.entries(aaa)) {
         ^


for ( let key of Object.entries(aaa)) {
}
Uncaught Error: Function "entries" not found!
 at line 1 col 25
for ( let key of Object.entries(aaa)) {

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach

global.forEach(element => console.log(element));

Uncaught Error: Function "forEach" not found!
 at line 1 col 8
global.forEach(element => console.log(element));
       ^

Although I was able to iterate over the history array using just indexers into the array, the rest has become the big surprise, and not what I thought I could get done.

For the curious onlookers:

Please heed the code file internal Warning and read everything before uploading

WARNING - Not for beginners or the impatient. Intended for individuals with intermediate to advanced skills. Will lock up the device it is installed on.

Better yet, beginners please don't upload at all!

Attachments:

• trace20200421.txt
• codeTestMemoryUsage20200421.js

[espruino-discuss]

Posted at 2020-04-22 by Robin

Part 3

Just noticed this, that wasn't apparent as I typed the posts. Traversing backwards, I see the large code chunk is part of a known function named 'dl()' In the history, which I believe is what is uploaded before parsing as block #440 and then again as the string identifier in memory at block #428 that points to the function configured in memory as block #438, made of other linked blocks, it's ID of #437 and the function as a FlatString object at #637

     #440[r1,l0] String [200 blocks] "function dl() {\n\n  console.log( \"    \" );


  #428[r1,l2] Name String [1 blocks] "dl"    #438[r1,l0] Function {
      #437[r1,l2] Name String [1 blocks] "\xFFcod"        #637[r1,l0] FlatString [150 blocks] "console.log( \"    \" );

What I would like to do is be able to traverse the trace results and get at block #637 but not knowing the actual function by name. Maybe a traverse the results and list the blocks containing more than 'x' (?ten?) blocks might be a good start.

As typeof trace() returns ="undefined" I'm starting to wonder if it is in fact just a function that prints to the console, values that are extracted under some rules, as I expected "object" to be returned.

> 'If you paste the whole trace up I could show you'

So, yes please, a small sample to get going would be appreciated.

Incidentally, I'm initially guessing there is a parsing issue when using both embedded individual commented lines and block commented lines within a function whose length spans a large amount forcing a newline char to get inserted which then affects code content parsing during runtime. Just a guess, but ran into a similar situation back in 2017 I just can't reproduce that manually, yet.

What happens in the suspect code file, is an attempt to execute a function roughly 20 JsVars in size when 400 JsVars are available and I get an immediate error/lockup and unable to access the console. Using the above idea, will attempt subsequent test passes inserting/moving this trace read function each time to see what is going on immediately before lockup. This code file pseudo representation is extremely stable until attempting to write a large block knowing there isn't enough memory. Thus, a close approximation to what occurs in the other file.

Also, found this unproven snippet: global["\xFF"].history = []; that was a suggestion to reduce memory, but felt that performing intentional destruction without allowing Espruino to clean up the linked tables, that it is in fact an inproper suggestion.

[gfwilliams]

Posted at 2020-04-22 by @gfwilliams

I would expect that each name be different, should the attribute be specific to a function.

I don't think you're understanding - \xFFcod is basically a field inside the function object. So every function has its own one. The choice of string/flatstring depends on what was seemed most efficient to Espruino when it was defined.

trace("history")

Again, I don't think you really took in what I said previously. You just defined a String with contents history and you're outputting that, not what was called history.

If you want history, you look at:

#1[r2,l1] Object {
  #2[r1,l2] Name String [1 blocks] "\xFF"    #3[r1,l1] Object {
      #6[r1,l2] Name String [2 blocks] "timers"        #8[r2,l0] Array(0) [ ]
      #9[r1,l2] Name String [2 blocks] "watches"        #11[r2,l0] Array(0) [ ]
      #15[r1,l2] Name String [2 blocks] "history"        #19[r1,l0] Array(54) [

And go back down the trace output, using the name in the lines above:

global["\xff"]["history"]

for (let [key, value] of Object.entries(aaa)) {

Espruino doesn't support that syntax.

global.forEach(element => console.log(element));

A few lines earlier you did typeof global and found it was an object. Objects don't have forEach.

Object.keys(global).forEach(name=> ...); is fine though

As typeof trace() returns ="undefined" I'm starting to wonder if it is in fact just a function that prints to the console

Yes, it prints.

a small sample to get going would be appreciated.

Ok, so there's what I mentioned above with history.

So let's say you're after 637... I've copied your trace contents below, but have removed any lines of the same indenting to make it clearer.

>trace()
#1[r2,l1] Object {
  #428[r1,l2] Name String [1 blocks] "dl"    #438[r1,l0] Function {
      #437[r1,l2] Name String [1 blocks] "\xFFcod"        #637[r1,l0] FlatString [150 blocks] "

So now, you just work back. You can see \xffcod then dl then you're back at the global object, so:

global["dl"]["\xffcod"]

[espruino-discuss]

Posted at 2020-04-23 by Robin

Wed 2020.04.22

Thanks for the answers above, still working through that.

Would you please provide the link to garbage collecting entry point source, specifically seeking the rules determining when GC should occur, and what governs the order that history is reduced. I should then be able to use VSCode to navigate the internal functions. Thank you.

[gfwilliams]

Posted at 2020-04-23 by @gfwilliams

Looks like GitHub is down right now so I can't point you to the exact points, but it's easy enough to search for jsvGarbageCollect.

Specifically:

  if (loopsIdling==1 &&
      minTimeUntilNext > jshGetTimeFromMilliseconds(10) &&
      !jsvMoreFreeVariablesThan(JS_VARS_BEFORE_IDLE_GC)) { ...
        jsvGarbageCollect();

History is cleared line-by-line, oldest first, when memory actually completely runs out. I think it may also be removed completely if a Flat String allocation is required and then fails.

I really think you're going down a rabbit hole here when I don't think there's any need though.

[espruino-discuss]

Posted at 2020-04-23 by Robin

Thr 2020.04.23

Thank you for the function name, that should suffice.

'I don't think there's any need though'

I'd like to see what should be going on and when, before I ask a ton of unnecessary questions.