How to clean HttpCC->httpCRq from memory?

[espruino-discuss]

Posted at 2021-06-06 by Geza

I use the MDBT42Q (2v09) with the the ESP8266. Sometimes the memory starts to run low.
In this case, the size of HttpCC increased:

>E.getSizeOf(global,2);
=[
  {
    name: "\xFF",
    size: 1176,
    more: [
      {
        name: "HttpCC",
        size: 359 }

print(global["\xFF"].HttpCC);
[
  httpCRq: { "type": 1,
    "#onconnect": function (b) {var a='';b.on('data',function(b){a.length<8&&(a+=b)}),b.on('close',function(b){e(a.substring(0,5))})},
    "res": httpCRs: { "hdrs": false },
    "opt": {
      "protocol": "http:",
      "method": "POST",
      "host": "111.1.111.11",
      "path": "/ords/server/rest/espruino/",
      "pathname": "/ords/server/rest/espruino/",
      "search": null, "port": 8081, "query": null,
      "headers": {
        "Content-Type": "application/json",
        "Content-Length": 250 }
     },
    "#onerror": function (a) {d='',e(),digitalPulse(LED1,1,600),log('err1.pc'),ir('Err:1')},
    "dSnd": "",
    "sckt": 2, "cls": true,
    "dRcv": "HTTP/1.1 200 OK\r\nConnection: close\r\nDate: Thu, 03 Jun 2021 19:53:31 GMT\r\n\r1"
   }...

if i use the following command in WEB IDE (left side), it will clean the memory:

global["\xFF"].HttpCC=[];

But i can't use it in js.
How can I achieve the same in my javascript code?

[espruino-discuss]

Posted at 2021-06-06 by @allObjects

@geza, with exception of a few things, all you do in the left side of the IDE / Console, you can do on your application code, because all is js and it is executed (see REPL) by the Espruino JS interpreter on your MDBT42Q.

I assume you 'cannot do it' because the things you try to throw out is at the times a very part of your application / communication while running / begin connected / having pending requests / responses.

I do not expect a memory leak and therefore the HttpCC object (Array?) doe snot keep growing. Looks to me that individual elements come and go and grow and shrink over time.

What you may look at is to cleanup received / sent data that is still lingering around even after the communication 'transaction' completed (until the next one is setup and replaces it).

May be that with large amounts of data you can apply streaming as described here: https://www.espruino.com/Internet#transferring-large-amounts-of-data

[espruino-discuss]

Posted at 2021-06-06 by Robin

Sun 2021.06.06

'it will clean the memory:    global["\xFF"].HttpCC=[];    But i can't use it in js'
'How can I achieve the same in my javascript code?'

Does running the following command more frequently, both inform and improve on making memory available?

process.memory()

'Run a Garbage Collection pass, and return an object containing information on memory usage'

http://www.espruino.com/Reference#process

[espruino-discuss]

Posted at 2021-06-06 by Geza

@allObjects thank you for your thoughts.

with exception of a few things, all you do in the left side of the IDE / Console, you can do on your application code, because all is js
and it is executed (see REPL) by the Espruino JS interpreter on your
MDBT42Q.

I thought the same, but:

function sethttp() {
  global["\xFF"].HttpCC=[];
}

The result:

>sethttp()
Uncaught SyntaxError: Got [ERASED] expected ID
 at line 1 col 8
global.[ERASED]
              ^
in function "sethttp" called from line 1 col 9
sethttp()
        ^

WEB Ide's left side:

>global["\xFF"].HttpCC=[];
=[  ]

I do not expect a memory leak and therefore the HttpCC object (Array?)
doe snot keep growing. Looks to me that individual elements come and
go and grow and shrink over time.

Normally this is indeed the case, but sometimes it is not. I have attached a picture of memory usage. In case the memory decreased, the size of the HttpCC array increased.

---

Attachments:

• Screenshot 2021-06-06 at 15.50.50.png

[espruino-discuss]

Posted at 2021-06-06 by Geza

@robin yes, I use the process.memory() and in some cases it doesn't help. In those cases, I can only empty the HttpCC array manually.

[espruino-discuss]

Posted at 2021-06-07 by @allObjects

There is some memory discussion in conversation about Help needed on trace() to locate user var reference and contents. It may shed some light on this ...[ERASED] thing.

[gfwilliams]

Posted at 2021-06-09 by @gfwilliams

Uncaught SyntaxError: Got [ERASED] expected ID

This one may actually be unrelated - as @MaBecker says the code itself should work fine. Are you writing to Storage from inside your application?

As a hack, this might fix it:

function sethttp() { "ram"
  global["\xFF"].HttpCC=[];
}

While this shouldn't happen in 2v09, in older firmwares you could get in a state where a function was loaded and referenced Storage, but the contents of storage had moved. If we could reproduce with some simple code then I could try and get a fix in

about the actual memory leak...

It'd be good to try and track down why this is happening (if there's some really simple code you can share which reproduces the leak?). Usually having a HttpCC just means the HTTP connection never got closed (eg you were using http.request and didn't call .end on it), but it could be a newer version of the ESP8266 firmware changed what data it sent slightly and then the socket close event is no longer handled.

[espruino-discuss]

Posted at 2021-06-10 by @MaBecker

This is the conversation about that topic

[espruino-discuss]

Posted at 2021-06-10 by Geza

@allObjects thanks for the suggestion

[espruino-discuss]

Posted at 2021-06-10 by Geza

This one may actually be unrelated - as @MaBecker says the code itself should work fine. Are you writing to Storage from inside your application?

Yes, i use it for logging

It'd be good to try and track down why this is happening

@gfwilliams You think well (as always)..the problem was basically caused by

NRF.setScan(function(dev) {..}); 

because it was still running when the require("http").get(...) started. It also produced several strange things after a few days... for example memory leak, http socket close error, even the temperature reading became inaccurate.

Thanks for all the support!

[gfwilliams]

Posted at 2021-06-10 by @gfwilliams

Glad you found the problem! I guess maybe there was some logging to Storage inside your setScan handler?

I just had a quick go at reproducing this and it seems I can actually do it pretty easily (even on desktop) with:

const Storage = require('Storage');
Storage.eraseAll();
Storage.write('a', new Uint8Array(500)); //<--- this is what causes the problem
Storage.write('code', 'function hello() { print("Hello1"); }');
Storage.write('code', 'function hello() { print("Hello2"); }');
Storage.write('code', 'function hello() { print("Hello"); }');
eval(Storage.read('code'));
trace(hello);
hello();
Storage.compact();
hello();
trace(hello);

I have just committed a fix for this so there should be a build ready in the next few minutes.

It may actually be that this is the cause of the vast majority of your issues (if code that was stored in flash is no longer being executed correctly)

[espruino-discuss]

Posted at 2021-06-10 by Geza

I guess maybe there was some logging to Storage inside your setScan handler?

I store the result of setScan in an array, but inside the http handler write the log. I log not only the errors but also the exceptions. I'm trying the fix, which is interesting! I attach the memory usage graph.

---

Attachments:

• Screenshot 2021-06-10 at 16.47.41.png

[espruino-discuss]

Posted at 2021-06-11 by @allObjects

To get you logging out of the request memory scope, you could create a valueObj for the data to log - var valueObj = { p1:v1, p2:v2, ...} and then do the logging deferred with setTimout(function(valueObj){<logging code; >},<timeoutTime>,valueObject). This may for a short time use a bit more memory but all communication transaction held resources get released earlier. The timeout time has not to be a real defer, it is there to break the execution flow