STM32F4. Read data from UART. ERROR: Out of Memory!

[espruino-discuss]

Posted at 2014-07-11 by user7265

Hi, Gordon, Hi all!
Working with the board STM32F4.
I have a problem when I try to transfer file from PC to UART (pins PA2, PA3).
If the file size small (less 200 bytes) all is well.
If the file size a bit more, then sometimes console shows the error memory.
If the file size like 2000 bytes, then always console shows the error memory.

I try change baudrate of UART from 9600 to 115200, and have a successful file transfer size in 2 528 bytes. But after a few hours situation with the problem persists.
STM32F4 has a lot of memory. Why this error occurs?

My project:

var LED_ORANGE = LED1;
var LED_GREEN  = LED2;
var LED_RED    = LED3;
var LED_BLUE   = LED4;
var arr = null;


function Toggle_Led(LED) {
 on = !on;
 digitalWrite(LED, on);
 return 1;
}

function Show(str) {
  console.log('**' + str);
}

function Thread1() {
  Toggle_Led(LED_ORANGE);

}

var Counter = 0;
Show("Start!");


// Тест UART
//==================================================================
Serial2.setup(9600, {rx:A3,tx:A2});
//USB.setup(9600, {dm:A11,dp:A12});
Serial2.on('data', function (data) {  // Receive function
  console.log(Counter + ") send");
  process.memory();
  if (data.length > 0) {  
    digitalWrite(LED_GREEN, 1);
    //console.log(data + "\n");
    
    if (arr == null)  {
      arr = new Array();
    }
    arr.push(data);          // Add new element

    // Таймаут приёма
    //==============================================================
    setTimeout(function (e) {
      if (arr == null)  return;
      //for (var i=0; i<arr.length; i++)
      //{
      //  console.log(arr[i]);
      //}
      var str = arr.join("");
      console.log(str);
      
      arr = null;
      Serial2.println(str);        // Send
      
      digitalWrite(LED_GREEN, 0);
      digitalWrite(LED_RED, 0);
    }, 5000);
    //==============================================================
  } else {
    digitalWrite(LED_RED, 1);
    console.log("Bad");
  }
  
  Counter++;
});
//==================================================================



setInterval(Thread1,3000);
Show("Stop!");
save();

My File:
{
"crosstime": {
"1": [
1,
10,
3,
3
],
"2": [
1,
10,
3,
3
],
"3": [
1,
20,
3,
1
]
},
"racks": {
"1": {
"1": 1,
"2": 1,
"3": 0,
"4": 0,
"5": 5,
"6": 5
},
"2": {
"1": 2,
"2": 2,
"3": 0,
"4": 0,
"5": 5,
"6": 5
},
"3": {
"1": 1,
"2": 1,
"3": 0,
"4": 0,
"5": 5,
"6": 5
},
"4": {
"1": 2,
"2": 2,
"3": 0,
"4": 0,
"5": 5,
"6": 5
}
},
"groups": {
"1": {
"1": [
1,
3,
4,
5
],
"2": [
2,
2,
2,
2
],
"3": [
2,
2,
2,
2
]
},
"2": {
"1": [
2,
2,
2,
2
],
"2": [
1,
3,
4,
5
],
"3": [
2,
2,
2,
2
]
},
"3": {
"1": [
2,
2,
2,
2
],
"2": [
2,
2,
2,
2
],
"3": [
2,
3,
4,
2
]
}
},
"desc": {
"note": "\u041f\u0440\u043e\u0441\u0442\u0435\u0439\u0448\u0438\u0439 \u0441\u0432\u0435\u0442\u043e\u0444\u043e\u0440 \u0438\u0437 3 \u0433\u0440\u0443\u043f\u043f \u0438 3 \u0444\u0430\u0437.",
"phases": "3",
"name": "\u0422\u0435\u0441\u0442",
"groups": "3",
"addr": "\u0421\u0442\u0435\u043d\u0434"
}
}

[espruino-discuss]

Posted at 2014-07-11 by user7265

Error:

0) send
1) send
2) send
3) send
4) send
5) send
6) send
7) send
8) send
9) send
10) send
11) send
12) send
13) send
...
...
...
126) send
127) send
128) send
129) send
130) send
131) send
132) send
ERROR: Out of Memory!
WARNING: Unable to create string as not enough memory
ERROR: Out of Memory!
ERROR: Out of Memory!
WARNING: Unable to create string as not enough memory
ERROR: Out of Memory!
ERROR: Out of Memory!
WARNING: Unable to create string as not enough memory
ERROR: Out of Memory!
ERROR: Out of Memory!
WARNING: Unable to create string as not enough memory
ERROR: Error processing Serial data handler - removing it.
Execution Interrupted during event processing.
at line 3 col 18
  process.memory();
in function called from system
WARNING: String buffer overflowed maximum size (512)
WARNING: String buffer overflowed maximum size (512)
...
...
...
WARNING: String buffer overflowed maximum size (512)
WARNING: String buffer overflowed maximum size (512)
WARNING: String buffer overflowed maximum size (512)
ERROR: Out of Memory!
ERROR: Out of Memory!
...
...
...
ERROR: Out of Memory!
ERROR: Out of Memory!
Execution Interrupted during event processing.
at line 2 col 18
      if (arr == null)  return;
                  
in function called from system
{
    "crosstime": {
        "1": [
            1,
            10,
            3,
            3
        ],
        "2": [
            1,
            10,
            3,
            3
        ],
        "3": [
            1,
            20,
            3,
            1
        ]
    },
    "racks": {
        "1": {
            "1": 1,
            "2": 1,
            "3": 0,
            "4": 0,
            "5": 5,
            "6": 5
        },
        "2": {
            "1": 2,
            "2": 2,
            "3": 0,
            "4": 0,
            "5"
> 

[gfwilliams]

Posted at 2014-07-11 by @gfwilliams

Ok, first check that you're using the latest version... Version 1v65 had a memory leak when receiving from Serial ports.

And then, you should switch from using arrays to using a single string that you append.... Instead of arr.push(data); and var str = arr.join(""); just use str += data;

Arrays in JavaScript are sparse, so each element has to store both the index and the value. In your case, it could be that for every character you are storing the index and a string, which will eat up the memory very quickly. It's much better to put all your information into one string - see http://www.espruino.com/Performance for more info.

The reason it works at higher baud rates is that data contains more characters at a time. This means less entries in the array, which means less memory usage.

[espruino-discuss]

Posted at 2014-07-11 by user7265

SUCCESS (9600 baudrate)
Before transmitting the file (400 bytes) process.memory() returns:
{"free":5296,"usage":154,"total":5450,"history":134,"stackEndAddress":536984008,"flash_start":134217728,"flash_binary_end":1

ERROR (9600 baudrate)
After transmitting the file (400 bytes) process.memory() returns:
{"free":58,"usage":5392,"total":5450,"history":1,"stackEndAddress":536984008,"flash_start":134217728,"flash_binary_end":134413196,"flash_code_start":135135232,"flash_length":1048576}

SUCCESS (115200 baudrate)
After transmitting the file (2 528 bytes) process.memory() returns:
{"free":1195,"usage":4255,"total":5450,"history":140,"stackEndAddress":536984008,"flash_start":134217728,"flash_binary_end":134413196,"flash_code_start":135135232,"flash_length":1048576}

[espruino-discuss]

Posted at 2014-07-11 by user7265

Thank you for answer! Be sure to tell about the results.

[espruino-discuss]

Posted at 2014-07-11 by DrAzzy

I think I see the problem - you're setting a timeout to read back the data EVERY TIME you read from the serial port, so you end up with a crapload of timeouts. The number of reads you get before it dies at 9600 baud is (based on a quick estimate) consistent with storing the data (in a simple array, so it wastes tons of memory) AND creating a timeout for every byte. The solution is to only set the timeout once.

var LED_ORANGE = LED1;
var LED_GREEN  = LED2;
var LED_RED    = LED3;
var LED_BLUE   = LED4;
var arr=null;  
var timeout = null; //variable for timeout
function Toggle_Led(LED) {
 on = !on;
 digitalWrite(LED, on);
 return 1;
}
function Show(str) {
  console.log('**' + str);
}
function Thread1() {
  Toggle_Led(LED_ORANGE);
}
var Counter = 0;
Show("Start!");
// Тест UART
//==================================================================
Serial2.setup(9600, {rx:A3,tx:A2});
//USB.setup(9600, {dm:A11,dp:A12});
Serial2.on('data', function (data) {  // Receive function
  console.log(Counter + ") send");
  process.memory(); //This does nothing, other than possibly slow down code execution - did you mean to log it with console.log()? 
  if (data.length > 0) {  
    digitalWrite(LED_GREEN, 1);
    //console.log(data + "\n");
    
    if (arr == null)  {
      arr = new Array();
    }
    arr.push(data);          // Add new element
    // Таймаут приёма
    //==============================================================
    if (timeout == null) {  //If we haven't started the timeout yet, start it now, since we're getting data
        timeout=setTimeout(function (e) {
          console.log(process.memory());
          if (arr == null)  return;
          //for (var i=0; i<arr.length; i++)
          //{
          //  console.log(arr[i]);
          //}
          var str = arr.join("");
          console.log(str);
          
          arr = null;
          Serial2.println(str);        // Send
      
          digitalWrite(LED_GREEN, 0);
          digitalWrite(LED_RED, 0);
        }, 5000);
    } //Close the if...
    //==============================================================
  } else {
    digitalWrite(LED_RED, 1);
    console.log("Bad");
  }
  
  Counter++;
});
//==================================================================
setInterval(Thread1,3000);
Show("Stop!");
save();

For better memory usage, something like this would be a good start - by not creating the array and just building the string as it comes.

var LED_ORANGE = LED1;
var LED_GREEN  = LED2;
var LED_RED    = LED3;
var LED_BLUE   = LED4;
var str = ""; //no array, just a string.
var timeout = null; //variable for timeout
function Toggle_Led(LED) {
 on = !on;
 digitalWrite(LED, on);
 return 1;
}
function Show(str) {
  console.log('**' + str);
}
function Thread1() {
  Toggle_Led(LED_ORANGE);
}
var Counter = 0;
Show("Start!");
// Тест UART
//==================================================================
Serial2.setup(9600, {rx:A3,tx:A2});
//USB.setup(9600, {dm:A11,dp:A12});
Serial2.on('data', function (data) {  // Receive function
  console.log(Counter + ") send");
  process.memory();
  if (data.length > 0) {  
    digitalWrite(LED_GREEN, 1);
    str += data;
    //console.log(data + "\n");
    // Таймаут приёма
    //==============================================================
    if (timeout == null) {  //If we haven't started the timeout yet, start it now, since we're getting data
        timeout=setTimeout(function (e) {
          console.log(process.memory());
          if (str == "")  return;
          console.log(str);
          Serial2.println(str);        // Send
          digitalWrite(LED_GREEN, 0);
          digitalWrite(LED_RED, 0);
          timeout=null; //clear the timeout variable at the end
          str=""; //clear the string
        }, 5000);
    } 
    //==============================================================
  } else {
    digitalWrite(LED_RED, 1);
    console.log("Bad");
  }
  
  Counter++;
});
//==================================================================
setInterval(Thread1,3000);
Show("Stop!");
save();

Also, if you want it to work when running from flash, you might have to put the Serial2.on(...); inside an onInit() function. I don't recall whether Espruino will remember callbacks like that on save() or whether you need to set them in onInit().

(Note: My code is untested, might have typo/etc in it. Edit: just fixed one of them)

Edit: Even when it's "successful" in the case you've shown above, it looks like it's leaking oodles of memory....

[espruino-discuss]

Posted at 2014-07-14 by user7265

DrAzzy thank you. The problem is really in incorrect use of setTimeout. This function was spending a lot of memory, because i often called this.
Friends, you all helped me a lot thank you so much!

[espruino-discuss]

Posted at 2014-07-15 by user7265

Gordon can you explain why object setTimeout spend a lot of memory?

[gfwilliams]

Posted at 2014-07-15 by @gfwilliams

You can probably see by typing trace() after a timeout is set. The timeout itself might take maybe 6 memory units, but the real killer is the function that you define to run in the timeout.

That needs the function code itself (which is copied each time because it was defined inside the function), as well as the scope in which the function was originally defined.

But either way I imagine you had lots of timeouts queued up. It's the kind of thing that would be really bad news even on a PC with loads of memory - it just becomes more of a problem more quickly on a constrained device.

[espruino-discuss]

Posted at 2014-07-15 by user7265

Thank you very much for the detailed explanation.