NewBee - unknown "possible" memory leak?

[espruino-discuss3]

Posted at 2014-06-28 by user7114

Being a hard core Arduino user, I decided to switch to the Espruino.
I put together a DS18B20 data logger. After a few days, I noticed
on the I2C LCD, that memory usage was increasing from 646 to 651.
Any help is appreciated to track down this memory leak. (See code below)

BTW ... I am very impressed with the amount of program code needed for this project. It would of taken thousands of lines of code for the Arduino.

/*
DS18B20 Data Logger
Rev. 1.0b
6/26/14
Espruino: v 1.3: firmware: v 65 WEB-IDE: v 45
Windows 8.1 / Chrome
Hardware: I2C 4x20 LCD, DS18B20, I2C RTC DS3231
Bugs: Memory usage 646 but after a few days increases to 651 ???

Program description:
Uses 1-wire DS18B20 for the temperature sensor.
The DS18B20 temperature is logged along with time/date to SD and LCD.
1-wire and 1 I2C bus is used.

Program credits:
SD data logger code snippet was used from an unknown Espruino forum member.

Note: 3.3k pull-ups for I2C LCD/RTC.
*/

function onInit(){
var ow = new OneWire(A1);
var sensor = require("DS18B20").connect(ow);
sensor.setRes(12); 
digitalWrite([LED1,LED2,LED3],0b100);
var lcd = require("HD44780").connectI2C(I2C1);
I2C1.setup({scl:B8,sda:B9});
lcd.clear(); 
setTimeout("digitalWrite([LED1,LED2,LED3],0b010);", 1000);
setTimeout("digitalWrite([LED1,LED2,LED3],0b001);", 2000);
setTimeout("digitalWrite([LED1,LED2,LED3],0);", 3000);
// Delay start-up for diagnostics (above) and for LCD clear
I2C1.setup({scl:B8, sda:B9});
var rtc = require("DS3231").connect(I2C1);
}

var fs = require('fs');
var testName    = "TEST152L.LOG";
var testDetails = "Data Logging DS18B20 to SD";
var logInterval = 5000;
var logEntry    = 0;

var ow = new OneWire(A1);
var sensor = require("DS18B20").connect(ow);

I2C1.setup({scl:B8,sda:B9});
var rtc = require("DS3231").connect(I2C1);

/* Set Dow Time/date - comment out lines out below after first use!
Usage:
rtc.setDow("day of the week") Monday,Tuesday ...
rtc.setDate(date,month,year);1-31,1-12,0-99
rtc.setTime(hours,minutes); Military Time 
Set time exactly on minute roll-over (+ 1  min)
*/

//rtc.setDow("Thursday");
//rtc.setDate(26,6,14);
//rtc.setTime(19,25); 

I2C1.setup({scl:B8, sda:B9});
var lcd = require("HD44780").connectI2C(I2C1);
lcd.clear();


// Initialise log file here with header information
console.log(testDetails);
fs.appendFileSync(testName, testDetails);

// Function to log some data every so often
setInterval(function(){
    // Indicate each time function is executed 
    digitalPulse(LED1, 1, 100);
// Log some data.
    logEntry++;
 var logString = logEntry + " " + (sensor.getTemp(true)* 1.8 + 32 + " F. ") +
    (rtc.readDateTime()) + " "  + "\r\n";

    //console.log(logString); <---- do not leave in without USB (will halt)
    //console.log(process.memory().usage); <--- do not leave in without USB (will halt)
    fs.appendFileSync(testName, logString);
    lcd.setCursor(0,0);
    lcd.print("DS18B20 Logging ..."); 
    lcd.setCursor(0,1);
    // lcd needs a string ????
    lcd.print("TempF: " + new String(sensor.getTemp(true)*1.8 + 32));
    lcd.setCursor(0,2);
    lcd.print(rtc.readDateTime());
    lcd.setCursor(0,3);
    lcd.print(new String(process.memory().usage));
    
}, logInterval);

[espruino-discuss3]

Posted at 2014-06-28 by @gfwilliams

Hmm. I'm not 100% sure if it is a leak... usually leaks cause 1 or 2 vars to be lost each time, so the loss is often a lot more drastic. Maybe check in another day and see if it has got worse? It could just be a variable that is storing a larger string than usual.

---

Posted at 2014-07-05 by user7114

"One drip from a water faucet can be gallons/liters of water per week"

An Espruino " very slow memory leak" from variables can also be troublesome.
The memory usage, on average, running the DS18B20 data logger is about 10 memory units per week.
Any suggestions welcomed to fix this problem.

---

Posted at 2014-07-05 by @gfwilliams

Are you running the latest 1v66? I found a memory leak in the serial port receive handler (which is fixed in that). It's possible that a little noise gets on pin A10, is interpreted as a character, and then causes the leak?

If you are still hitting problems with 1v66, are there any error messages shown? It's possible that some error when accessing a file causes a leak.

Otherwise, if you can find something that increases the rate of leaking memory, it might be a hint as to what the problem is.

---

Posted at 2014-07-05 by user7114

Are you running the latest 1v66?

Yes, Last week I just obtained 1v66 and had to reload the firmware.

It's possible that a little noise gets on pin A10, is interpreted as a
character, and then causes the leak?

I not sure I understand. The data logger program (above) does not even use A10?

If you are still hitting problems with 1v66, are there any error
messages shown?

Without the computer connected to the Espruino, there would be no way of telling?
I cannot tie my PC up for weeks at a time for this testing.
Note: I created a "standalone" hardware configuration in which the Espruino, DS18B20, LCD 4x20 runs off of a wall-wart.

Maybe, its the 1-wire "12 bit" DS18B20 and I will try substitute this temperature sensor with a constant?

---

Posted at 2014-07-05 by @gfwilliams

The data logger program (above) does not even use A10?

No, but when USB is unplugged, Espruino automatically switches the console over to use it. I just tried randomly poking the board around A10 and memory usage rose - it could be the problem.

Please could you look at Serial1.available() and see if it reads nonzero? If you then type Serial1.read() the data will be read out of the buffer and memory usage will drop.

The buffer for Serial1 is 'only' 256 bytes or so so it won't ever stop your code from working. I'll see if I can do something about it though - perhaps by not storing the data unless someone has previously called Serial1.read/available

---

Posted at 2014-07-05 by user7114

Ok, Serial1.available() reads 0(zero) on the I2C LCD 4x20 display when the Espruino
is not connected to the USB. I will give it more run time testing.
Note: I am reading "Serial1.available()" every 5 seconds.

Note: The I2C LCD display comes in extremely handy for troubleshooting an
"unconnected" Espruino! (For $10 USD it can't be beat anywhere)
http://yourduino.com/sunshop2/index.php?l=product_detail&p=332
"sorry for the plug"

---

Posted at 2014-07-08 by user7114

After a few days of 5 second temperature logging, the memory usage units increase from 657 to 667. The Serial1.available() still had zero on the LCD display. <-----
So, in conclusion, it was not the console serial port A10?

The "possible" very slow memory leak might be comming from any of the sensor or devices. I2C LCD, I2C RTC, 1-Wire DS18B20 or the buit-in SD card? It could be even in program, module or firmware?

New memory leak testing changes:
Disabled the 1-wire DS18b20 sensor read with a fixed constant.
On line 76 (above), I replaced the 1-wire "(sensor.getTemp(true)* 1.8 + 32 + " F. ")" with a fixed constant "1234" and on line 86 (above), I replaced "lcd.print("TempF: " + new String(sensor.getTemp(true)*1.8 + 32));" with a fixed constant of "5678"

After a few days of testing, the memory "unit" usage again increased by 10 and the Serial1.available() still stayed at zero. Still no help - any suggestions
welcomed <------<<<<

Espruino: v 1.3: firmware: v 66, WEB-IDE: v 45, Windows 8.1 / Chrome

---

Posted at 2014-07-08 by DrAzzy

Sounds like it's down to either I2C (for the screen), or the SD card access.

I assume you've verified that the log files are clean? That would be something to check - if it's being caused by SD card access, maybe you'd see a sign of it there...

---

Posted at 2014-07-09 by user7114

I assume you've verified that the log files are clean?

The data on the SD card seems OK

Since I posted the code (above), if someone else could run and verify the same memory leak problem? After all, the complete DS18B20 logger is hooked up on a "capacitance" breadboard and the problem could be hardware.

---

Posted at 2014-07-09 by @gfwilliams

Right, I guess it's unlikely to be Serial1 then. Can you try changing:

function onInit(){
var ow = new OneWire(A1);
var sensor = require("DS18B20").connect(ow);
sensor.setRes(12); 
digitalWrite([LED1,LED2,LED3],0b100);
var lcd = require("HD44780").connectI2C(I2C1);
I2C1.setup({scl:B8,sda:B9});
lcd.clear(); 
setTimeout("digitalWrite([LED1,LED2,LED3],0b010);", 1000);
setTimeout("digitalWrite([LED1,LED2,LED3],0b001);", 2000);
setTimeout("digitalWrite([LED1,LED2,LED3],0);", 3000);
// Delay start-up for diagnostics (above) and for LCD clear
I2C1.setup({scl:B8, sda:B9});
var rtc = require("DS3231").connect(I2C1);
}

to:

var ow,sensor,lcd,rtc;
function onInit(){
ow = new OneWire(A1);
sensor = require("DS18B20").connect(ow);
sensor.setRes(12); 
digitalWrite([LED1,LED2,LED3],0b100);
lcd = require("HD44780").connectI2C(I2C1);
I2C1.setup({scl:B8,sda:B9});
lcd.clear(); 
setTimeout("digitalWrite([LED1,LED2,LED3],0b010);", 1000);
setTimeout("digitalWrite([LED1,LED2,LED3],0b001);", 2000);
setTimeout("digitalWrite([LED1,LED2,LED3],0);", 3000);
// Delay start-up for diagnostics (above) and for LCD clear
I2C1.setup({scl:B8, sda:B9});
rtc = require("DS3231").connect(I2C1);
}

and removing the code you have below that also sets those sensors up? When you write var in a function, it defines a local variable - so you were defining those variables and then they were being forgotten when the function exited. That shouldn't have caused a memory leak, but it might have caused some problems.

Can you also try typing trace() right after you load the program (and copy/paste it to a file) and then 'trace()' after it's lost some memory (and save that to a file as well)? We can then just 'diff' the two traces and see if it's actually a bigger than normal string or similar that is causing the leak.

---

Posted at 2014-07-09 by user7114

and removing the code you have below that also sets those sensors up?

Here's the problem ...
"If I remove the code you have below" it won't run in RAM because the function
onInit() has the device(s) setup code? Only when I do a flash "Save()" will the
function onInit() will run? The problem is ... if the code will not run
in RAM first, it sure will not run in flash by doing a save()?
For your testing, I had to leave in the I2C setup code.

On another very minor issue ...
I believe you put a patch in v66 for Serial1 noise. In v65 by printing to the LCD Serial1.available() nothing happens on the console screen. In v66, leaving
the console connected (USB), I get " a list of "Warning: String buffer overflowed maximum size (512)" when ever I print to the LCD the function "Serial1.available()?

Test code ...

/*
DS18B20 Data Logger
Rev. 1.0b
7/9/14
Espruino: v 1.3: firmware: v 66 WEB-IDE: v 45
Windows 8.1 / Chrome
Hardware: I2C 4x20 LCD, DS18B20, I2C RTC DS3231
Bugs: Memory usage 657 but after a few days increases to 667 ???

Program description:
Uses 1-wire DS18B20 for the temperature sensor.
The DS18B20 temperature is logged along with time/date to SD and LCD.
1 1-wire and 1 I2C bus is used.

Program credits:
SD data logger code snippet was used from an unknown Espruino forum member.

Note: 3.3k pull-ups for I2C LCD/RTC.
*/


var ow,sensor,lcd,rtc;
function onInit(){
ow = new OneWire(A1);
sensor = require("DS18B20").connect(ow);
sensor.setRes(12); 
digitalWrite([LED1,LED2,LED3],0b100);
lcd = require("HD44780").connectI2C(I2C1);
I2C1.setup({scl:B8,sda:B9});
lcd.clear(); 
setTimeout("digitalWrite([LED1,LED2,LED3],0b010);", 1000);
setTimeout("digitalWrite([LED1,LED2,LED3],0b001);", 2000);
setTimeout("digitalWrite([LED1,LED2,LED3],0);", 3000);
// Delay start-up for diagnostics (above) and for LCD clear
I2C1.setup({scl:B8, sda:B9});
rtc = require("DS3231").connect(I2C1);
}


var fs = require('fs');
var testName    = "TEST152Q.LOG";
var testDetails = "Data Logging DS18B20 to SD";
var logInterval = 5000;
var logEntry    = 0;

var ow = new OneWire(A1);
var sensor = require("DS18B20").connect(ow);

I2C1.setup({scl:B8,sda:B9});
var rtc = require("DS3231").connect(I2C1);

/* Set Dow Time/date - comment out lines out below after first use!
Usage:
rtc.setDow("day of the week") Monday,Tuesday ...
rtc.setDate(date,month,year);1-31,1-12,0-99
rtc.setTime(hours,minutes); Military Time 
Set time exactly on minute roll-over (+ 1  min)
*/

//rtc.setDow("Thursday");
//rtc.setDate(26,6,14);
//rtc.setTime(19,25); 

I2C1.setup({scl:B8, sda:B9});
var lcd = require("HD44780").connectI2C(I2C1);
lcd.clear();


// Initialise log file here with header information
//console.log(testDetails); <---- do not leave in without USB (will halt)
fs.appendFileSync(testName, testDetails);

// Function to log some data every so often
setInterval(function(){
    // Indicate each time function is executed 
    digitalPulse(LED1, 1, 100);
// Log some data.
    logEntry++;
 var logString = logEntry + " " + (sensor.getTemp(true)* 1.8 + 32 + " F. ") +
    (rtc.readDateTime()) + " "  + "\r\n";
  
// var logString = logEntry + " " + 1234 +
   // (rtc.readDateTime()) + " "  + "\r\n";


    //console.log(logString); <---- do not leave in without USB (will halt)
    //console.log(process.memory().usage); <--- do not leave in without USB (will halt)
    fs.appendFileSync(testName, logString);
    lcd.setCursor(0,0);
    lcd.print("DS18B20 Logging ..."); 
    lcd.setCursor(0,1);
    // lcd needs a string
    lcd.print("TempF: " + new String(sensor.getTemp(true)*1.8 + 32));
    //lcd.print("TempF: " + "5678");
    lcd.setCursor(0,2);
    lcd.print(rtc.readDateTime());
    lcd.setCursor(0,3);
    lcd.print(new String(process.memory().usage));
    lcd.print(" ");
    //lcd.print(new String(Serial1.available()));
  
  
     
  
}, logInterval);

---

Posted at 2014-07-10 by @gfwilliams

All you have to do to avoid the setup issue is to specifically put a call to onInit() at the end of your code...

If you're having the issues with the buffer overflowing then that could have been causing the leak (especially under 1v65). For now, try adding the line pinMode(A10,'input_pullup'). That will try and make sure that the Serial line doesn't get data that it shouldn't have.

---

Posted at 2014-07-11 by user7114

@gfwilliams... I did everything what you instructed (above) "to the T"
I did a trace file at the beginning of the testing and could not
do another trace file because the memory "usage" did not change!
In fact, the memory usage is rock solid as in the rock of Gibraltar!
After a day of running off of the USB, I switched overnight to a
remote offline power supply with the same results - so far.
It must be the pullup on the serial1 receive line?
I will be running it for a few more days of testing.

BTW ... Its cool to have the Espruino to tell the user that the
"console was moved from serial1"

Update ... after four days of remote logging testing, the memory usage did
not change. End of testing.

---

Posted at 2014-07-14 by @gfwilliams

Great! Glad it's sorted!

In the 1v67 release I've actually added that pullup by default too (so you won't have to explicitly add it).