Lost storage data on esp8266

[espruino-discuss]

Posted at 2020-04-20 by user111618

Hey guys, I found an issue that sometimes I will lose all data on the storage. I use websocket to get new code from the server and write it into the storage. At first, I simply erase and write, but then I realized free space becomes less and less so I add compact between erase and write, it works normally most of the time, but it did lost all data sometimes.

I would like to know how to prevent this, I'm a noob, thanks.

[espruino-discuss]

Posted at 2020-04-20 by @MaBecker

Can you share some infos.

• Board
• Espruino version
• code snippet

---

Posted at 2020-04-21 by user111618

Yeah, the board is esp8266-01s and the version of espruino is 2.04. Usually, the program will get new code from WebSocket server and run

require("Storage").erase("usercode");
require("Storage").compact();
require("Storage").write("usercode", newCode);

---

Posted at 2020-04-21 by Raik

Maybe it is related to the duplicate files issue, see http://forum.espruino.com/conversations/345948/#comment15216864

Can you do a

require("Storage").list("usercode")

to see if there are duplicate files? But, they may they only appear if you are acutally experiencing the "data loss".

You could possibly also update to 2.05 and then see if the issue is gone.

---

Posted at 2020-04-21 by @MaBecker

Yes, a upgrade would be helpful as @raik suggested.

---

Posted at 2020-04-22 by user111618

Yes, due to some code changes, the issue does not appear frequently. I will try the same actions on 2.05 and post the result. Thanks for your help.

---

Posted at 2020-04-22 by user111618

Yeah, I will try that, thank you.

---

Posted at 2020-04-25 by user111618

@MaBecker @raik I tried to update to 2.05 and run some tests. The result is this issue does not vanish.
I have code in .boot0 which is

if(fs.getFree()<4096) fs.compact();

and I can see espruino will compact and restart when the free space down below 4096, then I run fs.compact() again in console after it restarted, and after reboot, all data were lost. fs.getFree() shows me 16384 on esp8266-01s, and fs.list() has an empty array as result.

---

Posted at 2020-04-25 by user111618

and I haven't tried this on nodemcu, but I guess I will get the same result.
So someone please tell me how to deal with it, is my action wrong? Or just an issue in espruino, I don't want to take it out to code or upgrade after I set it into a dishwasher. Thanks!

---

Posted at 2020-04-25 by @MaBecker

esp8266-01s

With 1 Mb ?

---

Posted at 2020-04-25 by @MaBecker

@jumjum is the firmware builder online ?

---

Posted at 2020-04-28 by user111618

Yes

---

Posted at 2020-04-28 by @MaBecker

I guess require("Storage").compact(); can't work because storage space is to small for what you are trying.

What about eraseAll(), write .boot and usercode, or by a 4MB board or

build a 1MB firmware Espruino version without FOTA but lots of space like a 4MB board.

        BOARD=ESP8266_4MB make clean
        BOARD=ESP8266_4MB RELEASE=1 NO_FOTA=1 FLASH_1MB=1 make 

Fixed typo

---

Posted at 2020-04-29 by user111618

I have NodeMCU(32MB version), but the size(physical) is too big. And my project only needs 2 IOs.

Besides, this happened when I tried to require("Storage").compact();, the free space on the flash at least has 60%, and my purpose is to compact it myself instead of automatically(when the free space run out), cause that will lead to data loss.

And I don't know how to compile it myself, I cloned the source and tried to compile, but I have no idea where I could find the hex file, I am noob. :D.

But anyway, thank you for the reply.

---

Posted at 2020-05-04 by user111618

I made a version using your arguments and flashed the 01s, but the issue still happened when I do the same actions. But I do get 196608 space like 4MB board.
And, could I ask why the NO_FOTA=1 don't work for 4MB board, like NodeMCU, I can't still use FOTA on it. My purpose is to get more space beyond 196608 on 4MB board, thank you.

---

Posted at 2020-05-04 by @MaBecker

And, could I ask why the NO_FOTA=1 don't work for 4MB board, like NodeMCU

If running run a 4MB firmware image on a 1MB chip, flash over the air is not possible.

Flash over 1MB is not uses with the Storage module, but can be used with the Flash library.

---

Posted at 2020-05-04 by @MaBecker

but the issue still happened when I do the same actions

Please share snippet to reproduce it.

---

Posted at 2020-05-04 by user111618

I mean, I don't FOTA, so I would like to disable it on 01s(1mb board) and NodeMCU(12F). Now, I did it on 01s but failed on NodeMCU by using different firmware, but two firmware both added argument NO_FOTA=1.
Are there any methods to do this on NodeMCU? Thanks.

---

Posted at 2020-05-04 by user111618

To reproduce it, you need to:
(1).Get a string, length about 1000(I download string from WebSocket).
(2).

fs = require("Storage");
fs.erase("aName");
fs.write("aName", thatString);

(3). Code in .boot0:

if(fs.getFree()<4096) fs.compact();

if the free space did less than 4096 and compact() has run and reboot itself after compact, then run fs.compact(); again, if those actions right, you will lost all data in flash.

---

Posted at 2020-05-04 by user111618

So according to this issue, I guess flash must have an amount of the abandoned data before running compact, if not, the compact function will cause data loss.

---

Posted at 2020-05-04 by @MaBecker

To reproduce it, you need to:
fs = require("Storage");
fs.erase("aName");
fs.write("aName", thatString);
(3). Code in .boot0:
if(fs.getFree()<4096) fs.compact();

Well this is no code to reproduce the error.