NVS Garbage Collection Continuously Running While Logging Large Amounts of Data

[Ananyaaynana]

I'm encountering an issue where the NVS (Non-Volatile Storage) is continuously performing garbage collection when I attempt to log a large number of IDs to the flash memory using the NVS API.

Description:
I'm using the NVS API nvs_write to store a large number of IDs in the flash memory. Once a sector becomes full, NVS triggers garbage collection as expected. However, the garbage collection process seems to be running perpetually, even after the initial sector is cleared.

App.overlay Partition Information:

&flash0 { status = "okay"; partitions { compatible = "fixed-partitions"; #address-cells = <1>; #size-cells = <1>; queue_partition: partition@276000 { label = "queue"; reg = <0x00276000 0x0180000>; }; }; };

Question: Is it possible to store a large number of IDs in flash memory using NVS without encountering continuous garbage collection? Why is this happening?
Environment:
Board: esp32c3_devkitm
OS: Linux

[Laczen]

Hi @Ananyaaynana, thank you for reporting this.

From your overlay file I would conclude that you are using nvs on a 720kB partition. This is huge for nvs. On your device I think the sectors are 4kB and there are 180 sectors. With the standard setup of nvs you will get 179 sectors of useable space for item storage (both data and ate's). When you keep adding data nvs will end up in the latest sector and then walk over all sectors to see if any data has been deleted, if there has been no data deleted nvs will erase the last sector and rewrite all data from the first sector. Then it checks if there is sufficient space to add the new data and writes it, if there is insufficient space it will continue to the next sector and try again. As you are using 180 sectors and if there is no data deleted this can take a long time (it will stop after 180 tries if there is nothing deleted, returning that there is no space).

I think you are correctly reporting that it "seems" to be running perpetually.

It would be good to add how many items are stored, what the size of the items is and if any of them are deleted.

You can reduce the number of erases and the time spend in erase by increasing the sector size (and reducing the sector count). In your case a good choice would be a sector size of 32kB, this will require changes to the size as well.

[Laczen]

@Ananyaaynana, from your issue #74984 it seems that you are trying to make a buffer for storing data that will later on be processed. Using a new id for each data entry is not the best solution. As NVS treats each id as equally important it will soon end up in a fully used flash storage.

A simpler method is to store all the data under the same id and keep track of how many items you need to process as a separate id. Each time a item is stored the " buffer size" is increased and each time a item is processed the "buffer size" is decreased. Using the routine nvs_read_hist reading back the older data is possible (there is no need to reed further than the buffer size).
Doing it this way will avoid copying data that is no longer required and old data is automatically removed.

[Laczen]

@Ananyaaynana do you need more help with this question?

[Ananyaaynana]

I implemented it using FCB as having a large number of ID's was not possible.

[Laczen]

I implemented it using FCB as having a large number of ID's was not possible.

That is also OK.

[Laczen]

The problem was resolved by using a different storage method/solution.