[sqlite] Block I/O

[Lauconaug]

Q	A
BC Break	yes/no
Version	x.y.z

Support Question

Hello, my magnetico container is using a lot of I/O. Is this normal?
Here's the information I got after running the container for 16 hours:
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
magnetico 57.14% 507.4MiB / 512MiB 99.11% 20.9GB / 8.56GB 2.03TB / 2.68TB 13

Regards

[tgragnato]

Hi @Lauconaug!
Only SQLite can be responsible for block I/O in magnetico, so I’m assuming you are using it for data persistence.

For every result, the crawler checks the database for the existence of its infohash and inserts the new data only if it is not already present. I'm also assuming that, due to a low memory limit, magnetico is unable to cache the dataset's infohashes. As a result, the SQLite routines perform an index lookup with cache misses for most incoming crawl results from the DHT network. This leads to frequent disk access and increased block I/O.

To address this issue, you could allocate more memory to the container to allow caching of a larger portion of the dataset in RAM. This would help reduce the frequency of disk I/O operations caused by cache misses.

If the dataset is expected to grow significantly, you might also consider migrating to a more scalable database solution. Options like PostgreSQL or CockroachDB can handle larger datasets with higher throughput when configured properly.