Faster Import?

[zaptrem]

I have to import almost 400,000 tracks, and it appears that doing so will take forever and a day. I tried using GNU Parallel to speed up the process but it got even slower (multiple processes fighting over the SQLite lock maybe)? Is there any other way to speed up large imports?

[sampsyo]

Hello! This is kind of a wild guess, but I think the bottleneck for most imports is storage (i.e., reading the music files from your disk). The importer is multithreaded, so it's already overlapping computation with network communication and disk I/O. So one thing to remember is that you can let things "run ahead" while you wait at the prompt and it will do a lot of the I/O work in the background. Beyond this, there may not be much that can be done—except disabling plugins you don't need (principally Chromaprint fingerprinting).

[zaptrem]

As I’m importing a ton of tracks for ML use I disabled the prompting and told it to skip things it’s unsure about (weirdly I’m surprised it continued to skip things when Chromaprinting should have been able to find them). Disk activity was extremely low and CPU utilization was also pretty low.

Also, it seems you’re a professor at the same university in which I’m a student! Small world.

[sampsyo]

How about that!

I don't have an immediate diagnosis if you're seeing low CPU and disk utilization. Not sure where the bottleneck is in that case, but one strong contender is waiting on network responses from MusicBrainz and Acoustid (both of which are rate-limited). You might consider running your own MusicBrainz mirror if that's the issue.