Very slow when ripping multiple CDs at once due to performance bug in sox_ng

[dseomn]

While trying to rip 5 CDs at the same time, my CPU seems to be a bottleneck. Looking at top, sox seems to using the vast majority of the CPU time, and the Calculating peak level step is taking a while. The CPU is an Intel i5-11600 (6 cores, released in 2021), so I was surprised that that was a bottleneck for ripping CDs.

Looking at the documentation and code, it looks like the only thing the peak level is used for is to discard silent tracks 0. Would it make sense to add an option to skip peak detection on tracks 1+ to remove the CPU bottleneck?

[github-actions]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing instructions.

[MerlijnWajer]

I think we also write the peak values to the log files don't we? I think this might be one of those things that might be required by certain trackers, so I am not sure if it makes sense to make it optional, but could be convinced otherwise.

[MerlijnWajer]

Also, mostly just as an amusing aside, but before this change you wouldn't have been able to rip multiple CDs in parallel at any reasonable speed at all: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=51a858817dcdbbdee22cb54b0b2b26eb145ca5b6

[dseomn]

I think we also write the peak values to the log files don't we? I think this might be one of those things that might be required by certain trackers, so I am not sure if it makes sense to make it optional, but could be convinced otherwise.

Oh, I didn't realize that file was used for anything other than a human to look at what the program did. I don't usually rip that much at once, and maybe it's also a performance bug in sox, so feel free to close this if nobody else is having trouble with it.

Also, mostly just as an amusing aside, but before this change you wouldn't have been able to rip multiple CDs in parallel at any reasonable speed at all: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=51a858817dcdbbdee22cb54b0b2b26eb145ca5b6

Huh, that's remarkably recent given how long Linux has been able to use SCSI-like CD-like drives. From that commit description it's not clear to me whether the lock means one drive can read at a time, or one drive can be sent an asynchronous read command at a time but multiple async commands can be in flight. (That's not a part of the kernel I've looked at before.)

[MerlijnWajer]

Oh, I didn't realize that file was used for anything other than a human to look at what the program did. I don't usually rip that much at once, and maybe it's also a performance bug in sox, so feel free to close this if nobody else is having trouble with it.

I think we can leave this open and see what folks think, just wanted to give my two cents.

Huh, that's remarkably recent given how long Linux has been able to use SCSI-like CD-like drives. From that commit description it's not clear to me whether the lock means one drive can read at a time, or one drive can be sent an asynchronous read command at a time but multiple async commands can be in flight. (That's not a part of the kernel I've looked at before.)

Prior to that patch it was possible to rip from several CD-like drives, but if a read command on one drive would hang a long time before returning (this can happen a lot with bad disks, and cdparanoia will hammer the bad disks especially) all the other drives would not be able to read anything until the call returned.

[dseomn]

This finally bugged me enough to do some testing, and I'm almost positive the performance issue is a bug somewhere other than whipper. I filed https://sourceforge.net/p/sox/bugs/382/ where running sox 3 times in series took 0.379s but the same commands in parallel took 38.170s. I don't know if it's a bug in sox, the kernel's CPU scheduler, or a defect in my hardware, or something else, but it doesn't seem like a problem in whipper.

[MerlijnWajer]

Thank you for following up (curious to see what turns out to be the bug here in sox), shall we keep the issue open in case others find that they might want this feature?

[dseomn]

So I made a mistake there. Apparently Debian switched from sox to sox_ng. Just filed https://codeberg.org/sox_ng/sox_ng/issues/826 there. When I searched to make sure there wasn't already an open issue, I found https://codeberg.org/sox_ng/sox_ng/issues/464 which seems to be key to the bug.

If they fix the bug quickly, then closing this is fine. If not, would it be possible for whipper to add the --single-threaded flag? Or maybe close this now, and I'll open a new request for that flag later if the bug isn't fixed another way?

[MerlijnWajer]

I think we could add that flag given that it seems to be supported by the original sox implementation as well.

[dseomn]

From the sox_ng bug, setting SOX_OPTS=--single-threaded in the environment works too. Between that as an easy workaround and the progress they seem to be making on figuring out the bug, I think it probably makes sense to just close this bug. Thanks for the help! And I changed the issue title to hopefully make it easier for others to find the workaround if they have the same problem.