Optimized Background Noise Augmentation for Large Background Files

[PratikKulkar]

Proposed Algorithm:

Get duration of the background file (bg_file) in seconds.
Sample a random value from the range [0, bg_file_seconds - event_file_seconds).
Read the background file from sampled_value to sampled_value + event_file_seconds.

This approach ensures that:

We only load a portion of the background file required for the augmentation.
It maintains randomness in background selection while reducing memory overhead.
It is adaptable to cases with varied sample rates and event/background file durations.

Experiments and Results:

I’ve tested this algorithm using:

Event durations ranging from 1 to 9 seconds.
Background durations ranging from 81 to 10,000 seconds.
Sample rates: 16,000 Hz, 22,500 Hz, and 44,100 Hz.

This optimized approach significantly reduces memory usage while maintaining augmentation quality. I’ve attached the comparison plot showcasing the performance difference for your reference.

Improves scalability by avoiding unnecessary memory consumption for large files.
Enhances performance in real-time audio augmentation workflows.
Can be integrated as a feature or an option in AddBackgroundNoise to provide more flexibility to users.

Please let me know your thoughts on this proposal and if any further details or clarifications are needed.

In the below figure First plot shows difference between memory usage over the test_cases normalized for by 1e6 and next graph is time taken comparison of old vs proposed.

[iver56]

Hey :) Yes, this makes sense when the background noise files are quite long. The sound durations can be cached too, to reduce the amount of lookups, which may be relevant for people running spinning hard drives.

One complication is that it would not work for all file formats

[PratikKulkar]

Hey :) Yes, this makes sense when the background noise files are quite long. The sound durations can be cached too, to reduce the amount of lookups, which may be relevant for people running spinning hard drives.

One complication is that it would not work for all file formats

Hi @iver56,

Thank you for the response! I wanted to clarify your point about "caching sound durations"—could you please elaborate on what you mean by that? Are you suggesting caching the duration values in memory to avoid redundant file lookups during augmentation, especially on systems with slower disk I/O (like spinning hard drives)?

Regarding the file formats, I have tested librosa.get_duration() with several common audio formats, including .wav, .m4a, .aiff, .flac, .mp3, .ogg, and .wv. I noticed that only .m4a and .wv are not supported, as even librosa.load() doesn’t recognize them.

That said, if these unsupported formats were to appear in our dataset, we could consider converting them into compatible formats (e.g., .wav or .flac) during preprocessing to ensure smooth integration into the pipeline. Let me know if you think this would be a reasonable workaround or if there's a better alternative you recommend.

[iver56]

Are you suggesting caching the duration values in memory to avoid redundant file lookups during augmentation, especially on systems with slower disk I/O (like spinning hard drives)?

Yes 👍

That said, if these unsupported formats were to appear in our dataset, we could consider converting them into compatible formats (e.g., .wav or .flac) during preprocessing to ensure smooth integration into the pipeline. Let me know if you think this would be a reasonable workaround or if there's a better alternative you recommend.

One viable workaround could be to load the whole audio file into memory and then slice it

[PratikKulkar]

Are you suggesting caching the duration values in memory to avoid redundant file lookups during augmentation, especially on systems with slower disk I/O (like spinning hard drives)?

Yes 👍

That said, if these unsupported formats were to appear in our dataset, we could consider converting them into compatible formats (e.g., .wav or .flac) during preprocessing to ensure smooth integration into the pipeline. Let me know if you think this would be a reasonable workaround or if there's a better alternative you recommend.

One viable workaround could be to load the whole audio file into memory and then slice it

Thank you for the confirmation, @iver56!

Regarding caching, I believe caching typically works best when access patterns are predictable or repetitive. Since we're randomly selecting background segments during augmentation, caching might not yield significant benefits. I also haven't extensively used caching mechanisms in Python, but an alternative solution could involve storing durations in a dictionary for quick lookups.

One concern I have is the key size, as using full file paths as keys might impact performance. A possible workaround could be mapping files to integers, like {num -> bgfile}. We could then sample a random index (e.g., random.randint(0, len(bg_files))) to select a background file. Simultaneously, a separate dictionary {num -> seconds} would store the corresponding durations for quick retrieval. This approach might balance memory usage and performance better.

As for unsupported formats, a viable option is to load the entire audio file into memory and slice it as needed. However, we would need to ensure the libraries we use support those specific formats (like .m4a and .wv). If you have any suggestions for libraries that handle these formats well, I’d love to explore them.

Let me know if you think this approach sounds reasonable or if I should start implementing one of the solutions mentioned!

Best regards,
Pratik

[PratikKulkar]

Hello @iver56,
I have implemented caching for time information using a dictionary, with the file name as the key. I am attaching some test case results for your review. Could you please take a look and suggest any improvements?

[iver56]

Thanks for the update, and for the patience! I haven't forgotten about this, and will check it out tomorrow