Capture latency

[M1cha]

Thank you for this awesome project 👍

I'm currently using it to capture USB audio data, so I can use headphones on my Nintendo Switch while also streaming the audio (and video) to the PC. This works quite well but there's a pretty constant latency of about 250ms. I can easily deal with that by offsetting the video by that amount in OBS, but was wondering if you know where this might come from and how far down this could be optimized.

[ataradov]

I'm not sure what you are trying to do. Why do you need it synchronized? Would not audio be a constant stream? How do you know it is out of sync?

Generally, there was no attempt to optimize capture latency, there are many levels of buffering in the hardware and software. I'm surprised you get anything consistent there.

[M1cha]

I'm capturing video through an hdmi capture card, but audio through the usb-sniffer (which sniffes a usb-wireless audio dongle). This circumvents the issue, that The console can't output audio via HDMI and USB at the same time.

I wrote all of the software running on the pc myself (https://github.com/M1cha/usbaudio-sniffer/tree/main), so I'm mainly wondering if you have any pointers as to where I'd have to look inside the firmware (or even the fpga?).

[ataradov]

I don't really know where a consistent delay may come from here. It may be just the buffering in the bulk transfers on the PC.

On the FPGA side, it pushes the data out into FX2LP as fast as it would accept them. If you read stuff from USB fast enough, there should be no latency here.

There will be some delay in the FX2LP buffering, but that would result in jerky transfers. It accumulates the data until there is enough to fill a full endpoint size. But this jerkiness is likely way smaller than 250 ms.

[M1cha]

The issue was 100% in my code. The mutex on the data queue made the audio side wait all the time because the usb side writes to it at a high rate in many small chunks.

I switched to a different way for passing the data and the latency is effectively 0 now (compared to the capture card). It's actually quite amazing given that you said there was no attempt to optimize capture latency 😅 . Again, awesome work.

[ataradov]

There was no attempt to focus on latency, but here was a lot of effort to push the data out as fast as feasibly possible, since hardware buffers are pretty small. I guess this automatically results in low latency hardware. The most of the delays happen on the PC where buffers are big and parsing takes time. But it did not matter, since Wireshark UI is the slow part in normal use.