Skip to content

Generalize DeviceInterface to include audio decoding #926

@scotts

Description

@scotts

Currently, our DeviceInterface is implicitly a video device interface. Inside SingleStreamDecoder, we dispatch differently based on audio and video:

if (streamInfo.avMediaType == AVMEDIA_TYPE_AUDIO) {
convertAudioAVFrameToFrameOutputOnCPU(avFrame, frameOutput);
} else {
deviceInterface_->convertAVFrameToFrameOutput(
avFrame, frameOutput, preAllocatedOutputTensor);
}

Ideally, we'd like to turn that line into just:

deviceInterface_->convertAVFrameToFrameOutput(
        avFrame, frameOutput, preAllocatedOutputTensor);

That is, we handle audio and video the same. In order to do that, we need to somehow get SingleStreamDecoder::convertAudioAVFrameToFrameOutputOnCPU into a device interface. Design questions:

  1. Should we extend CpuDeviceInterface to handle audio, in which case we'd dispatch inside of it?
  2. Or should we keep CpuDeviceInterface to be just video only, and create a new kind of device interface that is just audio?

I think 1 might be the better option, but I'm not sure. I'm confident we'll only ever do audio decoding on the CPU.

Metadata

Metadata

Assignees

No one assigned

    Labels

    refactorImproves code itself, but does not fix a bug or add new functionality.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions