Playback of multiple offset audio files with playback speed and seek capabilities

[NickeManarin]

I have an editor in which the user can insert audio files.
The audio files can have different lengths and be inserted at distinct positions in time.

Initialization:

public EditorViewModel() 
{
    _outputDevice = new WaveOutEvent();
    _mixer = new MixingSampleProvider(WaveFormat.CreateIeeeFloatWaveFormat(44100, 2))
    {
        ReadFully = true
    };
    _bufferedWaveProvider = new BufferedWaveProvider(_mixer.WaveFormat);
    _outputDevice.Init(_bufferedWaveProvider);
}

This code is a working in progress, I'm still not sure if it will work:

private void MixAudioTracks()
{
    _mixer.RemoveAllMixerInputs();

    foreach (var track in Tracks)
    {
        var inputStream = new AudioFileReader(track.AudioPath)
        {
            Volume = track.Volume
        };
        var offsetStream = new OffsetSampleProvider(inputStream.ToSampleProvider())
        {
            DelayBy = TimeSpan.FromTicks(track.StartTime),
            SkipOver = TimeSpan.FromTicks(track.StartTimeOffset),
            Take = TimeSpan.FromTicks(track.Length - track.EndTimeOffset)
        };

        //Used to slowdown or speed up the audio.
        var sampleChannel = new SmbPitchShiftingSampleProvider(offsetStream) 
        {
             PitchFactor = track.SpeedFactor
        };
        _mixer.AddMixerInput(sampleChannel);
    }

    var floatBuffer = new float[_bufferedWaveProvider.BufferLength / 4];
    var bytesRead = _mixer.Read(floatBuffer, 0, floatBuffer.Length);

    //Convert float array to byte array
    var byteBuffer = new byte[bytesRead * 4]; //4 bytes per float
    Buffer.BlockCopy(floatBuffer, 0, byteBuffer, 0, byteBuffer.Length);

    _bufferedWaveProvider.AddSamples(byteBuffer, 0, byteBuffer.Length);
}
    
public void Play() 
{
   _outputDevice.Play();
}
    
public void Pause() 
{
   _outputDevice.Pause();
}
    
public void Seek(long position)
{
    //How to seek?
}

I need to combine the audio tracks based on their start, playback speed and length and be able to seek to a specific position in time.
Is there any solution for this?

[NickeManarin]

The code kind of works, I just need to understand how to provide the samples to the BufferedWaveProvider as needed.
I don't want to read all audio into memory every time, but I'm not sure how to integrate with the mixer and OffsetSampleProvider.

Maybe I have to create a BufferedWaveProvider for each audio source and then pass to the OffsetSampleProvider ?
Or maybe I'll add all the logic into my

Ofc, no seek functionality yet.

---

EDIT

I was able to sort of implement a seek functionality and buffer fill, but the issue is that the mixer is clearing its input automatically after the length of the audio is reached, even if I seek its audio sources to 00:00.

The MixAudioTracks method is similar, the difference is that I'm storing the AudioFileReader streams so that I can seek the time later.
It would be better if the MixingSampleProvider had its own CurrentTime or that it was possible to access the parent SampleProvider.

Right now the only way to access the AudioFileReader is to store separately.

private void MixAudioTracks()
{
    _mixer.RemoveAllMixerInputs();
    _audioFileReaders = [];

    foreach (var segment in Tracks)
    {
        var inputStream = new AudioFileReader(track.ContentCachePath)
        {
            Volume = track.Volume
        };
        var offsetProvider = new OffsetSampleProvider(inputStream.ToSampleProvider())
        {
            DelayBy = TimeSpan.FromTicks(track.StartTime),
            SkipOver = TimeSpan.FromTicks(track.StartTimeOffset),
            Take = TimeSpan.FromTicks(track.Length - track.EndTimeOffset)
        };
        var pitchProvider = new SmbPitchShiftingSampleProvider(offsetProvider)
        {
            PitchFactor = segment.SpeedFactor / (float)SpeedFactor
        };

        _audioFileReaders.Add(inputStream);

        _mixer.AddMixerInput(pitchProvider);
    }

    FillBuffer(true);
}

For example, here when seeking the source audio files, I need to know start/length and trim/offsets of each file separately:

private void SeekAudio()
{
    foreach (var fileReader in _audioFileReaders)
    {
        fileReader.CurrentTime = TimeSpan.FromTicks(CurrentRenderTime); //TODO: Adjust with track Start and Offsets
    }

    FillBuffer(true);
}

It would be nice if the buffer was auto-fill. Why do I need to manually do this?
Maybe there's a way for the mixer to know when to request samples?

private void FillBuffer(bool force = false)
{
    //Not sure if right: only fill buffer if necessary.
    if (!force && _lastSampleTime >= 0 && 
        (_lastSampleTime < CurrentTime || _lastSampleTime + _bufferedWaveProvider.BufferDuration.Ticks < CurrentTime))
        return;

    //if (force || CurrentTime < _lastSampleTime)
    _bufferedWaveProvider.ClearBuffer();

    var floatBuffer = new float[_bufferedWaveProvider.BufferLength / 4];
    var bytesRead = _mixer.Read(floatBuffer, 0, floatBuffer.Length);

    //Convert float array to byte array
    var byteBuffer = new byte[bytesRead * 4]; //4 bytes per float
    Buffer.BlockCopy(floatBuffer, 0, byteBuffer, 0, byteBuffer.Length);

    _bufferedWaveProvider.AddSamples(byteBuffer, 0, byteBuffer.Length);

    //Update the last sample timestamp.
    _lastSampleTime = CurrentTime;
}

Then when seeking or during playback, I need to call the FillBuffer();

private void PlaybackLoop()
{
    //Pseudo-code

     //If playback looped to start:
     CurrentTime = 0;
     SeekAudio();

   
   CurrentTime += 100;
   //Else
   FillBuffer();
}

[NickeManarin]

I now have a working code, but I'm not sure if it's the best way to do it.
It doesn't look that good because the buffer load is not in sync and audio gets cut off or a silence is inserted.

public ExporterViewModel()
{
    _outputDevice = new WaveOutEvent();
    _mixer = new MixingSampleProvider(WaveFormat.CreateIeeeFloatWaveFormat(48000, 2))
    {
        ReadFully = true
    };
    _bufferedWaveProvider = new BufferedWaveProvider(_mixer.WaveFormat)
    {
        BufferDuration = TimeSpan.FromSeconds(5)
    };
    _outputDevice.Init(_bufferedWaveProvider);
}

This method must be called on seek:

    private void MixAudioTracks()
    {
        _mixer.RemoveAllMixerInputs();

        foreach (var track in Tracks)
        {
            foreach (var segment in track.Segments)
            {
                //If audio segment is out of range, ignore it.
                if (CurrentTime > segment.StartTime + segment.Length)
                    continue;

                var inputStream = new AudioFileReader(segment.AudioPath)
                {
                    Volume = segment.Volume, //0f to 1f
                };

                var offsetProvider = new OffsetSampleProvider(inputStream.ToSampleProvider());

                //Audio segment needs to be played at the exact time, based on its position, length, trims/offset
                if (segment.StartTime < CurrentTime)
                {
                    offsetProvider.SkipOver = TimeSpan.FromTicks(CurrentTime - segment.StartTime + segment.StartTimeOffset);
                    offsetProvider.Take = TimeSpan.FromTicks(segment.Length - segment.EndTimeOffset - CurrentTime - segment.StartTime);
                }
                else
                {
                    offsetProvider.DelayBy = TimeSpan.FromTicks(segment.StartTime - CurrentTime);
                    offsetProvider.SkipOver = TimeSpan.FromTicks(segment.StartTimeOffset);
                    offsetProvider.Take = TimeSpan.FromTicks(segment.Length - segment.EndTimeOffset);
                }

                var pitchProvider = new SmbPitchShiftingSampleProvider(offsetProvider)
                {
                    PitchFactor = segment.SpeedFactor / PreviewerSpeedFactor
                };

                _mixer.AddMixerInput(pitchProvider);
            }
        }
        
        FillBuffer(true);
    }

The fill buffer is always called on each playback tick, but only refills the buffer if necessary.
I'm using the arbitrary 10_000_000 ticks time reduction to load the buffer before it's needed, or else there's an audible silence.
But this makes the audio to be cut a bit. I'm not sure how to fix this issue yet.

    private void FillBuffer(bool force = false)
    {
        //Only fill buffer if forced, never filled, current time in past or current time past buffer length.
        if (!force && _lastSampleTime >= 0 && (_lastSampleTime > CurrentRenderTime ||
            _lastSampleTime + _bufferedWaveProvider.BufferDuration.Ticks - 10_000_000 > CurrentTime))
            return;

        _bufferedWaveProvider.ClearBuffer();

        var floatBuffer = new float[_bufferedWaveProvider.BufferLength / 4];
        var bytesRead = _mixer.Read(floatBuffer, 0, floatBuffer.Length);

        //Convert float array to byte array
        var byteBuffer = new byte[bytesRead * 4]; //4 bytes per float
        Buffer.BlockCopy(floatBuffer, 0, byteBuffer, 0, byteBuffer.Length);

        _bufferedWaveProvider.AddSamples(byteBuffer, 0, byteBuffer.Length);

        //Update the last sample timestamp.
        _lastSampleTime = CurrentTime;
    }

On my playback tick, the FillBuffer() and MixAudioTracks are called accordingly:

private void PlaybackLoop()
{
    if (CurrentTime >= EndTime)
    {
        CurrentTime = 0;
        MixAudioTracks();
        return;
    }

    CurrentTime += framerateTicks; //Advances time based on framerate.
    FillBuffer();
}