Can I read each midi channel separately?

[Andrei-Tache]

Hi there, I'm looking to use fluidsynth in a 3d game and would need to read the audio data for each channel separately so I can play it in 3d space for spatial audio. Is this possible with FluidSynth?

I've tried muting all channels and unmuting one at a time (with fluid_synth_cc(channel, 7, 0)), then reading the data with fluid_synth_write_float(), but this creates distortions in the audio.

My current plan would be to use multiple synths with one instrument each, but this of course would mean more cpu/ram usage. Is there a better approach to this?

Thanks!

[Andrei-Tache]

I seem to have gotten it working with fluid_synth_process(), I can now get the data for each channel and there doesn't seem to be any cross-talk.

I don't know if this is the intended way to do this or if its stupid and I should do something different. If someone more knowledgeable could take a quick look I'd greatly appreciate it :)

Below is my code. (Apologies for using C#, I am running this in Unity)

IntPtr[] nativeBuffers;
GCHandle[] handles;
public float[][] chLeft;
public float[][] chRight;

void Start() {
        // Setup
        settings = FluidSynth.new_fluid_settings();
        FluidSynth.fluid_settings_setnum(settings, "synth.gain", 1.0);
        FluidSynth.fluid_settings_setint(settings, "synth.reverb.active", 0);
        FluidSynth.fluid_settings_setint(settings, "synth.chorus.active", 0);
        FluidSynth.fluid_settings_setint(settings, "synth.audio-groups", 16);
        FluidSynth.fluid_settings_setint(settings, "synth.audio-channels", 32);
        synth = FluidSynth.new_fluid_synth(settings);

        // Soundfont
        string sfPath = Application.streamingAssetsPath + "/Soundfont.sf2";
        soundFontID = FluidSynth.fluid_synth_sfload(synth, sfPath, 1);
        if (soundFontID == -1) {
            GameManager.LogMessage("Could not load SoundFont for FluidSynth.", true);
        }

        nativeBuffers = new IntPtr[32];
	handles = new GCHandle[32];
        chLeft = new float[16][];
        chRight = new float[16][];

        for (int ch = 0; ch < 16; ch++) {
            chLeft[ch] = new float[BUFFER_SIZE];
            chRight[ch] = new float[BUFFER_SIZE];

            handles[ch * 2] = GCHandle.Alloc(chLeft[ch], GCHandleType.Pinned);
            handles[ch * 2 + 1] = GCHandle.Alloc(chRight[ch], GCHandleType.Pinned);

            nativeBuffers[ch * 2] = handles[ch * 2].AddrOfPinnedObject();
            nativeBuffers[ch * 2 + 1] = handles[ch * 2 + 1].AddrOfPinnedObject();
        }
}

// Unity function called automatically to read audio data
void OnAudioFilterRead(float[] data, int channels) {
        // Clear previous audio data
        unsafe {
            for (int i = 0; i < 32; i++) {
                float* ptr = (float*)nativeBuffers[i];
                for (int j = 0; j < BUFFER_SIZE; j++) {
                    ptr[j] = 0f;
                }
            }
        }

        int frames = data.Length / channels;
        FluidSynth.fluid_synth_process(synth, frames, 0, null, 32, nativeBuffers);

        // Copy each channels' data to use outside this script
        unsafe {
            for (int ch = 0; ch < 16; ch++) {
                float* leftPtr = (float*)nativeBuffers[ch * 2];
                float* rightPtr = (float*)nativeBuffers[ch * 2 + 1];

                for (int i = 0; i < frames; i++) {
                    chLeft[ch][i] = leftPtr[i];
                    chRight[ch][i] = rightPtr[i];
                }
            }
        }

        // Playback (for testing)
        for (int i = 0; i < frames; i++) {
            float sampleL = 0f;
            float sampleR = 0f;

            for (int ch = 0; ch < 16; ch++) {
                sampleL += chLeft[ch][i];
                sampleR += chRight[ch][i];
            }

            if (channels == 2) {
                data[i * 2] = sampleL;
                data[i * 2 + 1] = sampleR;
            } else {
                data[i] = (sampleL + sampleR) * 0.5f;
            }
        }
}

void OnDestroy() {
        FluidSynth.delete_fluid_synth(synth);
        FluidSynth.delete_fluid_settings(settings);

        if (handles != null) {
            for (int i = 0; i < handles.Length; i++) {
                if (handles[i].IsAllocated) handles[i].Free();
            }
        }
}

[derselbst]

Fluid_synth_process is the way to do it. Pls do yourself a favor and only allocate a single native sample buffer as explained in the usage example: https://www.fluidsynth.org/api/fluidsynth_process_8c-example.html#a0 Audio channels should be 16, see https://www.fluidsynth.org/api/fluidsettings.xml#synth.audio-channels Your code is currently lacking a lot error handling (null pointer after new_fluid_*, fluid_failed). Since you seem to be doing custom audio panning in your app, consider only rendering either left or right channel by explicitly nulling the other buffer, or by mixing both channels onto the same buffer (should work but i cant remeber to have ever tested this). When your game is ready and publicly available, feel free to list it here: https://github.com/FluidSynth/fluidsynth/wiki/Applications If your game is closed source pls ship fluidsynth as shared library, https://github.com/FluidSynth/fluidsynth/wiki/LicensingFAQ Message ID: <FluidSynth/fluidsynth/repo-discussions/1577/comments/13360309@

…

github.com>

[Andrei-Tache]

Thank you for the feedback! I added the error handling and changed the channels to 16.

However, I'm not sure how I would allocate a single sample buffer, or how to mix the stereo channels into one buffer. I looked at the example code, but I'm not familiar with C code so its hard to fully understand how it works. Could you give me an example of how I would do this?

[ThierryBachmann]

Why not using a already made Unity package for this? There is a at least two free packages in the Unity store:
Maestro - Midi Player Tool Kit - Free based on a C# rewrite of fluidsynth.
Fluid MIDI Player based on fluidsynth binary.

[Andrei-Tache]

I tried these assets, but neither offer instrument spatialization afaik. I see that the pro version of your asset does support it; I might give it a try, but I would prefer to get just the synth working as I don't think I need the more advanced features.

As a side note, can I ask how you are routing the audio from fluid_synth_process to the audio sources? Are you using AudioClips to stream the data?

[ThierryBachmann]

Yes, AudioClip is the key.
All the Maestro sources are available. So, with the free version you will be able to understand how it works ... if you have a little patience. The same approach applies to the Pro version.
But, perhaps this forum is not the place to talk about it, the administrator may be unhappy. I encourage you going to the Maestro discord.

[derselbst]

The array of float pointers that you pass to fluid_synth_process can all point to different locations within the same sample buffer. The pointers can also alias each other. I suggest you start to understand the simple stereo use case followed by mono rendering before looking into multichannel rendering. I am currently only available via mobile and unable to provide c# examples. Im sure chatgpt would be helpful for you to understand it as well.

…

Message ID: <FluidSynth/fluidsynth/repo-discussions/1577/comments/13369303 @github.com>