Allow sending larger SysEx messages

[probonopd]

Cartridge dumps are larger than we can send out over MIDI. Currently,

• With 512 bytes of data, it WORKS both over USB MIDI and rtpMIDI
• With 514 bytes, it WORKS over rtpMIDI but NOT over USB MIDI
• With 1024 bytes, it WORKS over rtpMIDI but NOT over USB MIDI

It seems like we need to break down ("chunk") larger SysEx messages into parts.

However, it appears that Circle refuses to send "incomplete" SysEx messages.

Reference:

• Sending large and sending chunked MIDI messages fails rsta2/circle#563

//
// midichunker.h
//
// MiniDexed - Dexed FM synthesizer for bare metal Raspberry Pi
// Copyright (C) 2022-25  The MiniDexed Team
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <http://www.gnu.org/licenses/>.
//

#pragma once
#include <cstddef>
#include <vector>
#include <cstdint>

class MIDISysExChunker {
public:
    MIDISysExChunker(const uint8_t* data, size_t length, size_t chunkSize = 256);
    bool hasNext() const;
    std::vector<uint8_t> next();
    void reset();
private:
    const uint8_t* m_data;
    size_t m_length;
    size_t m_chunkSize;
    size_t m_offset;
};

//
// midichunker.cpp
//
// MiniDexed - Dexed FM synthesizer for bare metal Raspberry Pi
// Copyright (C) 2022-25  The MiniDexed Team
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <http://www.gnu.org/licenses/>.
//

#include "midichunker.h"
#include <algorithm>
#include <circle/logger.h>

static const char* From = "midichunker";

MIDISysExChunker::MIDISysExChunker(const uint8_t* data, size_t length, size_t chunkSize)
    : m_data(data), m_length(length), m_chunkSize(chunkSize), m_offset(0) {}

bool MIDISysExChunker::hasNext() const {
    return m_offset < m_length;
}

std::vector<uint8_t> MIDISysExChunker::next() {
    if (!hasNext()) return {};
    size_t remaining = m_length - m_offset;
    size_t chunkLen = std::min(m_chunkSize, remaining);
    // Only the last chunk should contain the final 0xF7
    if (m_offset + chunkLen >= m_length && m_data[m_length-1] == 0xF7) {
        chunkLen = m_length - m_offset;
    } else if (m_offset + chunkLen > 0 && m_data[m_offset + chunkLen - 1] == 0xF7) {
        chunkLen--;
    }
    LOGNOTE("Chunker: m_offset=%d, chunkLen=%d, remaining=%d, m_length=%d, m_chunkSize=%d", (int)m_offset, (int)chunkLen, (int)remaining, (int)m_length, (int)m_chunkSize);
    std::vector<uint8_t> chunk(m_data + m_offset, m_data + m_offset + chunkLen);
    m_offset += chunkLen;
    return chunk;
}

void MIDISysExChunker::reset() {
    m_offset = 0;
}

[soyersoyer]

Since MiniDexed uses the ~latest Circle, this can also be used:
rsta2/circle@4fd36a7

[probonopd]

Could you have a try at integrating it? I'd appreciate it, since I could never get mine to work properly.

[soyersoyer]

Weird, for me, it can send 2048-Byte SysEx without any changes over USB. It's just important to have a valid SysEx message.
But it also works with 4, 128 and 512 chunk size.

[soyersoyer]

    TDeviceMap::const_iterator Iterator = s_DeviceMap.find(deviceName);
    if (Iterator != s_DeviceMap.end()) {
        Iterator->second->Send(voicedump, sizeof(voicedump), nCable);
        LOGDBG("Send SYSEX voice dump %u to \"%s\"", nVoice, deviceName.c_str());
    } else {
        LOGWARN("No device found in s_DeviceMap for name: %s", deviceName.c_str());
    }

I think instead of the above, you can write only this:

Send(voicedump, sizeof(voicedump), nCable);
LOGDBG("Send SYSEX voice dump %u to \"%s\"", nVoice, deviceName.c_str());

Send is a virtual function, and it is overloaded in CMIDIKeyboard.