Recording Sysex messages when a message is already in progress

[tloose]

I thought I had an issue with Sysex messages from my instrument, but I now believe the issue has to do with starting MIDI.Read(), which then invokes the SysexHandler DURING a SysEx message arriving at the MIDI In port. That seems to confuse the SysEx parser, and it stays confused until I reset.

Does anyone have any insights? What should I be doing at the start of recording a MIDI stream? I have tried making sure the MIDI input buffer is empty before starting recording.

I have not seen this issue with other MIDI messages, only SysEx...

[franky47]

So to be clear, you're starting reading MIDI while a SysEx has already started (ie: the SOX 0xF0 is missing), and the callback is still triggered at the end, with an incomplete message?

There's a proposal to handle SysEx separately with an improved parser, that should deal with this kind of case, feedback is welcome.

[tloose]

That is the case. I'm less concerned about the incomplete message, and more concerned about all the subsequent complete sysex messages also being parsed incorrectly. What method can I use to 'reset' the sysex parsing?

…

On Mon, Oct 23, 2023, 11:57 PM François Best ***@***.***> wrote: So to be clear, you're starting reading MIDI while a SysEx has already started (ie: the SOX 0xF0 is missing), and the callback is still triggered at the end, with an incomplete message? There's a proposal <#331> to handle SysEx separately with an improved parser, that should deal with this kind of case. — Reply to this email directly, view it on GitHub <#339 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT34RNPMK3DJ27L2BFFYTFLYA5DCHAVCNFSM6AAAAAA6MTN2PKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNRVGUZTG> . You are receiving this because you authored the thread.Message ID: <FortySevenEffects/arduino_midi_library/repo-discussions/339/comments/7365533 @github.com>

[franky47]

You should not need to do anything, I'm actually surprised that the callback gets triggered with an incomplete messages, a stray EOX (which would happen if the Arduino starts reading mid-SysEx) would usually be ignored, and the parser automatically reset.

There's a test case for this:

arduino_midi_library/test/unit-tests/tests/unit-tests_MidiInput.cpp

Lines 896 to 914 in 7e9222b

	TEST(MidiInput, strayEox)
	{
	// A stray End of Exclusive will reset the parser, but should it ?
	SerialMock serial;
	Transport transport(serial);
	MidiInterface midi((Transport&)transport);
	static const unsigned rxSize = 4;
	static const byte rxData[rxSize] = {
	0x8b, 42, 0xf7, 12
	};
	midi.begin(MIDI_CHANNEL_OMNI);
	serial.mRxBuffer.write(rxData, rxSize);
	EXPECT_EQ(midi.read(), false);
	EXPECT_EQ(midi.read(), false);
	EXPECT_EQ(midi.read(), false);
	EXPECT_EQ(midi.read(), false);
	}

Would you be able to record the MIDI stream without going through the parser (ie: just using Serial methods) and dump it here please?

[tloose]

I will continue my efforts to isolate exactly what is going on. I started out testing by using a Rodgers organ, but that is not convenient (it's 50+ miles away). I am testing now by using a second Arduino board with MIDI interface, then writing code to send MIDI sequences. I'll look at some of the test case MIDI sequences to use with the second board set to see what I can generate. My current test setup sends a 44 byte SysEx array five times with a single 0x00 between each SysEx message (I now understand the 0x00 is a mistake). If I get the timing between starting recording and starting the MIDI message I can get the Sysex parser out of sync. I need to better understand what is going on with this (and remove the stray 0x00 bytes); better control of the timing, etc. I appreciate your help and insight with this. It's been quite an interesting project.

…

On Tue, Oct 24, 2023 at 8:19 AM François Best ***@***.***> wrote: You should not need to do anything, I'm actually surprised that the callback gets triggered with an incomplete messages, a stray EOX (which would happen if the Arduino starts reading mid-SysEx) would usually be ignored, and the parser automatically reset. There's a test case for this: https://github.com/FortySevenEffects/arduino_midi_library/blob/7e9222b4634cdc510f7b76e4c356dfb25f69b174/test/unit-tests/tests/unit-tests_MidiInput.cpp#L896-L914 — Reply to this email directly, view it on GitHub <#339 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT34RNKNQMKJMN4OHPTPIBTYA657VAVCNFSM6AAAAAA6MTN2PKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNZQGI2TG> . You are receiving this because you authored the thread.Message ID: <FortySevenEffects/arduino_midi_library/repo-discussions/339/comments/7370253 @github.com>

-- TimL

[franky47]

While a stray 0x00 is indeed not spec-compliant, you could emulate the receiver starting mid-message by pausing the emitter after the 0xF0, plugging/resetting the receiver then resuming the emitter (with either a timer or a button).

MIDI only specifies the baudrate of 31250, not the actual time between bytes in a message.

[tloose]

Good idea.

…

On Tue, Oct 24, 2023, 10:24 AM François Best ***@***.***> wrote: While a stray 0x00 is indeed not spec-compliant, you could emulate the receiver starting mid-message by pausing the emitter after the 0xF0, plugging/resetting the receiver then resuming the emitter (with either a timer or a button). MIDI only specifies the baudrate of 31250, not the actual time between bytes in a message. — Reply to this email directly, view it on GitHub <#339 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT34RNLFQW5UPCKL6UWHO3DYA7MSTAVCNFSM6AAAAAA6MTN2PKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNZRGU2DS> . You are receiving this because you authored the thread.Message ID: <FortySevenEffects/arduino_midi_library/repo-discussions/339/comments/7371549 @github.com>

[tloose]

I have been able to reliably recreate the issue. I created a MIDI file of 56 consecutive SysEx messages, each 44 bytes long. I can send this to my device and record it reliably. If I send it, and stop the recording in the middle of one of the SysEx messages, the Sysex handler doesn't see the 0xF7 at the end, so it does not terminate the message properly and I end up with a bad MIDI file in the app. Also, (i presume) the remnants of the aborted SysEx message are still in the Sysex handler, and the *next* recording starts out poorly. I believe the SysEx handler sees the leading 0xF0 of the *next* Sysex message as part of the left-over message from the previous file. It thinks it has an 0xF0 ...... 0xF0 message. Then it seems the data bytes from this Sysex Message as extraneous, followed by the 0xF7 which should have terminated the message - which gets interpreted as an Escape sequence start. This "out of sync" behavior of the Sysex handler continues until something happens to stop it, even across multiple MIDI sequences being read into files. I think I need a way to reset the Sysex handler when I close the recorded MIDI file writing process. That way I can flush out the parsing of the SysEx message when I end recording a file, and ensure that the next recording starts out well initialized. Do you have any insights on this?

…

On Tue, Oct 24, 2023 at 11:50 AM Timothy Loose ***@***.***> wrote: Good idea. On Tue, Oct 24, 2023, 10:24 AM François Best ***@***.***> wrote: > While a stray 0x00 is indeed not spec-compliant, you could emulate the > receiver starting mid-message by pausing the emitter after the 0xF0, > plugging/resetting the receiver then resuming the emitter (with either a > timer or a button). > > MIDI only specifies the baudrate of 31250, not the actual time between > bytes in a message. > > — > Reply to this email directly, view it on GitHub > <#339 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AT34RNLFQW5UPCKL6UWHO3DYA7MSTAVCNFSM6AAAAAA6MTN2PKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNZRGU2DS> > . > You are receiving this because you authored the thread.Message ID: > <FortySevenEffects/arduino_midi_library/repo-discussions/339/comments/7371549 > @github.com> >

-- TimL