"escape char" for text console?

[mfp20]

Hi,

I need one escape char to send arbitrary binary data to text console. So that current parser can do something like: "first char in this line equals to 'my special exape char'? Peek second char, that's the amount of more chars to receive, and after amount is received call my_custom_parser(amount, dataptr)".

The purpose is to receive serialized blocks and segments; it's COBS binary data.

Which char isn't already in use?

[terjeio]

Which char isn't already in use?

Only CR and LF is used in the ASCII control character range, and no top bit set characters. The standard way of sending messages back is by enclosing them in square brackets and prefixing data, e.g [BD:<data>] CR LF Of course ], CR and LF cannot be present in the <data>.

Top bit set printable characters in ISO8859 encodings should not be used - they could be part of feedback messages in gcode.

[mfp20]

I don't understand. I'm looking at protocol_main_loop(), it receives the incoming chars one by one, and on CR/LF it sends the line to gcode execute. There are some special chars tough: '/', '$', '[', '(', and ';'.

Anyway, I see in stream.h some control chars defined:

#define ASCII_SOH  0x01
#define ASCII_STX  0x02
#define ASCII_ETX  0x03
#define ASCII_EOT  0x04
#define ASCII_ACK  0x06
#define ASCII_BS   0x08
#define ASCII_TAB  0x09
#define ASCII_LF   0x0A
#define ASCII_CR   0x0D
#define ASCII_XON  0x11
#define ASCII_XOFF 0x13
#define ASCII_NAK  0x15
#define ASCII_EOF  0x1A
#define ASCII_CAN  0x18
#define ASCII_EM   0x19
#define ASCII_ESC  0x1B
#define ASCII_DEL  0x7F
#define ASCII_EOL  "\r\n"

Is it possible to use those to make an 'out of band' binary message? I mean stuff that doesn't end up echoed and parsed in console.

[terjeio]

to send arbitrary binary data to text console

I read this as output from the controller ending up in a sender or a terminal app.

I'm looking at protocol_main_loop()

This is processing input. If you meant input to the controller then you can "hook into" the function pointer unknown control characters are passed over. I have done this in the YModem code that traps ASCII_SOH to trigger reception by taking over the input stream.

[mfp20]

I'm looking at protocol_main_loop()

This is processing input. If you meant input to the controller then you can "hook into" the function pointer unknown control characters are passed over. I have done this in the YModem code that traps ASCII_SOH to trigger reception by taking over the input stream.

I see, cool, exactly what I was searching for. But I can't re-use ASCII_SOH if you have already used that. We should implement a control message parser.

BTW, I can't find where grbl.on_unknown_realtime_cmd gets called. It isn't in protocol_main_loop() and anywhere else in core code.

[terjeio]

But I can't re-use ASCII_SOH if you have already used that.

Well, I do not see many enabling YModem support - more a coding exercise and to show what can be done without changing any code in the core. And there are other control characters that are currently unused to choose from.

I can't find where grbl.on_unknown_realtime_cmd gets called. It isn't in protocol_main_loop() and anywhere else in core code.

core/protocol.c

Lines 857 to 860 in 6f5d1cd

	default:
	if(c < ' ' || (c >= 0x7F && c <= 0xBF))
	drop = grbl.on_unknown_realtime_cmd == NULL || grbl.on_unknown_realtime_cmd(c);
	break;

[mfp20]

But I can't re-use ASCII_SOH if you have already used that.

Well, I do not see many enabling YModem support - more a coding exercise and to show what can be done without changing any code in the core. And there are other control characters that are currently unused to choose from.

Practically yes. But in theory we should use those codes to implement one single "smart terminal"; that's the purpose they were originally protocolled for.

I can't find where grbl.on_unknown_realtime_cmd gets called. It isn't in protocol_main_loop() and anywhere else in core code.

core/protocol.c

Lines 857 to 860 in 6f5d1cd

	default:
	if(c < ' ' || (c >= 0x7F && c <= 0xBF))
	drop = grbl.on_unknown_realtime_cmd == NULL || grbl.on_unknown_realtime_cmd(c);
	break;

Yeah, thanks, but I saw that in protocol_enqueue_realtime_command(), that in turn hal.stream.enqueue_realtime_command = protocol_enqueue_realtime_command, but hal.stream.enqueue_realtime_command() is a different things. Looks like it is called to issue a realtime command in stream.c

[terjeio]

But in theory we should use those codes to implement one single "smart terminal"; that's the purpose they were originally protocolled for.

I have to admit I do not understand the meaning of this statement. What I have done is that the number of command characters has been expanded on (over those supported by legacy Grbl) and I have provided a way for plugin code to add further expansion. My gcode sender takes advantage of the new command codes to provide enhanced functionality, other senders may not. E.g. my sender has code to transfer files via the YModem protocol for storage on a local SD Card.

hal.stream.enqueue_realtime_command

This is gone in the latest version and replaced with hal.stream.set_enqueue_rt_handler(). The related function pointer now lives locally in the stream implementations. All received characters are passed through the related function pointer and it normally set to point to protocol_enqueue_realtime_command(). protocol_enqueue_realtime_command() picks off known command characters and submits them for processing, and unknown command characters are passed over grbl.on_unknown_realtime_cmd() if set.

So, to bypass protocol_enqueue_realtime_command() processing hal.stream.set_enqueue_rt_handler() has to be called with a replacement function, typically one that does not strip any characters. stream_buffer_all() is provided by the core for that purpose. This is done in the YModem implementation as it is a binary protocol - and all characters are needed for that to function.

[terjeio]

Here is the cost of trapping YModem SOH:

0801d6dc <trap_initial_soh>:

// Check input stream for file YModem start of header (soh) characters.
// Redirect input stream and start protocol handler when found.
static bool trap_initial_soh (char c)
{
    if(c == ASCII_SOH || c == ASCII_STX) {
 801d6dc:	1e43      	subs	r3, r0, #1
 801d6de:	2b01      	cmp	r3, #1
{
 801d6e0:	b570      	push	{r4, r5, r6, lr}
    if(c == ASCII_SOH || c == ASCII_STX) {
 801d6e2:	d830      	bhi.n	801d746 <trap_initial_soh+0x6a>

        memcpy(&active_stream, &hal.stream, sizeof(io_stream_t));           // Save current stream pointers,
 801d6e4:	4e1c      	ldr	r6, [pc, #112]	; (801d758 <trap_initial_soh+0x7c>)
 801d6e6:	4c1d      	ldr	r4, [pc, #116]	; (801d75c <trap_initial_soh+0x80>)
 801d6e8:	f106 05a4 	add.w	r5, r6, #164	; 0xa4
 801d6ec:	cd0f      	ldmia	r5!, {r0, r1, r2, r3}
 801d6ee:	c40f      	stmia	r4!, {r0, r1, r2, r3}
 801d6f0:	cd0f      	ldmia	r5!, {r0, r1, r2, r3}
 801d6f2:	c40f      	stmia	r4!, {r0, r1, r2, r3}
 801d6f4:	cd0f      	ldmia	r5!, {r0, r1, r2, r3}
 801d6f6:	c40f      	stmia	r4!, {r0, r1, r2, r3}
 801d6f8:	e895 000f 	ldmia.w	r5, {r0, r1, r2, r3}
 801d6fc:	e884 000f 	stmia.w	r4, {r0, r1, r2, r3}
        rt_handler = hal.stream.set_enqueue_rt_handler(stream_buffer_all);  // stop core real-time command handling and
 801d700:	f8d6 30c4 	ldr.w	r3, [r6, #196]	; 0xc4
 801d704:	4816      	ldr	r0, [pc, #88]	; (801d760 <trap_initial_soh+0x84>)
        hal.stream.read = stream_get_null;                                  // block core protocol loop from reading from input

        on_execute_realtime = grbl.on_execute_realtime;                     // Add YModem protocol loop
        grbl.on_execute_realtime = protocol_loop;                           // to grblHAL foreground process

        memset(&ymodem, 0, sizeof(ymodem_t));                               // Init YModem variables
 801d706:	4c17      	ldr	r4, [pc, #92]	; (801d764 <trap_initial_soh+0x88>)
        rt_handler = hal.stream.set_enqueue_rt_handler(stream_buffer_all);  // stop core real-time command handling and
 801d708:	4798      	blx	r3
 801d70a:	4b17      	ldr	r3, [pc, #92]	; (801d768 <trap_initial_soh+0x8c>)
        on_execute_realtime = grbl.on_execute_realtime;                     // Add YModem protocol loop
 801d70c:	4a17      	ldr	r2, [pc, #92]	; (801d76c <trap_initial_soh+0x90>)
        rt_handler = hal.stream.set_enqueue_rt_handler(stream_buffer_all);  // stop core real-time command handling and
 801d70e:	6018      	str	r0, [r3, #0]
        hal.stream.read = stream_get_null;                                  // block core protocol loop from reading from input
 801d710:	4b17      	ldr	r3, [pc, #92]	; (801d770 <trap_initial_soh+0x94>)
 801d712:	f8c6 30b8 	str.w	r3, [r6, #184]	; 0xb8
        on_execute_realtime = grbl.on_execute_realtime;                     // Add YModem protocol loop
 801d716:	4b17      	ldr	r3, [pc, #92]	; (801d774 <trap_initial_soh+0x98>)
 801d718:	6999      	ldr	r1, [r3, #24]
 801d71a:	6011      	str	r1, [r2, #0]
        grbl.on_execute_realtime = protocol_loop;                           // to grblHAL foreground process
 801d71c:	4a16      	ldr	r2, [pc, #88]	; (801d778 <trap_initial_soh+0x9c>)
 801d71e:	619a      	str	r2, [r3, #24]
        memset(&ymodem, 0, sizeof(ymodem_t));                               // Init YModem variables
 801d720:	2100      	movs	r1, #0
 801d722:	f44f 628a 	mov.w	r2, #1104	; 0x450
 801d726:	4620      	mov	r0, r4
 801d728:	f001 f800 	bl	801e72c <memset>
        ymodem.process = await_soh;
 801d72c:	4b13      	ldr	r3, [pc, #76]	; (801d77c <trap_initial_soh+0xa0>)
 801d72e:	64a3      	str	r3, [r4, #72]	; 0x48
        ymodem.next_timeout = hal.get_elapsed_ticks() + 1000;
 801d730:	f8d6 314c 	ldr.w	r3, [r6, #332]	; 0x14c
 801d734:	4798      	blx	r3
 801d736:	f500 707a 	add.w	r0, r0, #1000	; 0x3e8
 801d73a:	6460      	str	r0, [r4, #68]	; 0x44
        ymodem.fs = sdcard_getfs();
 801d73c:	f7ff ff82 	bl	801d644 <sdcard_getfs>
 801d740:	6020      	str	r0, [r4, #0]
 801d742:	2000      	movs	r0, #0

        return false;                                                       // Return false to add character to input buffer
    }

    return on_unknown_realtime_cmd == NULL || on_unknown_realtime_cmd(c);
}
 801d744:	bd70      	pop	{r4, r5, r6, pc}
    return on_unknown_realtime_cmd == NULL || on_unknown_realtime_cmd(c);
 801d746:	4b0e      	ldr	r3, [pc, #56]	; (801d780 <trap_initial_soh+0xa4>)
 801d748:	681b      	ldr	r3, [r3, #0]
 801d74a:	b11b      	cbz	r3, 801d754 <trap_initial_soh+0x78>
 801d74c:	4798      	blx	r3
 801d74e:	f000 0001 	and.w	r0, r0, #1
 801d752:	e7f7      	b.n	801d744 <trap_initial_soh+0x68>
 801d754:	2001      	movs	r0, #1
 801d756:	e7fa      	b.n	801d74e <trap_initial_soh+0x72>

A bit over 50 instructions in unoptimized code... Let us assume that all this code gets executed regardless of a SOH get found or not and estimate 2 clock cycles per instruction. I have a 5 million bytes gcode file that takes around 10 hours to execute. The YModem protocol adds 500 million clock cycles to the total number of clock cycles - for a 100MHz processor that is a whopping 5 seconds. Are those clock cycles taken from the idle loop or from real work? Does it matter?

Outside of idling most of the time is spent in the planner.c, stepper.c and driver.c (step output) when actual work is done. Time spent handling stream I/O and parsing the gcode is relatively insignificant. I just ran at test with the file mentioned above - in check mode it took less than 7 minutes to complete on a STM32F466, again with optimizations disabled. Compared to the 10 hours total execution time this is peanuts. And nobody is going to notice the 5 seconds that is potentially added to it. BTW the F466 runs at 180 MHz so it is even less than that...

I am kind of a speed freak and I have optimized the code more than the original, some basic stuff like using downcounting do...while loops where possible instead of upcounting for each..., some more advanced by using linked lists instead of indexed arrays. Some of this offsets the cost of using function pointers, which for most parts of the code is insignificant.

Get your hands dirty by running grblHAL to get familiar with it. The core is designed to allow plugin code without having any idea of what it does, take advantage of that and worry less about the performance impact until you have something up and running. Then analyze to see if the impact is something to worry about.

[mfp20]

Here is the cost of trapping YModem SOH:

0801d6dc <trap_initial_soh>:

// Check input stream for file YModem start of header (soh) characters.
// Redirect input stream and start protocol handler when found.
static bool trap_initial_soh (char c)
{
    if(c == ASCII_SOH || c == ASCII_STX) {
 801d6dc:	1e43      	subs	r3, r0, #1
 801d6de:	2b01      	cmp	r3, #1

For curiosity sake: how did you get C code and translated ASM inlined like this? And in general what tools are you using to develop? (in particular editor and debugging tools)

The YModem protocol adds ... a whopping 5 seconds.
...
7 minutes to complete on a STM32F466, again with optimizations disabled. Compared to the 10 hours total execution time this is peanuts. And nobody is going to notice the 5 seconds that is potentially added to it. BTW the F466 runs at 180 MHz so it is even less than that...

Thanks for the time spent to show a real scenario. That's the overprovisioning case! What about less capable supported platforms (LPC maybe?)? What about running at higher step rates?

Please, don't spend more of your time for these questions. I'm just trying to stick to your guidelines and the limits you stated (ex: currently grblHAL is able to have a performance level on teensy and don't want to see reduced perf). So I guessed to need a conservative approach (performance wise) in order to not fail the future PR.
But if you tell me that the minimum supported board is overprovisioned... well... I relax a bit!

I am kind of a speed freak and I have optimized the code more than the original,

Gotcha! so you agree with me that ... if I don't get enough performance you'll deny the PR!

Get your hands dirty by running grblHAL to get familiar with it. The core is designed to allow plugin code without having any idea of what it does, take advantage of that and worry less about the performance impact until you have something up and running. Then analyze to see if the impact is something to worry about.

I MUST get my hands dirty by running grblHAL and get familiar with it, that for sure. I didn't make it yet simply because I don't have a running printer.

Short story long: I used to have a very cheap chinese toy (on which I tried marlin, old grbl and klipper) but I dismantled to re-use th v-slots. And being a custom shaped one (1 X axis, 4 Y-Z axes for multitooling), it is taking dust since 18 months unfinished, as I need to know how to control that weird animal. I've 4 MKS GENL that need to work in sync in order to be able to issue a tool switch gcode that re-offsets the X axis under the right Y-Z 'crane'. I need to know how to wire it, and to do that, I need to know what hw will be controlling the machine. The atmega2560 on those boards is too slow for high stepping rates, and suffers of I/O bottleneck (ie: the tty-USB chip is crap, and the boards are placed pretty far one another, so SPI/I2C might have too much noise). So I've used a pre-perforated copper board to build the prototype of adapter, to sit an RPi Pico on top of the MKS GENL, adapting the signal levels (5v-3.3V) for motor signals and SPI. The end result is a Pico-powered motion controller on top of an MKS GENL carrier board (which requires a custom firmware to make it work as a grblHAL plugin). I might arm 1 board only (ie: 1 X-Y-Z axis only), and run vanilla grblHAL on it, but it would be a lot of work for possibly no upgrade path (software wise) if I don't complete the firmware, or you don't accept the PR, or whatever other negative circumstance.

Anyway, thanks for the advices, the musts and must nots, I think we are done here. I mean, the question itself was answered very early in this discussion; and then you kindly helped me with all my doubts and misunderstandings.

[terjeio]

For curiosity sake: how did you get C code and translated ASM inlined like this? And in general what tools are you using to develop?

STM32CubeID, the assembly listing is generated by default. It is Eclipse based. Most of the other platforms I develop for is also based on Eclipse. For ESP32 and RP2040 I am using VSCode and I compile using the Arduino IDE for SAM3X8, SAMD21 and iMXRT1062 (I am using Eclipse as an editor instead of the useless Arduino IDE editor).

What about less capable supported platforms (LPC maybe?)?

The slowest 32-bit MCUs I have drivers for runs at 48 MHz, so double the time to 10 seconds. And then halve it again if optimizing for speed? And be aware that actual time spent in the routine is usually a lot less than the estimated 100 cycles.

What about running at higher step rates?

Since the routine run in an interrupt context it may affect step jitter, it is the reason I redirect the stream by swapping function pointers and read from it in the foreground process. The YModem protocol will never be active during motion so for that it is a non-issue except for the SOH test - I have not calculated the impact but is in the region of 100-200ns per received character @ 100MHz? However, it is not likely to be noticeable - this since ARM interrupts are assigned a priority and stepper interrupts can preempt the character receive handler (and other lower priority handlers as well). So, not a worry for me.

Note grblHAL is not for 3D printing and will likely never be. Non-standard M-codes can be (and are) added by plugins but will never make it to the core. Non-standard G-codes are harder to open up for plugins to add, AFAIKT proper handling of modal groups will be a challenge if allowing that. NIST RS274NGC v3, parts LinuxCNC and Mach3 are the standards I aim to implement in the core.

[mfp20]

What about less capable supported platforms (LPC maybe?)?
What about running at higher step rates?

Since the routine run in an interrupt context it may affect step jitter, it is the reason I redirect the stream by swapping function pointers and read from it in the foreground process.

Ok, that's one of the negative side effects I was babbling about some messages above... redirecting/re-injecting... in this case is changing the function ptrs instead.

The YModem protocol will never be active during motion so for that it is a non-issue except for the SOH test - I have not calculated the impact but is in the region of 100-200ns per received character @ 100MHz?

Because gcodes are relatively small amount of text, the jitter introduced might be low. If the size/frequency of I/O increases (ex: sending buffer data instead of gcodes), it may become noticeable and even screw up the build process...

However, it is not likely to be noticeable - this since ARM interrupts are assigned a priority and stepper interrupts can preempt the character receive handler (and other lower priority handlers as well). So, not a worry for me.

... but it is not a problem indeed, on ARM platforms, because of the NVIC (if I remember well the name of the IRQ controller on ARM). What about the other platforms then? Ex: ESP32 isn't arm.

In other words: it may become a problem if sending buffer data on esp32 because of its higher I/O irq frequency AND the lack of NVIC prioritising the IRQs.

Note grblHAL is not for 3D printing and will likely never be. Non-standard M-codes can be (and are) added by plugins but will never make it to the core. Non-standard G-codes are harder to open up for plugins to add, AFAIKT proper handling of modal groups will be a challenge if allowing that. NIST RS274NGC v3, parts LinuxCNC and Mach3 are the standards I aim to implement in the core.

Again, for curiosity sake: WHY NOT 3D PRINTING? I mean... I saw it having spindle and laser tools, but I tought that was one of the possible uses. Not specifically excluding 3d printing.

That's cool as long as the core allow heterogeneous machines, and plugins to add custom functionality. Is there anything else that might deny 3d printing in the foresable future?

[terjeio]

What about the other platforms then? Ex: ESP32 isn't arm.

You can prioritize interrupts on an ESP32, but can higher priority ones preempt lower? If not make use of the fact that it has two cores?

WHY NOT 3D PRINTING?

Lots of gcodes to add, incompatible (with RS274NGC) use of parameter words and, for me, the biggest hurdle - time. And 3D printers are slow and imprecise by nature - that allows adding stuff like a display without (?) affecting print quality.

Is there anything else that might deny 3d printing in the foresable future?

I for one will not take on the additional burden of beeing part adding, testing and maintaining the code needed. I have other projects I want to spend time on. If there was a large committed team taking care of the repo then perhaps that could decide to go ahead, I would then retire from the project.