Is it intended to use ascii SOH \x01 for ascii_null alias instead of \x00?

[GlassGruber]

I'm opening a discussion thread instead of an issue ticket because maybe it's intended for some reasons and is known by experienced users.

It took me quite a bit to get it but parsing the output of miller modifying the record separator with convenient ascii_null alias is not actually the ascii sequence \x00 as I expected, but the SOH (Start of Heading) character \x01.

And indeed in the docs is stated correctly that:

ascii_null = "\x01"

I admit I totally blinded the \x01, but reading ascii null my brain assumed by default \x00.

Is this an error or intended?

With this sample script you can use to test that with ascii_null is not acting as an ascii null byte character

#!/bin/bash

mlr --version

# -d '' means use null byte as line separator
while read -r -d '' line; do
    echo "$line"
# below using ascii_null is printing nothing, 
# replacing ascii_null with '\x00' works as expected
done < <(mlr --icsv --ors ascii_null cat << CSV
id,first_name,last_name
1,Corbet,Torrejon
2,Alwin,Lucien
3,Tades,Huson
4,Rachel,Cuniffe
5,Armand,Skase
6,Demetre,Blase
CSV
)

[johnkerl]

Is this an error or intended?

It's been a long time but IIRC this is unintentional.

I'll fix this, along with a heavy warning in the release notes ...

• ascii_null will now be \x00
• ascii_soh will now be \x01

[GlassGruber]

Ok! I see you already opened an issue, thank you and nice work with miller btw! 🎩👌

[johnkerl]

@GlassGruber #1620

My very deep apologies -- this is embarrasing. All four of NUL SOH STX and ETX RS aliases were off by one :(

I've done an awful lot of low-level programming over the years including serial/UART on-circuit-board development and I really ought to have known better :(

[GlassGruber]

Ouch, that's unfortunate indeed! But hey as the old saying says off by one errors are the bane of programmers right? We've all been there and done that! :)

I was thinking that maybe could be better to replace with new aliases all together, something like adding '_char' to the end to avoid breaking backward compatibility.
So for example there will be ascii_null_char = '\x00' and the ascii_null is left as deprecated alias mapped to the new ascii_soh_char for example.

But I don't know if this makes sense ...

[johnkerl]

@GlassGruber yes that's actually a safer approach ... 🤔

[johnkerl]

#1621