How to use DSL to delete non-UTF-8 hex characters from a string?

[pjfarleyiii]

I have some CSV data that contains strings that have embedded non-UTF8 hex characters, specifically ISO8859-1 superscripts 1, 2, and 3, x'b9', x'b2', and x'b3', without the leading UTF-8 control character before each one. They are just naked ISO8859-1 one-byte characters with those hex values.

In order to further process these string-valued CSV fields n some DSL programs I am writing, I need to be able to delete these characters from the field.

It seems I can't use the string escape value "\xb9", etc. in the regexp string value (in gsub argument 2 for instance), and just pasting a copy of the superscript character into a string coded to be the regexp results in miller complaining that it found invalid UTF-8 characters during regexp parsing. REPL example:

[mlr] supers = "\xb9|\xb2|\xb3"
[mlr] $Name = gsub($Name, supers, "");
error parsing regexp: invalid UTF-8: `�|�|�`

How can I set up a gsub or other replacement function to remove all of these characters from a string-valued CSV field please?

Peter

[Edit: I found the doc on the ssub function, and that works using the naked hex values, but that only replaces the first instance of the value. How would I code it to replace ALL such values in a string? There does not seem to be an sgsub function that would do that.]

[aborruso]

@pjfarleyiii could you attach a sample file?

[pjfarleyiii]

One-record test file uploaded, only two CSV columns, Date and Name (header included). Name field has the superscript "1" character ("\xb9") in it.

mlrtest.csv

[aborruso]

It's not the answer to your question, but it's a way to solve it

iconv -f utf-8 -t utf-8 -c mlrtest.csv >out.csv

[pjfarleyiii]

That will convert the "naked" x'b9' to the UTF-8 version, but it is an extra conversion step outside of the miller scripting that I had planned for this data. I was hoping for a way to perhaps put an ssub function call into a loop to get all of the occurrences changed, but so far I haven't been able to think of a loop termination condition that wouldn't use a regular expression failing to find any more occurrences of these characters, and I can't use a regular expression because they are not UTF-8 values. Maybe test the length of the (possibly modified) string and if the length is the same as it was in the prior loop then stop (i.e. we didn't delete any characters that time around so we must be done)?

[pjfarleyiii]

I was able to code a while loop to solve the issue, pasted below for future readers.

sup1 = "\xb9"; sup2 = "\xb2"; sup3 = "\xb3";
curln = strlen($Name); prvln = 0;
while (curln > prvln) {
    prvln = curln;
    $Name = ssub($Name, sup1, "");
    $Name = ssub($Name, sup2, "");
    $Name = ssub($Name, sup3, "");
    curln = strlen($Name);
}

[johnkerl]

See also https://miller.readthedocs.io/en/latest/special-symbols-and-formatting/#how-to-escape-in-regexes which has been updated

[pjfarleyiii]

I understand about the UTF-8 issue. Limitations of an implementation language are tough to get around. From my point of view, a gssub solution is perfectly acceptable.

Since I am on a Windows platform here, will there be a Windows binary with gssub available soon or should I just wait for the next stable release? I do have a working solution using version 6.1.0, so I am not in any pain if I should wait.

[Edit: I just saw the 6.2.0 release in the miller Releases page. Thank you!]

[johnkerl]

@pjfarleyiii 6.2.0 indeed! However for future reference you can visit https://github.com/johnkerl/miller, Actions tab, latest "Miller-Go" job, Artifacts, "mlr-windows-latest" -- this will give you a zip file with mlr.exe inside.

[pjfarleyiii]

Thanks again for the gssub release, I will try it next. One thing I am somewhat vague about -- r-strings are not something I have looked into in python yet (but I will now), so I am not sure what the decision to use "implicit r-strings" really means. Does it mean that no RE meta-characters can be escaped at all except for the "\." escaped period, as noted in the discussion of #297 that you linked to?

[johnkerl]

In Python there is "a\tb" and r"a\tb" -- the latter being "r-strings".

In Miller there isn't any explicit syntax but the second argument to sub and gsub, and the right-hand side of =~ and !=~, are implicitly r-strings.

What this means is that string literals anywhere else in DSL expressions, "a\tb" becomes a, the tab character, and b. For implicit r-strings, these are kept as a, \, t, b -- then passed along to the Go regexp library which does as it likes with the \ and the t.

This implicit r-stringing was necessary for people to make full use of available regexp syntax in Miller. Without it, lots of things were impossible in Miller as #297 describes.

[johnkerl]

Also @pjfarleyiii #997

[johnkerl]

BTW I changed the verbs to kebab-case for consistency with other verbs.