Changing the Output Record Separator for CSV Files

[osevill]

For csv files, is there a way in the current version of Miller to change the (output) file format line endings from the default \n (in Linux and MacOS) to \r\n considering that the file will be consumed on a Windows platform?

Running the following on MacOS, mlr v6.12, I've noticed that when I start with a source file with \r\n line endings, the output file has unix-style line endings:
mlr --csv --from ./source_file.csv put '$new_column = $field_1 . "|" . $field_2 . "|" . $field_3' > ./output_file.csv

I found this older documentation referring to version 5.5. The "Autodetect of line endings" section at the bottom of the page seems to suggest you could explicitly specify the Output Record Separator, --ors.
https://johnkerl.org/miller-releases/miller-5.5.0/doc/file-formats.html

The latest documentation for v6.12 suggests the output record separator is not alterable for csv files (see "Which separators apply to which file formats", at the bottom of the page):
https://miller.readthedocs.io/en/latest/reference-main-separators/

When I tried adding --ors '\r\n' to the above mlr put statement, it seemed to be ignored.

I typically do the file format conversion to Windows line endings in a text editor, but was looking to save a step by doing it in the mlr put statement.

Thanks!

[aborruso]

OT: If you don't have to manage and if you don't have carriage returns inside cell values, you can use classic shell commands

mlr --csv --from ./source_file.csv put '$new_column = $field_1 . "|" . $field_2 . "|" . $field_3' | dos2unix -f | unix2dos -f >./output_file.csv

[osevill]

Thank you for your suggestion, and for pointing out that cell values can't have carriage returns when using dos2unix and unix2dos.

[johnkerl]

When I tried adding --ors '\r\n' to the above mlr put statement, it seemed to be ignored.

Yes. As of Miller 6 I stopped trying to manage line endings, letting the Go I/O libraries handle it.

Meanwhile, Miller has a few "auxiliary entry points": these are basically little standalone programs bundled with Miller.

mlr help auxents

Miller has a few otherwise-standalone executables packaged within it.
They do not participate in any other parts of Miller.

Available entries:
  mlr aux-list
  mlr hex
  mlr lecat
  mlr termcvt
  mlr unhex
For more information, please invoke mlr {subcommand} --help.

mlr termcvt

Usage: mlr termcvt [option] {zero or more file names}
Option (exactly one is required):
--cr2crlf
--lf2crlf
--crlf2cr
--crlf2lf
--cr2lf
--lf2cr
-I in-place processing (default is to write to stdout)
-h or --help: print this message
Zero file names means read from standard input.
Output is always to standard output; files are not written in-place.

Effectively, mlr termcvt --lf2crlf is unix2dos and mlr termcvt --crlf2lf is dos2unix.

These are not verbs -- so you cannot do mlr --flags some-verb then termcvt .... You can do

mlr --flags some verb ... | mlr termcvt --lf2crlf > output-file-name-goes-here

[osevill]

Thank you for your suggestion. I didn't know "auxiliary entry points" existed (I just checked the documentation and see that it's there.)
I tried it out and it seems to work fine. On a Mac, I would have to install unix2dos, so using mlr termcvt --lf2crlf allows me to use miller alone to do the job.

Two questions:

• Does it have the same limitation as unix2dos that @aborruso mentioned above, namely that it's a problem when data elements have embedded carriage returns?
• I'm having difficulty understanding the help combination of -I in-place processing (default is to write to stdout) and Output is always to standard output; _files are not written in-place._. It sounds like the latter point contradicts the purpose of the -I flag.

When I try this:
mlr termcvt -I --lf2crlf ./test_file.csv
to update test_file.csv in-place, I get this error:

mlr termcvt: open ./test_file.csv-termcvt-temp: no such file or directory

It's not a problem to just redirect the output to a new file, but was curious how to use the -I flag.

Thx!