Skip to content

Redirected output, row-value shift, and other features

Compare
Choose a tag to compare
@johnkerl johnkerl released this 12 Aug 03:36

The principal feature of Miller 4.4.0 is redirected output. Inspired by awk, Miller lets you tap/tee your data as it's processed, run output through subordinate processes such as gzip and jq, split a single file into multiple files per an account-ID column, and so on.

Details: http://johnkerl.org/miller/doc/reference.html#Redirected-output_statements_for_put

Other features:

  • mlr step -a shift allows you to place the previous record's values alongside the current record's values: http://johnkerl.org/miller/doc/reference.html#step
  • mlr head, when used without the group-by flag (-g), stops after the specified number of records has been output. For example, even with a multi-gigabyte data file, mlr head -n 10 hugefile.dat will complete quickly after producing the first ten records from the file.
  • The sec2gmtdate verb, and sec2gmtdate function for filter/put, is new: please see http://johnkerl.org/miller/doc/reference.html#sec2gmtdate and http://johnkerl.org/miller/doc/reference.html#Functions_for_filter_and_put.
  • sec2gmt and sec2gmtdate both leave non-numbers as-is, rather than formatting them as (error). This is particularly relevant for formatting nullable epoch-seconds columns in SQL-table output: if a column value is NULL then after sec2gmt or sec2gmtdate it will still be NULL.
  • The dot operator has been universalized to work with any data type and produce a string. For example, if the field n has integers, then instead of typing mlr put '$name = "value:".string($n)' you can now simply domlr put '$name = "value:".$n'. This is particularly timely for creating filenames for redirected print/dump/tee/emit output.
  • The online documents now have a copy of the Miller manpage: http://johnkerl.org/miller/doc/manpage.html
  • Bugfix: inside filter/put, $x=="" was distinct from isempty($x). This was nonsensical; now both are the same.

Brew update: Homebrew/homebrew-core#3820