Writing Millerish to clean empty lines and whitespaces

[NikosAlexandris]

Cleaning, sort of the hard way

Initially, some input files with 2 columns each

➜  mlr --c2x cat *correlations.csv |head -20
myd21a1_emissivity_29_landsat8_albedo_corr                   -0.306360
myd21a1_emissivity_31_landsat8_albedo_corr                   -0.075479
myd21a1_emissivity_32_landsat8_albedo_corr                   0.021519
13_order_2_madrid_myd21a1_emissivity_29_landsat8_albedo_corr -0.240440
13_order_2_madrid_myd21a1_emissivity_31_landsat8_albedo_corr -0.001169
13_order_2_madrid_myd21a1_emissivity_32_landsat8_albedo_corr 0.115604

myd21a1_emissivity_29_landsat8_evi_corr                   0.134936
myd21a1_emissivity_31_landsat8_evi_corr                   0.079163
myd21a1_emissivity_32_landsat8_evi_corr                   0.025329
13_order_2_madrid_myd21a1_emissivity_29_landsat8_evi_corr 0.137769
13_order_2_madrid_myd21a1_emissivity_31_landsat8_evi_corr -0.009855
13_order_2_madrid_myd21a1_emissivity_32_landsat8_evi_corr -0.089816

myd21a1_emissivity_29_landsat8_mndwi_corr                   -0.062381
myd21a1_emissivity_31_landsat8_mndwi_corr                   -0.015565
myd21a1_emissivity_32_landsat8_mndwi_corr                   -0.002446
13_order_2_madrid_myd21a1_emissivity_29_landsat8_mndwi_corr -0.091097
13_order_2_madrid_myd21a1_emissivity_31_landsat8_mndwi_corr 0.021983
13_order_2_madrid_myd21a1_emissivity_32_landsat8_mndwi_corr 0.056731

Then, a slightly sed-adventurous cleaning action (meanwhile, I revisited the Glossary a couple of times, for example the XTAB term)

➜  mlr.linux.x86_64 --csv --c2x cat *correlations.csv |head |sed -r '/^\s*$/d' |sed 's/[[:space:]][[:space:]]*/,/g'
myd21a1_emissivity_29_landsat8_albedo_corr,-0.306360
myd21a1_emissivity_31_landsat8_albedo_corr,-0.075479
myd21a1_emissivity_32_landsat8_albedo_corr,0.021519
13_order_2_madrid_myd21a1_emissivity_29_landsat8_albedo_corr,-0.240440
13_order_2_madrid_myd21a1_emissivity_31_landsat8_albedo_corr,-0.001169
13_order_2_madrid_myd21a1_emissivity_32_landsat8_albedo_corr,0.115604
myd21a1_emissivity_29_landsat8_evi_corr,0.134936
myd21a1_emissivity_31_landsat8_evi_corr,0.079163
myd21a1_emissivity_32_landsat8_evi_corr,0.025329

Consulting the documentation

Searching for the term white in johnkerl.org/miller6/ the command clean-whitespace appeared!
Reading also through the table of contents (right margin) in Reference > Main reference > List of verbs, I spotted skip-trivial-records! Just what I required to recompose the cleaning action:

➜  mlr.linux.x86_64 --c2x cat *correlations.csv |mlr.linux.x86_64 head -n 9 then skip-trivial-records then clean-whitespace |mlr.linux.x86_64 --ifs ' ' cat
myd21a1_emissivity_29_landsat8_albedo_corr,-0.306360
myd21a1_emissivity_31_landsat8_albedo_corr,-0.075479
myd21a1_emissivity_32_landsat8_albedo_corr,0.021519
13_order_2_madrid_myd21a1_emissivity_29_landsat8_albedo_corr,-0.240440
13_order_2_madrid_myd21a1_emissivity_31_landsat8_albedo_corr,-0.001169
13_order_2_madrid_myd21a1_emissivity_32_landsat8_albedo_corr,0.115604
myd21a1_emissivity_29_landsat8_evi_corr,0.134936
myd21a1_emissivity_31_landsat8_evi_corr,0.079163
myd21a1_emissivity_32_landsat8_evi_corr,0.025329

Removing the head commands (used above to get the same number of output lines) and using simply mlr (I have an older mlr installation on my system, so I use a binary above downloaded from Releases) we have

➜  mlr --csv --c2x cat *correlations.csv |sed -r '/^\s*$/d' |sed 's/[[:space:]][[:space:]]*/,/g'

vs

➜  mlr --c2x cat *correlations.csv |mlr skip-trivial-records then clean-whitespace |mlr --ifs ' ' cat

Though it seems it's not possible to use less pipes in this example, don't we agree that the latter command is easier to read?
What it does is:

• transpose the input CSV files
• remove empty lines
• clean white spaces
• let Miller know the input field separator is the space character

Speaking Millerish...

Thinking about it, the command could also be like:

mlr --csv transpose *correlations.csv |mlr skip-trivial-records then clean-whitespace |mlr --ifs ' ' cat

Isn't this simple and expressive language that helps to concentrate in the actual work?