Shouldn't we do things better than Corutils?

[habruening]

• ls -l |grep Jul lists all files that are created in July but also files that have "Jul" in their file name. And it only works in languages that have the ASCII string "Jul" in the name of the 7th month of the year. So it is a buggy way of working with the shell.
• ls | grep is unable to create a list of all authors of all files in the current directory.
• touch "x y.txt" ; ls | grep "x y" | xargs cat does not work.
• ls produces a list of file names. But the file names cannot simply selected by clicking on them. You must manually select a file name character by character with your mouse cursor.

All the Coreutils are based on pure text. They don't understand any sort of data structure. That makes them very limited. Working with them requires workarounds, parsers, regular expressions, awk and loops everywhere just in order to get the data that you want. And due to all these laborious workarounds everything is fragile. Thousands of system internal scripts use human readable ASCII text for things that are only ever intended to be machine readable. But for machine readable data things like JSON or XML would work better. When Coreutils had been implemented, the authors should have asked themselves if this is the right way of working. They did not or at that time it was the right way. When reimplementing Coreutils, we should ask ourselves again if this is the right way of working.

How often have we seen for example a buggy script where the author did not take into account that names can contain spaces? We all know how complicated spaces in file names are. We are all used to avoid spaces in file names, because too many tools are buggy. Being compatible with buggy scripts is no good goal, because then in 40 years we still suffer from the spaces in file names problem. Spaces in file names seem to be one of the biggest problems in computer science.

I don't understand why you really want to reimplement something without any perspective of doing things in a better way. What should users motivate to use uutils if there are things like PowerShell or Nushell, which do things in a much more modern way? Wouldn't the effort be better invested there? And what can we expect from a new set of shell tools for a text terminal, that follows the VT100 standard from 1978?

[tertsdiepraam]

Maybe we should, but that's what nushell is for! It's just a different kind of project. With this project, all your scripts keep working.

It's not a zero-sum game. This project is actually cooperating with nushell! Because we implement things the way people usually expect, nushell can use our implementations in nice structured data.

PS. Your examples mostly illustrate how ls is optimized for human readability and should not be piped to other programs, but I understand your general point.

[sylvestre]

First, we think that the only way we will be successful with this project is to be a dropped in replacement for the GNU and other implementations. So, we have to implementation the limitations (spaces in path is a great example btw).

We have been adding extensions to improve our implementation:
https://github.com/uutils/coreutils/blob/main/docs/src/extensions.md

I also would like automatic json exports. However, it would very difficult to define a clear export format for all commands.
Suggestions welcome!

[habruening]

A simple JSON format is relatively obvious.

{ "command": "ls",
  "options": "-l",
  "where": "/tmp",
  "contents": {"xy.txt": {"type": "file",
                          "permissions": "rwxrw-r--"
                          "owner": "user_1",
                          "group": "users",

and so on

There is nothing special. But when you want to parse this (e.g. with Python), this format does not support you much, because JSON parsers normally create key value pairs full of strings only. Normally you don't want to parse into strings. You want some sort of data structures. So I recommend to always specify the type (or let me say the "semantical meaning") directly in the data. Then a parser does not need a sophisticated mechanism to find that out. It can create the right objects (whatever this might be) and even work with parts of the output.

The JSON format would look like this.

{ "type": "ls-output"
  "options": "-l",
  "where": "/tmp",
  "contents": {"xy.txt": {"type": "ls-file",
                          "permissions": {"type": "unix-file-permissions",
                                          "symbolic-notation": "rwxrw-r--"},
                          "owner": {"type": "unix-user",
                                    "username": "user_1"},
                          "group": {"type": "unix-group",
                                   "groupname": "users",

and so on

The advantage of this is also (as said above), that a parser can parse any sub structure of the whole JSON file. For example you can parse the string {"type": "unix-file-permissions", "symbolic-notation": "rwxrw-r--"} because you see immediately, what it is. This is very handy. It enables to write quick ad-hoc parsers for any sort of information. This is what people normally want to do when working on the shell.

But I personally prefer the data format EDN (https://github.com/edn-format/edn). That is similar to JSON, but with small extensions. It natively supports this typing mechanism and it does not require so many strings, because it supports keywords. EDN comes from the programming language Clojure, so it is a bit exotic. That is a real disadvantage. JSON parsers exist for all languages, EDN parsers not. For Rust it exists (https://github.com/bowbahdoe/edn-format). EDN is indeed very handy. The idea is simple. Instead of {"type": "MyType", "value": "xy"} you write #MyType {:value "xy"} or even #MyType {:value #MyTypeValue xy}. Then the parser always knows how to parse things and the parsing becomes a simple job.

The example from above would look like this:

#ls-output {:options "-l"
            :where #unix-path "/tmp"
            :contents {#unix-path "xy.txt" #ls-list-file {:permissions #symbolic-notation rwxrw-r--
                                                          :owner #unix-username user_1
                                                          :group #unix-groupname users

and so on

Note that this example does not use strings for the permissions, the user name and the group name, because these things are not to be strings. Such strings are just a human readable representations of some operating system dependent technical pieces of information. But when parsing, we normally don't want to treat them as strings (of course, we still can if we don't want to implement a special data structure).

All this is maybe not so human readable like pure JSON. But humans prefer the normal output anyway.

The more obvious questions (Why did I use a dictionary for the contents and not a list?) in my opinion do not matter very much. You once agree an something and that's it.

Maybe you like the EDN format. But even if not, what for me matters is that type attribute, because that enables simple parsing. If it is in the pure JSON syntax "type": "MyType" or in the EDN syntax #MyType does not matter.

[tertsdiepraam]

JSON is actually not so straightforward. Sure you could just make some format up, but in your ls example, how does it change with the flags? Would it he the same structure but empty fields if -l is not passed? How should permissions be formatted? In your dict representation sorting also doesn't apply. But note that nushell doesn't even have sorting because that's much easier to do with a separate command with structured data.

There are other issues too. It should for example be streamed out because that's how the coreutils tend to handle large amounts of data. It's worth considering that the coreutils are extremely string-focused. In implementing the nushell versions of some commands, they tend to come out much simpler if all the formatting options are not used.

It seems to me that it's a sufficiently different paradigm from these tools that they should probably be implemented separately. That's just my opinion though. It's a very interesting thing to explore further!

[sylvestre]

and there are more than 110 commands ;)

[habruening]

I think, you imagine it too complicated. I don't agree, that it is so hard to find answers to your concerns.

1. We should not use the same structure for all options, because the data that for example ls with different options delivers is actually different.

2. You may also ask why you would need a "-l" option, at all if you don't produce human readable text. You can JSONify just everything. Valid point. But it is helpful for humans to use the same options for the text based output than for the structured output. People want to quickly produce the data in the way how they have learnt it over 40 years. There is no need to force them to learn anything new. People want to play around on the command line and then write the commands into a script and pipe them. There is no need to change this way of working. People who want to change their way of working, will use Nushell.

3. We should not deliver empty fields. An empty field serves no other purpose than fulfilling a schema. We need no schema for data that does not exist. That would only make people happy who are used to Object Oriented statically typed programming languages. And even for them there is no use for non-existing fields. Uutils should not embrace the Object Oriented paradigm. It should be paradigm neutral and deliver things in the least complicated format. And that means, not delivering, things that do not exist. Trying to find a super perfect structure for everything, which hypothetically could exist can only make things complicated. Try to keep it simple and let data be data and nothing more! Nobody would ever read a schema. People type ls -l -json and see what comes out.

4. The sorting does not matter. JSON objects are not sorted. But nothing hinders you sort them. It is up to the consumer to dicide how to use the output. If you alternatively deliver a JSON array, it can be sorted. Any decision you make would be ok, I think. Everything has pros and cons. You cannot make everybody happy. But XML/JSON/EDN are structured enough to convert everything into everything.

5. Streaming is no problem. You can deliver the data as it is there. JSON object by JSON object. It is up to the consumer to decide how to take them. Streaming of JSON or XML data is nothing new.

6. Because we talk about more than 100 commands, it is even more important to have a structured output well established from begin on. It will never come later. It would be a real benefit for the uutils. I guess many people and the Linux distributions would quickly start using the structured input/output once it is there. We are not talking about a neglectable details. Chaining of the core utils is the backbone of GNU/Linux. It is worth spending a few more thoughts how to make this better, because that will be the backbone for the next 40 years. I don't think that strings will have a future for another 40 years. That question is even so important that it would be worth the effort to invent a complete own data format.

[sylvestre]

Patches welcome :)