ConvertTo-EOL cmdlet with -Unix and -Windows params

[ghost]

Often I use utilities like unixtodos.exe and dostounix.exe to fix the line endings of documents. It would be super helpful if PowerShell team consider it as a useful utility and provide it OOTB.

One solution is: https://stackoverflow.com/a/19128003 but it "almost" works as "mike z" mentioned.

[iSazonov]

ConvertTo-EOL -Unix
ConvertTo-EOL -Windows

Also we can consider adding -EOLUnix/-EOLWindows parameters in Get-Content/Set-Content/Out-File
Previously we discussed Encoding cmdlets - we could add the parameters in the cmdlets.

@mklement0 It seems you had suggestions about this?

[ghost]

@iSazonov, is there any kind of approval process or this could just be implement by anyone?

[iSazonov]

@kasper3 Right way is to create RFC in https://github.com/PowerShell/PowerShell-RFC

[iSazonov]

In-place update can be dangerous. We could create temp file, process and replace original file and have ip-place update as option.
Also should we accept strings from pipeline and as argument?

[iSazonov]

With -replace '\r?\n', '\n' I agree that we don't need accept strings from pipeline and as argument. And after that we don't need ConvertTo and can use another verb. Now question maybe enhance Set-Content? why we need new cmdlet?

[iSazonov]

So many parameters! Maybe:
Set-Content -WindowsToUnix
Set-Content -UnixToWindows

[SteveL-MSFT]

I think this has been a great discussion and the different options to consider I think warrants a RFC. Personally, I can see the EOL conversion to be part of a Convert-File cmdlet which can also handle encoding although we could just start with EOL first.

[jeffbi]

While I like the name, I'm a bit concerned that it would have pretty broad interpretation. What will users expect a cmdlet named Convert-File to do? Convert from, say, XML to JSON? From .doc to .docx? Or just encoding and line endings?

[jeffbi]

@mklement0: I think that describes the intent better.

[ghost]

@mklement0, you have much better hang on the shape of the cmdlet and arguments; would you consider opening an RFC based on the ideas shared here? :)
It would be great if performance characteristics are also kept in mind. For example, instead of reading the file (that can be huge in order of gigabytes) in memory, replace and write back the content; we can open the stream, walk through each character once, and write back. Overall complexity will remain O(1)+O(1). Maybe this kind of approach: https://stackoverflow.com/a/1971052. If it is going to run only on .NET Core 2.1+, then we have even performant APIs like System.IO.Pipeline, Span, Memory, System.Runtime.Intrincis to squeeze the last bit of performance. Some people compared and complained about PS performance vs. Python in this area: https://stackoverflow.com/q/9724521.

[SteveL-MSFT]

@mklement0 would it be better to have [-LineEnding <Lf|CrLf>]? (or even <Lf|CrLf|Windows|Unix> to make it easier for novice users)

[jeffbi]

Will MacOS users know they should be using Unix-style line endings?

Also, would the cmdlet be able to recognize OS 9 style line endings (CR only) and be able to convert from those. Probably not much demand, but it was a thought.

[jeffbi]

@mklement0 Convering from is all I was really concerned about.

My sister, brother-in-law, and niece almost certainly don't know that MacOS has Unix under the covers, but they are not likely to be the PowerShell target audience, so you're probably right about that. And I would rather expect that even casual Linux users would know that it is a flavor of Unix.

I have no strong preference here either. I'm just glad the cmdlet would be able to convert from old-style Mac line endings.

[iSazonov]

I never like Lf/CrLf abbriviatury. I'd prefer -LineEnding <Windows|Unix>
From my comment we could use -LineBreaks <Windows|Unix> or -LineTerminators <Windows|Unix>.

[ghost]

Maybe without s, LineTerminator, LineBreak or LineFeed? LineFeed would have been great, but since LF stands for 'Line Feed', it might be confusion (may require disambiguation: Carriage Return Line Feed vs. Line Feed) .

[iSazonov]

Usually EOL is used - so we could use -EndOfLine with -EOL alias.

If we want -EncodingFrom/-EncodingTo in the cmdlet I'd prefer more simple parameter sets - without -Lf/-CrLf.

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this. We cannot simply replace Cr and Cr/Lf without knowing the encoding of the file (ie, UTF-8 vs UTF-16). Therefore, Convert-TextFile cannot support translating line endings without the encoding work at the same time. As for the line ending parameters, the Committee recommends just -LineEnding <Cr|CrLf|Windows|Unix> and not have -Cr -CrLf.

[chrispollitt]

Is this feature going to be added. It would be very helpful. What is the status?

[rkeithhill]

Maybe such an implementation should be done in https://github.com/PowerShell/TextUtility?

GitHub

PowerShell/TextUtility

Microsoft.PowerShell.TextUtility module. Contribute to PowerShell/TextUtility development by creating an account on GitHub.

[iSazonov]

It is as easy as use System.IO.StreamReader and System.IO.StreamWriter.

[SteveL-MSFT]

This was brought up in today's Community Call and I think having it as a cmdlet part of TextUtility makes the most sense. Transferring this issue.

[SteveL-MSFT]

This is a dupe of #17