linenoise embeds non-free ConvertUTF{.cpp,.h}

[infinity0]

Hi, in the process of Debian packaging this piece of software we noticed that the source code of nix includes non-free (and buggy, unmaintained, obsolete) code from Unicode Inc, specifically in the linenoise embedded library.

The solution as suggested by Debian (and adopted by Ghostscript) is to rewrite the code to use libicu instead.

[dtzWill]

Hmm well that's unfortunate. Following your links (thanks!) I found that many projects include this, to the dismay of Debian packaging teams I'm sure. One project jumped at me, LLVM:

https://lintian.debian.org/full/[email protected]#llvm-toolchain-6.0_1_x3a6.0~_x2brc2-1

Which is interesting because:

• May be source of fixes / improvements:
  • https://github.com/llvm-mirror/llvm/blob/master/lib/Support/ConvertUTF.cpp
  • https://github.com/llvm-mirror/llvm/blob/master/lib/Support/ConvertUTFWrapper.cpp
• Perhaps this isn't a deal-breaker for Debian --so the time/resources it takes to resolve this aren't necessarily standing between Debian users and Nix goodness? :)

ICU seems an awfully heavy dependency, I'm not even sure what Nix's official stance on UTF8 is anyway --perhaps removing utf8 support from our REPL is best short-term solution?

[shlevy]

👍 for removing UTF8 support for now. ICU is pretty damn heavy... Maybe some day we can have it enabled optionally at runtime if it's available.

[dtzWill]

(Perhaps moving to editline would be worth (re-)considering?)

[shlevy]

@dtzWill How hard would it be to quickly remove utf-8 support while we investigate longer-term solutions?

[edolstra]

Just revert 21f5157? What was the reason for switching to linenoise-ng anyway?

[dtzWill]

It doesn't look very easy to remove. We could also revert to linenoise (not linenoise-ng) although that'd be unfortunate functionality-wise.

[dtzWill]

Since I was curious what it would take / act like, here's a quick port to using editline (no ncurses): https://github.com/dtzWill/nix/tree/feature/editline

Using linenoise-ng was done so that completions were listed instead of only being available one-at-a-time. See #1703 .

[edolstra]

@dtzWill Switching to editline seems good to me in principle.

[infinity0]

The issue was reported to LLVM upstream and they have it marked as "P release blocker" for nearly a year now, sigh.

editline sounds good to me, removing an embedded library is an extra bonus IMO.

[dtzWill]

Reverting to linenoise (no -ng) and picking up the contents of a PR that optionally list completions:

https://github.com/dtzWill/nix/tree/experimental/linenoise-list-all

(I just grabbed the code from the PR, we may wish to merge that with linenoise master for fixes?)

Not sure what is best way to go, but if not editline then modifying linenoise (as in that branch) looks pretty easy--much easier than de-UTF8'ing linenoise-ng.

Hope this helps!

[shlevy]

@dtzWill I'd prefer to defer to your technical opinion here, given that we're not OK embedding ConvertUTF. Should we do linenoise or editline? Can you open a PR for whichever?

[dtzWill]

Okay! I probably won't be able to get to this for a few days at least, but if that's alright I'll put together a PR for folks to review/test out.

[cstrahan]

FWIW, I've opened an issue upstream (and in a fork that looks nice):

• ConvertUTF.{cpp,h} are non-free arangodb/linenoise-ng#20
• ConvertUTF.{cpp,h} are non-free AmokHuginnsson/replxx#12

[KaiHa]

Hi @dtzWill, @shlevy, are you still planning to work on this issue. It would be great if we can get this solved.

[dtzWill]

rain1's linenoise fork looks promising: https://github.com/rain-1/linenoise .

Listing more than one completion is moderately important to me. Looks like rain1's fork has that change under review. Not sure what the problem with editline was.

[dtzWill]

I'm still not sure why using libeditline wasn't the clear winner, haha. Maybe because I didn't have a good way to vendor it?

Poking at the branch I linked above, seems good to me?
https://github.com/dtzWill/nix/tree/feature/editline

I fear I'm missing something, but for now I'll create a PR for it so we can poke at it and see if it works as we'd like.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-repl-and-readline/31655/2