Fix Home and End keys on macOS

[sshock]

Fix ruby/irb#330 by entering keyboard transmit mode (aka application mode) to ensure the escape sequences line up with terminfo.

Also, remove a hack that is no longer needed (see commit comment for explanation).

This replaces #462, which had some issues and was never finished up.

[sshock]

Note many Linux users are unaffected by this problem only because an unrelated bug is causing libncurses to fail to load, which causes reline to not use terminfo and instead rely on the comprehensive list.

That bug was fixed in fiddle and incorporated into ruby here in Oct 2022, but it's still broken in reline because reline needs to call Fiddle.dlopen instead of Fiddle::Handle.new.

I can fix that line, but I'll save it for a separate PR after this one goes in; and to be extra safe, I'll try to help get #433 in as well beforehand to reduce the chance of regression.

[sshock]

@ima1zumi @st0012 @tompng can you please take a look at this PR?

It fixes ruby/irb#330 once and for all, a bug that was reported 1.5 years ago; a variety of workarounds have been found since then, but we should fix it so people don't have to rely on workarounds.

It also removes an ugly hack that is no longer necessary thanks to this fix.

Also, keep in mind that the only reason more Linux users are not also seeing and reporting problems with Home and End is likely because (due to another bug) libncurses is failing to load, and then reline doesn't use terminfo at all but instead uses the comprehensive key binding list.

If you'd like to reproduce Home and End keys not working on Linux, 1. make sure you are using an xterm-based terminal, and 2. change Fiddle::Handle.new to Fiddle.dlopen in lib/reline/terminfo.rb and make sure libncurses is loading. Then you should see the problem. Then you can try again with the changes in this PR to see how it fixes the problem.

[tompng]

I confirmed that this pull request fixes the problem but I have some concerns.

Is application mode necessary?
I think no. Readline accepts both \eOH and \e[H for HOME key.

Does application mode always make escape sequence line up with terminfo?
No. In mac iTerm2, option+down_arrow sends \e\e[B(ESC + normal_mode_downarrow) instead of \e\eOB (ESC + application_mode_downarrow) in both modes.

The main problem is that Reline does not accept some escape sequence and such escape sequence exists more in normal mode, less in application mode. Using the less-problematic mode is a workaround. Not a fundamental fix.
It's worth fixing even if it's a workaround, but I prefer a workaround without using application mode like #433 (But should not set terminal emulator specific binding if terminfo is available) or #567.

[sshock]

I confirmed that this pull request fixes the problem but I have some concerns.

Is application mode necessary? I think no. Readline accepts both \eOH and \e[H for HOME key.

I understand the hesitation, but a quick search shows that other apps do it, and even some shells like zsh:

• ohmyzsh/ohmyzsh@ff0cafa

I think readline works because it relies on hard-coded mappings in the code rather than relying on terminfo.

Does application mode always make escape sequence line up with terminfo? No. In mac iTerm2, option+down_arrow sends \e\e[B(ESC + normal_mode_downarrow) instead of \e\eOB (ESC + application_mode_downarrow) in both modes.

I do agree that given the down_arrow sends \e[B in normal mode and \eOB in application mode, it would make more sense if option+down_arrow worked in a similar fashion.

But the argument is kinda moot because there is no terminfo entry for that. There is a terminfo capability named "kcud1" which means "down-arrow key", but there is no capname for "down-arrow key while holding down option". So we never would or even could add that to CAPNAME_KEY_BINDINGS.

The main problem is that Reline does not accept some escape sequence and such escape sequence exists more in normal mode, less in application mode.

I see it more this way: "The main problem is escape sequences exist in normal mode when an application runs in normal mode".

Terminfo can't provide two escape sequences for the same capname, so it just provides the application mode escape sequence. Therefore, applications that want to use terminfo should enter application mode.

Using the less-problematic mode is a workaround. Not a fundamental fix.

I see it as a fundamental fix because everything I've read suggests that applications that want to use terminfo should be in application mode, for example:

• https://stackoverflow.com/a/21965133

  • "terminfo only works under application mode"

• https://invisible-island.net/xterm/xterm.faq.html#xterm_arrows

  • Since termcaps and terminfo descriptions are written for full-screen applications, shells and similar programs often rely on built-in tables of escape sequences which they use instead.

It's worth fixing even if it's a workaround, but I prefer a workaround without using application mode like #433 (But should not set terminal emulator specific binding if terminfo is available) or #567.

If there are any serious concerns with entering application mode, then I agree with ditching this PR and going with #433 or #567 instead.

So far the only problem I can think of with enter application mode is just that it seems wrong to do that for something that is not actually a full-screen application (although with multi-line support it kinda gets closer). Practically, the only downside I can think of is that if a process is terminated abruptly, we might leave the terminal in application mode. I suppose that could be a valid concern.

Update: I did think of another downside to entering application mode. The time it takes to send the escape sequence for smkx (which in xterm is like 7 bytes) and rmkx (also 7 bytes) I think could cause a noticeable delay between prompts when running over a very slow connection. If we are concerned about this, then I think it is a good argument for choosing a different solution.

[tompng]

There are two application mode settings. DECKPAM(Application Keypad) and DECCKM(Application Cursor Keys). Emacs and Vim enables both. Enabling both will make the input sequence line up more with terminfo.
Why enable only one of the two setting? Actually, we cannot enable both now.
In https://invisible-island.net/xterm/ctlseqs/ctlseqs.txt, it describes that SS3 might be send for some keys.

Key              Numeric    Application   Terminfo   Termcap
---------------+----------+-------------+----------+----------
Space          | SP       | SS3 SP      | -        | -
Tab            | TAB      | SS3 I       | -        | -
Enter          | CR       | SS3 M       | kent     | @8
PF1            | SS3 P    | SS3 P       | kf1      | k1
PF2            | SS3 Q    | SS3 Q       | kf2      | k2
PF3            | SS3 R    | SS3 R       | kf3      | k3
PF4            | SS3 S    | SS3 S       | kf4      | k4
* (multiply)   | *        | SS3 j       | -        | -
+ (add)        | +        | SS3 k       | -        | -

I confirmed macOS Terminal.app sends SS3 M on accessibility virtual large keyboard's numpad ENTER key when both DECKPAM and DECCKM are enabled. We should handle these keys if we completely switching to application mode.

Currently, I don't feel we should use application mode, especially enabling only half of the settings. I think we should do a fix or workaround first without application mode. If there is more reason we should use application mode, we can reconsider it but in a separate issue from HOME and END keys.
After HOME and END is fixed, iTerm's OPTION+LEFT("\e" + "\e[D"), OPTION+RIGHT("\e" + "\e[C") which is supported in emacs but does not work in Reline problem is more prior than application mode IMO and application mode does not help solving this.

Appendix

In https://vt100.net/docs/vt510-rm/DECCKM.html, it says

If the DECCKM function is set, then the arrow keys send application sequences to the host.
If the DECCKM function is reset, then the arrow keys send ANSI cursor sequences to the host.

And in https://vt100.net/docs/vt100-ug/chapter3.html DECCKM section, it says

Under these conditions, if the cursor key mode is reset, the four cursor function keys will send ANSI cursor control commands.
If cursor key mode is set, the four cursor function keys will send application functions.

"application sequences" or "application functions" might be different and we should use terminfo.
"ANSI cursor control commands" or "ANSI cursor sequence" seems like a standard thing. I think hard-code standard things is not a bad idea.

In "Minimum requirements for VT100 emulation" section of https://www.real-world-systems.com/docs/ANSIcode.html, keyboards will send these sequences.

  [A       Sent by the up-cursor key (alternately ESC O A)
  [B       Sent by the down-cursor key (alternately ESC O B)
  [C       Sent by the right-cursor key (alternately ESC O C)
  [D       Sent by the left-cursor key (alternately ESC O D)
  OP       PF1 key sends ESC O P
  OQ       PF2 key sends ESC O Q
  OR       PF3 key sends ESC O R
  OS       PF3 key sends ESC O S

Since most terminal emulators are vt100 compatible, there is no ambiguity for these escape sequences. I think there are no need for terminfo-like thing for normal mode, but that does not mean we should always use application mode.

My plan is to add these keys and \e[F \e[H as default(cuu, cud, cuf, cub, end, home), interpret \e + key_sequence as meta_key + key_sequence, bind meta+cub to prev_word and fix OPTION+LEFT OPTION+RIGHT.

[sshock]

There are two application mode settings. DECKPAM(Application Keypad) and DECCKM(Application Cursor Keys). Emacs and Vim enables both. Enabling both will make the input sequence line up more with terminfo. Why enable only one of the two setting?

smkx enables both of them:

sshock@sshock:~$ infocmp -1 xterm | grep smkx
        smkx=\E[?1h\E=,

• \E[?1h = Application Cursor Keys (Set DECCKM)
• \E= = Application Keypad (DECPAM)

And rmkx resets them both:

sshock@sshock:~$ infocmp -1 xterm | grep rmkx
        rmkx=\E[?1l\E>,

• \E[?1l = Normal Cursor Keys (Reset DECCKM)
• \E> = Normal Keypad (DECPNM)

Actually, we cannot enable both now. In https://invisible-island.net/xterm/ctlseqs/ctlseqs.txt, it describes that SS3 might be send for some keys.

Key              Numeric    Application   Terminfo   Termcap
---------------+----------+-------------+----------+----------
Space          | SP       | SS3 SP      | -        | -
Tab            | TAB      | SS3 I       | -        | -
Enter          | CR       | SS3 M       | kent     | @8
PF1            | SS3 P    | SS3 P       | kf1      | k1
PF2            | SS3 Q    | SS3 Q       | kf2      | k2
PF3            | SS3 R    | SS3 R       | kf3      | k3
PF4            | SS3 S    | SS3 S       | kf4      | k4
* (multiply)   | *        | SS3 j       | -        | -
+ (add)        | +        | SS3 k       | -        | -

I confirmed macOS Terminal.app sends SS3 M on accessibility virtual large keyboard's numpad ENTER key when both DECKPAM and DECCKM are enabled. We should handle these keys if we completely switching to application mode.

Dang, I didn't notice that. And I confirmed as well with iTerm that hitting enter on my keypad sends "\EOM" in application mode.

So going into application mode will have a lot more side effects than I realized 😢 .

Currently, I don't feel we should use application mode, especially enabling only half of the settings. I think we should do a fix or workaround first without application mode. If there is more reason we should use application mode, we can reconsider it but in a separate issue from HOME and END keys.

Yeah, at this point I have to agree that we should not use application mode. Thanks for helping me see the other side effects it has.

In https://vt100.net/docs/vt510-rm/DECCKM.html, it says

If the DECCKM function is set, then the arrow keys send application sequences to the host.
If the DECCKM function is reset, then the arrow keys send ANSI cursor sequences to the host.

Yeah, based on my research as well, the sequences for arrow keys in normal mode appears to be pretty much standard across pretty much all terminals.

And in https://vt100.net/docs/vt100-ug/chapter3.html DECCKM section, it says

Under these conditions, if the cursor key mode is reset, the four cursor function keys will send ANSI cursor control commands.
If cursor key mode is set, the four cursor function keys will send application functions.

"application sequences" or "application functions" might be different and we should use terminfo. "ANSI cursor control commands" or "ANSI cursor sequence" seems like a standard thing. I think hard-code standard things is not a bad idea.

Agreed. I think we can feel good about hard-coding the escape sequences "\e[A", "\e[B", "\e[C", "\e[D" for the arrow keys as you have done in #567.

In "Minimum requirements for VT100 emulation" section of https://www.real-world-systems.com/docs/ANSIcode.html, keyboards will send these sequences.

  [A       Sent by the up-cursor key (alternately ESC O A)
  [B       Sent by the down-cursor key (alternately ESC O B)
  [C       Sent by the right-cursor key (alternately ESC O C)
  [D       Sent by the left-cursor key (alternately ESC O D)
  OP       PF1 key sends ESC O P
  OQ       PF2 key sends ESC O Q
  OR       PF3 key sends ESC O R
  OS       PF3 key sends ESC O S

Since most terminal emulators are vt100 compatible, there is no ambiguity for these escape sequences. I think there are no need for terminfo-like thing for normal mode, but that does not mean we should always use application mode.

Agreed. The cursor key escape sequences appear to be pretty standard. I have verified this as well with my research and some testing.

My plan is to add these keys and \e[F \e[H as default(cuu, cud, cuf, cub, end, home), interpret \e + key_sequence as meta_key + key_sequence, bind meta+cub to prev_word and fix OPTION+LEFT OPTION+RIGHT.

Sounds like a good plan. Although "\e[F" and "\e[H" for Home and End are not as standard as the cursor keys sequences, I think we should go ahead and push forward with #567.

[sshock]

Closing in favor of #567

[sshock]

Re-opening until after #567 is merged.

[tompng]

This is fixed by #569