YAET, a project I built using Xterm.js, has a WebviewAddon

[KernelDeimos]

Hello everyone, recently I released a terminal emulator under the MIT license using Xterm.js called YAET. What sets it apart from other terminal emulators is that shell scripts and other programs can display iframes using custom OSC sequences. Special thanks to @jerch on #2300 who helped me iron out how I formatted those.

This is very experimental so I already know dozens of things that are wrong with it, but I'd love to here about some more as well.

I wrote an addon for Xterm.js to get this working. Although most of my project is written in vanilla javascript I wrote the addon in typescript to keep the same conventions/structure used in other Xterm.js addons. That said, my addon is still really rough around the edges and I may have done some things improperly (and I typed things as any a few times like a chump; I'd definitely like to fix that). I mostly looked at the image addon as a reference, which is how I figured out how to use cell attributes to determine when I should render the iframes.

I've been asked about the usefulness of rendering webpages in the terminal. I'll explain briefly here. Once a postMessage API is added that can communicate back to the program running under the terminal, it will be possible to render rich UI components, fancy progress bars, and all sorts of things like that. Maybe not everyone's cup of tea, but I think it's very cool and it would not be possible without Xterm.js.

[jerch]

Wow - interesting, that you got it working this way. The image addon is a bit tricky as a base here - it does real cell alignment & cropping, which would not work with an iframe that easy (you cannot simply render half of an iframe, would need offset tricks instead). Thats also the reason for the (hacky) bufferline type extension - I had to put the image tile info into the terminal buffer somehow to allow real inline images, that correctly move with the scrollbuffer.

I see def. potential for such an addon, as I also strive for better document-based data handling in the terminal (see #4943 for a rough sketch). You also might want to have a look at DomTerm, imho @PerBothner already has built an HTML integration there.

That being said, I also want to place a warning here - security. HTML itself brings several vectors to the table, so you might want to secure/seclude the iframe as much as possible. Furthermore opening a webpage on the terminal's user side imposes a risk - the client side can be identified/tracked. Thats a surprising behavior in a terminal context, as they are typically used to do all communications through the terminal connection. So this either needs a very big disclaimer to teach peeps, or even better - try to figure out, how to properly re-route the page loading through the terminal session (really hard one, and prolly only possible in electron context).

[KernelDeimos]

Thanks for the feedback! I updated the readme on my project to note some security considerations and documented any security mechanisms I already put in place. Let me know if you'd like to be credited under the point about tracking the client - that's something which admittedly had not occurred to me. I also added a link to DomTerm, that's a really cool project - thanks for linking that here!

There's a postMessage API which enables some interesting things like these buttons:

I have a few things in place to make this as secure as possible
(only allowed on srcdoc iframes, visible ASCII-range input only, command to detach iframe) but it will be interesting to see if any pitfalls still arise anyway.

[jerch]

Let me know if you'd like to be credited under the point about tracking the client - that's something which admittedly had not occurred to me.

Not needed from my side.

Btw this "local machine" (where the user sits) vs. "remote host" (where the terminal session points to atm) is likely to create confusion without further explanation to users - imagine the remote host has different network access and sees host "a.b.c.d", while the local machine has no direct route to it. Then a ping a.b.c.d in the terminal works, but your iframe cannot load it. So its not only about security, but also network visibility in general. Your iframe basically side-channels from a different network entry point, if the terminal points to a remote host (thus may reveal the terminal user itself - that the fishy security aspect of it).

[PerBothner]

As @jerch mentioned, DomTerm has long supported embedded HTML (with sanitization, of course). The Features page and the Python tips page show some examples of what can be done.

See Wire-protocol - specifically "\e]72;" for inserting HTML and "\e]721;" for replacing previously-inserted HTML fragments. I believe "\e]721;" is a simple and flexible "API" that can be used to implement dynamic rich UI elemenets; no postMessage needed. Iframes are optional.

I have been working on HTML support for xterm.js in this branch.This builds on the BufferLine rewrite (PR #4928) branch. (Work on both has been very slow recently because I've been distracted by construction at my house.) I have enhanced DomTerm so it can experimentally and optionally use the xterm.js engine as opposed to DomTerm's native engine; I use this to test the xterm.js HTML branch.

While native DomTerm supports inlne HTML chunks, I believe the priority is "block" content (e.g. <div> elements). Most obviously as part of the "print" output of a REPL. It is critical that these HTML chunks can be scrolled, and serialized. Window resize should keep the HTML chunks and text chunks separate: Text resize should reflow (handled by the terminal code); HTML resize should generally be handled by browser engine. It is preferable that an HTML block should be treated by the terminal as a single logical "fat row" rather than some integral (and unpredictable) number of rows. Hence I build on top of the BufferLine rewrite.

[KernelDeimos]

This is fantastic! I've added a link to DomTerm on the readme for YAET. What you've mentioned about block content is really interesting because I'm taking a different approach that was natural for iframes: the iframe's height is pre-determined and stays the same for the span of its existence. Enabling both approaches in the same application would be very cool. I suspect it will be difficult to get this working right under terminal multiplexers like tmux.