Sending kernel output to a separate console

[kuba-moo]

Having kernel dump to stdout makes output parsing harder. If I specify two console=ttySx in virtme/architectures.py bash doesn't start, saying:

bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

Just adding a serial device and trying to spawn terminal on it or send kernel logs to it with setlogcons does not do anything.

Is there a good way to get kernel to print to a separate console?

[arighi]

Kernel log is sent to stderr while script output is sent to stdout, so they're already separated. If you want to isolate the kernel log you can just redirect stderr somewhere, like vng -vr -- uname -r 2>kernel.log.

[kuba-moo]

Looks like it was because I su - user and that doesn't change the ownership of the pts. So it works fine if I run a new shell.

Now that I have the harness code, however, I'd prefer to stick to it 😅 We have 300+ tests, I boot 16 VMs and have them pick up tests to run from a queue. Speeds things up quite a bit, on the debug build boot takes 17sec.

[kuba-moo]

I take that back sorry, it still doesn't work for me in the new shell. Nothing in the logs, selinux is permissive, no aa. I have no idea :|

[arighi]

Hm... the only reasons that I can think of are: 1) security-related restrictions (AA, selinux, etc.), but it doesn't seem to be the case, 2) VFS restrictions, but I don't see why a forked process can't access the stdout of the shell session... you're using bash right? Also /proc/self/fd/1 should point to /dev/pts/N, do you have any special permission there? user is in the tty group? 3) special kernel changes / special boot parameters (maybe you are booting the kernel with some special options...?)

[kuba-moo]

Must be something with the sudo su - user switching. I can open fd1 as the "normal" user, but:

$ exec 3<> /proc/self/fd/1
$ echo bla >&3
bla
$ sudo su - virtme
Last login: Wed Jan 31 16:11:57 UTC 2024 on pts/1
$ ll /proc/self/fd/1
lrwx------. 1 virtme virtme 64 Jan 31 16:14 /proc/self/fd/1 -> /dev/pts/1
$ exec 3<> /proc/self/fd/1
-bash: /proc/self/fd/1: Permission denied

Give me 10min, I need to break out retsnoop :S

[kuba-moo]

The error comes from generic_permission() so not security. Everything is inlined there so hard to trace.

Anyway, I think we got sidetracked :) Can we also make the non-command version print the kernel logs to stderr?

[arighi]

The error comes from generic_permission() so not security. Everything is inlined there so hard to trace.

Is the virtme user in the tty group?

Anyway, I think we got sidetracked :) Can we also make the non-command version print the kernel logs to stderr?

It should be doable, I'll look at that.

[arighi]

hm... nevermind my user is not in the tty group either, but everything works. I'll try to reproduce the problem locally switching to a different user.

[arighi]

ahh ok, I think I see the problem:

18:30 arighi@gpd$ ls -l /proc/self/fd/1
lrwx------ 1 arighi arighi 64 Jan 31 18:30 /proc/self/fd/1 -> /dev/pts/14
6.7.0-4-generic ~
18:30 arighi@gpd$ ls -l /dev/pts/14
crw--w---- 1 arighi tty 136, 14 Jan 31 18:30 /dev/pts/14
6.7.0-4-generic ~
18:30 arighi@gpd$ sudo su - sbuild
sbuild@gpd:~$ ls -l /proc/self/fd/1
lrwx------ 1 sbuild sbuild 64 Jan 31 18:31 /proc/self/fd/1 -> /dev/pts/15
sbuild@gpd:~$ ls -l /dev/pts/15
crw--w---- 1 root tty 136, 15 Jan 31 18:31 /dev/pts/15

When you sudo su - user the pts has root:tty ownership, instead with a "regular login user" the pts is owned by the user itself.

Not sure if there's a more elegant way, but an easy workaround could be to add user virtme to the group tty.

[kuba-moo]

Ah, you're right! That's it! I can run it in tmux or screen or something that spawns a new pts. Best thing virtme can do is print some suggestion like "try running in tmux"? Probably not worth it, my sudo su-ing thing may just be weird.

With that I can confirm that running a command does indeed print kernel logs to stderr. But that's only if we run a script, if we start a bash session everything goes to stdout. Is there a reason for that or simple matter of not being coded up, yet?

[arighi]

I was actually planning to add a check to verify if /proc/self/fd/{0,1,2} are usable and if not maybe print an explicit warning, it's pretty easy and it can probably save a lot of time for anyone that may run vng in a similar environment.

About sending the kernel log to stderr in interactive mode, no particular reason, just not coded yet and I think it would be nice to have it. I'll take a look. :)

[arighi]

...

With that I can confirm that running a command does indeed print kernel logs to stderr. But that's only if we run a script, if we start a bash session everything goes to stdout. Is there a reason for that or simple matter of not being coded up, yet?

Hey @kuba-moo let me know what you think about #66. I haven't been able to figure out a way to redirect earlyprintk messages, but at least we can redirect all the other printk's to stderr with this one applied.