Support restarting dbus-broker

[davide125]

Filing this to summarize our in-person conversations during DevConf and track any required work. Please let me know if I got anything wrong.

Currently restarting dbus-broker disconnects all clients from the bus. This mimics dbus-daemon behavior, and in fact dbus-daemon upstream recommends never restarting the daemon and rebooting the system instead. This is problematic for server deployments where reboots can be expensive, especially in case one needs to widely roll out a security update quickly. The scenarios are:

• support restarting a running dbus-broker service; this requires essentially checkpointing the current state (fds, connections, in flight messages, etc.) for the new process to restore
• support restarting dbus-broker after a bugfix/security update; this also requires updating the checkpointed state with any changes the new version might require
• support restarting dbus-broker after a major/feature update; this is an extension of the previous case

One way to handle this could be the "zero-downtime restart" approach where the old process forks off the new one, then stops accepting new connections and self terminates when it's done handling in-flight ones (nginx or haproxy are examples of this), using sd_notify to make systemd aware of the change; however, due to the nature of dbus this can potentially take an unbounded amount of time, so it may not be feasible in practice.

[pavlix]

I discussed this with @thom311 at devconf as well. Nice to see it's being considered.

[dvdhrm]

Just for completeness: I can see why this is handy. At the same time, this is a non-trivial extension.

In dbus-broker, we never access outside resources. The entire daemon is self-contained and fully controlled by its launcher. This makes restarting it easy (if you solve the serialization/deserialization part, which itself is a bigger effort). However, the outer part troubles me. The launcher we provide mirrors the behavior of dbus-daemon regarding configuration parsing, reloading, activation, etc. There might be circular dependencies there (like dbus-broker depending on systemd, and vice-versa), and it is not clear how such a system can be restarted in a safe way. Sure, you can just make them restart in parallel, and it will work most of the time. However, this is something I don't like being involved in. Distribution right now don't provide safe live-updated, and I feel a bit hesitant joining this unsafe area.

Long story short: Making dbus-broker restartable should be easy (in the sense of possible problems), but rather a bigger effort. Regarding the launcher, it is easy and less effort, but we would really have to mark this as "you're on your on there", if you restart across non-bugfix releases. While it will be consistent in itself, we can never guarantee consistency with outside dependencies, unless they provide a consistent restart-behavior.

[thom311]

An old discussion was here: https://wiki.ubuntu.com/DbusRestart

Occasionally this issue still pops up. Usually it pops up on the client-side, like "why does NetworkManager not reconnect to the bus when restarting dbus.service". But I still think that it doesn't scale that every client handles restart of the broker. IMO the only solution can be that the broker handles it transparently.

As @dvdhrm says, probably the broker could support it, if somebody implements it. But there also seems not enough necessity/interest for anybody to step up (which is fine)...

[kescherCode]

Restarting dbus-broker is already somewhat possible, as long as you make all clients reconnect somehow, either by having the client reconnect or by restarting them. (With systemd, you just do a daemon-reexec, and then you can deal with everything else). I already do this for server upgrades, and it beats the fact that a restart of the prior dbus-daemon made my systems unusable entirely beyond a reboot.

However, not having to do that would be amazing, of course.