Use a process supervisor

[rtreffer]

The bmcd service might crash in an OOM situation. It is not restarted as that's not a thing in sys-v-init.

I would like to see/use a proper process supervisor. systemd has one on that front for heavy systems, busybox has stripped down runit implementation.

Shell scripts can be used for that. E.g. mysql famously used a chain of sys-v-init -> mysqld_safe -> mysqld. I would not recommend this, especially if a low disk overhead solution like runit is available.

See e.g. #178 or #162 for a case where the bmcd would stay down.

I have tried to build a BMC with runit, it's quite easy, but probably needs discussion if that's a desirable path forward.

From bb8a76e86de62a7ca7fe58d5f6eea8d87f4a63d0 Mon Sep 17 00:00:00 2001
From: rtreffer <[email protected]>
Date: Wed, 21 Feb 2024 15:28:49 +0100
Subject: [PATCH] Add busybox/runit to the base image

Runit is a simple process supervisor, suitable for e.g. bmcd.
---
 tp2bmc/configs/busybox.config   | 3 +++
 tp2bmc/configs/tp2bmc_defconfig | 3 +++
 2 files changed, 6 insertions(+)
 create mode 100644 tp2bmc/configs/busybox.config

diff --git a/tp2bmc/configs/busybox.config b/tp2bmc/configs/busybox.config
new file mode 100644
index 00000000..48345897
--- /dev/null
+++ b/tp2bmc/configs/busybox.config
@@ -0,0 +1,3 @@
+# enable runit service supervision
+CONFIG_RUNSV=y
+CONFIG_RUNSVDIR=y
diff --git a/tp2bmc/configs/tp2bmc_defconfig b/tp2bmc/configs/tp2bmc_defconfig
index c50414de..0247398b 100644
--- a/tp2bmc/configs/tp2bmc_defconfig
+++ b/tp2bmc/configs/tp2bmc_defconfig
@@ -60,6 +60,9 @@ BR2_TARGET_UBOOT_FORMAT_CUSTOM=y
 BR2_TARGET_UBOOT_FORMAT_CUSTOM_NAME="u-boot-sunxi-with-spl.bin"
 BR2_TARGET_UBOOT_CUSTOM_DTS_PATH="$(BR2_EXTERNAL_TP2BMC_PATH)/board/tp2bmc/sun8i-t113s-turingmachines-tp2-bmc.dts"
 
+# Busybox customizations
+BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="$(BR2_EXTERNAL_TP2BMC_PATH)/configs/busybox.config"
+
 # Host packages
 BR2_PACKAGE_HOST_DOSFSTOOLS=y
 BR2_PACKAGE_HOST_GENEXT2FS=y
-- 
2.43.0

[walkjivefly]

God, let's never burden bmcd with anything as unpleasant as systemd.

[srcshelton]

… why not simply add an entry to /etc/inittab with respawn declared as the action?

[j0ju]

I much as I like the simple idea of ~srchelton for using respawn in inittab, it has several disadvantage as an shell use you cannot stop the bmcd anymore, without modifying files.
init has no service control.

I would go either with a simple shell wrapper like mysqld_safe or even simpler or using runit of busybox.

If enabling runit tools of busybox, please do also add sv and chpst
The first ist for managing services like starting and stopping, the latter is for writing own ../run files for other services with supervision and get rid of buildroots borken init system for services. For system setup this is IMHO fine.

Note: the Linux on the BMC seems to have memory management issues if you have write pressure to block devices via the MSD mode of nodes. The page cache seems to grow quite fast with dirty unwritten pages.

# running this on another terminal in the background lowers the throughput but minimizes OOMkiller runs eventually killing bmcd.
while sync; do sleep 2; done

The best way would be to reduce bugs and errors instead of service supervision because of buggy software, unfortunatly here we have multiple components interacting.