supervisord Starts All Processes at the Same Time

[fredpalmer]

My config:

[program:redis-testapp]
command=/opt/bcs/bin/redis-server /apps/testapp/releases/current/environments/all/redis.conf
stdout_logfile=/var/log/redis_testapp_log
stderr_logfile=/var/log/redis_testapp_log
startsecs=30
priority=1
autostart=true
autorestart=true

[program:celerybeat-testapp]
command=python -O manage.py celerybeat --loglevel=INFO --schedule=/apps/testapp/db/celerybeat_schedule_db
stdout_logfile=/var/log/celerybeat_testapp_log
stderr_logfile=/var/log/celerybeat_testapp_log
priority=999
startsecs=5
autostart=true

[program:celery-testapp]
command=python -O manage.py celeryd --loglevel=INFO --events
stdout_logfile=/var/log/celeryd_testapp_log
stderr_logfile=/var/log/celeryd_testapp_log
priority=100
startsecs=10
autostart=true

[program:gunicorn-testapp]
command=gunicorn_django --workers=10 --log-level info --timeout 500 --bind=127.0.0.1:8004
stdout_logfile=/var/log/gunicorn_testapp_log
stderr_logfile=/var/log/gunicorn_testapp_log
priority=999
startsecs=10
autostart=true

[program:memcached-testapp]
command=/opt/bcs/bin/memcached -m 128 -l 127.0.0.1 -p 11212 -u nobody -P /apps/testapp/run/memcached.pid
stdout_logfile=/var/log/memcached_testapp_log
stderr_logfile=/var/log/memcached_testapp_log
priority=11
autostart=true
autorestart=true

My output =>

2012-05-30 22:37:33,181 INFO daemonizing the supervisord process
2012-05-30 22:37:33,182 INFO supervisord started with pid 16230
2012-05-30 22:37:34,195 INFO spawned: 'redis-testapp' with pid 16232
2012-05-30 22:37:34,206 INFO spawned: 'memcached-testapp' with pid 16233
2012-05-30 22:37:34,214 INFO spawned: 'celery-testapp' with pid 16234
2012-05-30 22:37:34,238 INFO spawned: 'celerybeat-testapp' with pid 16235
2012-05-30 22:37:34,477 INFO spawned: 'gunicorn-testapp' with pid 16241
2012-05-30 22:37:35,240 INFO success: memcached-testapp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-05-30 22:37:39,434 INFO success: celerybeat-testapp entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2012-05-30 22:37:44,434 INFO success: celery-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:37:44,435 INFO success: gunicorn-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:38:04,197 INFO success: redis-testapp entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)

What I expected to happen =>

redis would start and supervisord would wait 30 seconds before starting any lower priority processes.

[disposable-ksa98]

I don't see what's the point of using supervisor if it does not get this right.

@mnaberez If there is an open issue for this, you should post the link so we can find it.

[rca]

I also need the ability to start processes in a particular order.

An event-based approach could work. For example, if I have a [program:agent] and a [program:client] I could subscribe the client to start up when the agent emits a started event. By default, if a program does not subscribe to any event, it will be started when supervisord starts.

[sevastos]

+1

[maximilize]

+1

[tomislacker]

+1

[dmitriy-kiriyenko]

+1, this major issue.

[jefferai]

+1

[hunterloftis]

+1

[orarbel]

+1

[abierbaum]

+1

[gigaroby]

+1

[KenjiTakahashi]

👍

[karanlyons]

+1

[voleg]

+1

[toxsick]

+1

[harmy]

must have

[xbeta]

is there a dependency in supervisor? if not, why not?

[analytically]

+1

[devrim]

+1

[alp82]

+1

[mtrienis]

+1

[Pindar]

+1

[miguno]

Not wanting to add to any perceived pressure, but I'd be +1 on the very same feature (process dependencies). :-)

[nand0p]

this would be very useful and mean less ugly hacks like sleep

[jowenn]

+1

[pangon]

+1

[ricardosasilva]

+1. Maybe an option to turn the process loading synchronous.

[eugene-kulak]

+1

[mtdeguzis]

Please click the thumbs up on the original post instead of just commenting "+1". Gets my hopes up when I see a notification :D

[AdiBrucker]

On option to solve this problem is to define in the supervisor.conf that the program will get autostart value of false for example:

[program:test]
Autostart:” false”
Command=“/home/user/test”

Second step is to use bash script to start the supervisorctl example:

Gnome-terminal -e “supervisord”
Gnome-terminal -e “supervisorctl”
Sleep 10
Gnome-terminal -e “supervisorctl start test”

*Thanks to Avner gidron for giving me the idea

[kumarshorav11]

Hi,
I have below script in the supervisor.cnf file :

[program:zookeeper]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar zoo.jar"
priority=1
autostart = true
autorestart = true

[program:kafka]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar kaf.jar"
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
priority=999
autostart = true
autorestart = true

I want to start first Zookeeper and then Kafka when Zookeeper is up. This script is some times not working as expected.
What would be the way to handle this through supervisor.
Please suggest.

[mtdeguzis]

@kumarshorav11 it's not pretty, but what I do is monitor for the process to come up first (whatever needs to wait)/. This is an example only.

httpd.conf

[program:httpd]
command=/opt/supervisor/httpd_supervisor
autorestart=true
startretries=3
# Start only after PHP FPM
priority=2
# redirect output to stdout/stderr and do not use a regular logfile
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

/opt/supervisor/httpd_supervisor:

#!/bin/bash

# Wait until PHP FPM is up to start, since supervisord likes to start everything at once...
# See https://github.com/Supervisor/supervisor/issues/122

echo -e "\n==> Starting httpd"
# PHP FPM
while [[ $(ps -aux | grep "[p]hp-fpm: master process") == "" ]];
do
    echo "Waiting for PHP FPM to come up to start httpd..."
    sleep 2s
    if [[ $(ps -aux | grep "[p]hp-fpm: master process") != "" ]]; then
        echo "PHP FPM looks to be started, continuing with Icinga daemon initialization"
        ps -aux | grep "[p]hp-fpm: master process"
    fi
done

# Another project had this, but we won't be using systemd
# service httpd start
/usr/sbin/httpd

# Allow any signal which would kill a process to stop server
trap "service httpd stop" HUP INT QUIT ABRT ALRM TERM TSTP

while pgrep -u apache httpd > /dev/null; do sleep 5; done

[danielmotaleite]

Another team with this problem, we need to start a apps only after dnsmasq correctly started, if not the app will resolve and cache the wrong DNS

So there is already the https://github.com/jasoncorbett/ordered-startup-supervisord, why this is not merged in the main supervisor, is there any reason for this not to be fixed?

edit: another similar fork https://github.com/bendikro/supervisord-dependent-startup

[cpj001]

Any elegant solution?

[mrkeyiano]

i'm all for this feature

[benma]

It has been fun tracking this issue for all these years, but I finally unsubscribed 😂

[feistiny]

My case is that I just want to control the stopping order when manually not in supervisord control period, A > B,C ,
Simple solution: in a shell script, write them one by one,
supervisorctl stop A:*
supervisorctl stop B:* C:*
supervisorctl start A:* B:* C:*
Or if there are other than B,C, do a grep like this
supervisorctl status | grep -v '^A:*' | grep -v '^B:*' | grep -v '^C:*' | awk -F':' '{print $1":*"}' | xargs | supervisorctl restart

[msangel]

[b3ni]

+1

[blackwood821]

+1

[keu]

all these years of tracking just means that supervisor is an outdated relict and it is time to start using something better. it was good for its time though

[CharlieReitzel]

Without implementing full dependencies, a simple option for [program:x] could be set to wait for the program to reach the RUNNING state (or any terminal state like EXITED, FATAL or UNKNOWN). A boolean value would be good enough. Programs/groups at the same priority level can be started in parallel.

e.g.

[program:foo]
command=/usr/local/bin/foo --do-stuff
waitForStart=true

Extra credit: Use an integer property to specify the max # of seconds to wait. e.g.

[program:foo]
command=/usr/local/bin/foo --do-stuff
waitForStartSeconds=300

Some programs take a few minutes to start ...

This idea was inspired by this Stack Overflow answer.

[michaelhammann]

For my use cases I need this feature and I implemented a depends_on parameter which takes care that processes are only spawned once all dependees are in RUNNING and if not they are scheduled and will be spawned as soon as possible. All processes will be spawned in the expected order.
I highly appreciate any reviews or comments.
#1449
This parameter is for now tested and used with autostart=true, supervisorctl and rpcinterface.

[LMCom]

@michaelhammann After more than 9 years, you seem like the Messiah to bring redemption to us mere mortals. I am not exaggerating. Thanks!

[Vorticity-Flux]

I needed to start a bunch of independent processes sequentially to avoid overloading underlying services. I've ended up wrapping each process in a bash script that does locking and start synchronisation. Not ideal but does the job.

In the [program:x] section

command=/bin/start_rate_limit.sh /tmp/start.lock 2 600 service_executable
stopasgroup=true ; Probably not be necessary, but for a good measure

The start_rate_limit.sh script:

#!/usr/bin/env bash
if [ $# -ne 4 ]
then
  echo "Usage: $0 lock_file_path min_interval_between_starts_seconds lock_timeout_seconds command"
  exit 1
fi

exec 99>>$1 || exit 1
echo Waiting to start $4
flock -w $3 99 || exit 1
{ sleep $2; echo Rate limit Lock released; flock -u 99; } &
echo Starting $4
99>&- exec $4

[pluim003]

Wow, after almost 10 years no solution?

I had this issue when forking an 'old' repository where supervisord was being used. It had a sleep after the first command, but that didn't work. My solution was:

[supervisord]
nodaemon = true

[program:influxdb]
priority = 1
command = bash -c "/etc/init.d/influxdb start && sleep 5"
autostart=true
autorestart=true

[program:telegraf]
priority = 2
command = bash -c "sleep 15 && /etc/init.d/telegraf start"
autostart=true
autorestart=true

[program:grafana]
priority = 3
command = bash -c "sleep 30 && /etc/init.d/grafana-server start"
autostart=true
autorestart=true

So all 3 are spawned at the same time, but telegraf and grafana won't actually start after a few seconds. This seems to work fine here for a few days. Only when showing ps -ef | grep grafana (or telegraf) will show a process with sleep in it, but I can live with that.

Tried with startsecs=... and waitforstartseconds=.. but that didn't work with this docker container, or at least not in all test-cases, so I reverted back to the mentioned method with the sleep before starting the service.

[amano-kenji]

I just found https://github.com/davmac314/dinit which can run as a system init or a user process supervisor.

It looks simpler than supervisor and supports dependencies properly.

You guys might want to check out dinit.

I also looked into runit and s6. s6 is clunky as a user process supervisor. Runit doesn't support dependencies. OpenRC doesn't support being run as a user process supervisor. Systemd is no good for this, either.

[AndrewSav]

@amano-kenji s6 is not bad if you'd like to spend a bit of time understanding how it works, it is really flexible in what it can do. If the goal is to plug something in and it works without any configuration, s6 is definitely a poor choice.

[amano-kenji]

The dealbreaker was s6-rc which cannot be used if s6 is not PID 1.
It is significantly more complex than dinit.

dinit is simple.