[Question] Use param server in docker containers and with docker compose

[tiesus]

Hi,
first of all thanks for the code. Is use the the persistent parameter server in docker and i have a question regarding the save process of persistent params to the yaml file. On my local machine params a dumped to the yaml if i shutdown the node. When i stop the docker container via docker compose the params are gone and not written to the yaml. Do you have a hint to to enable saving directly after setting the parameter?

[fujitatomoya]

@tiesus thanks for creating issue.

i say that related to how to stop the docker, and how ros2run deals with signals. (I am assuming that you start the parameter server via ros2 run parameter_server server, right?)

docker container stop, the main process inside the container will receive SIGTERM, and after a grace period, SIGKILL. that means, ros2 run process receives SIGTERM 1st , not server process. ros2 run process dose not deal with signals as expected, so eventually this goes to graceful shutdown with SIGKILL. that kills all processes in container right away. and this SIGKILL kills server process gracefully too, so server cannot store cached parameter to the file system during shutdown.

i would suggest the followings,

1. (proper approach) take add signal handler SIGIN/SIGTERM to ros2run. ros2/ros2cli#899 to your workspace and build / run if the problem is solves.
2. (work around) Do not use ros2 run. instead, execute the server executable directly as following. This does not generate ros2 run wrapper process described above, and rclcpp default signal handler catches SIGTERM to shutdown the node server to call the destructor, which stores the all persistent parameter into the file system.

root@tomoyafujita:~/ros2_ws/colcon_ws# ./build/parameter_server/server
[WARN] [1725725716.364192116] [parameter_server]: Custom YAML file /tmp/parameter_server.yaml not exist
[INFO] [1725725716.364245602] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '6' parameters already!
...

Really appreciate the issue. IMO this should be described as known issue until ros2/ros2cli#899 gets fixed.

[fujitatomoya]

• host system use, persistent parameter can be stored in the file system as designed.

### start parameter server
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 run parameter_server server
[WARN] [1725823597.065626047] [parameter_server]: Custom YAML file /tmp/parameter_server.yaml not exist
[INFO] [1725823597.065694268] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '6' parameters already!
^C[INFO] [1725823612.564857987] [rclcpp]: signal_handler(signum=2)

### set persistent parameters
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.some_int 81
Set parameter successful
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.a_string Konnichiwa
Set parameter successful
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.pi 3.14159265359
Set parameter successful
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.some_lists.some_integers 81,82,83,84
Set parameter successful

### restart parameter server and check persistent parameters.
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param list
/parameter_server:
  persistent.a_string
  persistent.pi
  persistent.some_int
  persistent.some_lists.some_integers
  qos_overrides./parameter_events.publisher.depth
  qos_overrides./parameter_events.publisher.durability
  qos_overrides./parameter_events.publisher.history
  qos_overrides./parameter_events.publisher.reliability
  start_type_description_service
  use_sim_time

• but with container environment, this does not work as explained above.

tomoyafujita@~/DVT/docker_ws/ros2_colcon/src/ros2/ros2_persist_parameter_server >docker stop 6670d8e2faf4
6670d8e2faf4
tomoyafujita@~/DVT/docker_ws/ros2_colcon/src/ros2/ros2_persist_parameter_server >docker start 6670d8e2faf4
6670d8e2faf4

root@tomoyafujita:/# ls /tmp/parameter_server.yaml
ls: cannot access '/tmp/parameter_server.yaml': No such file or directory

• unfortunately work-around does not work either.

(work around) Do not use ros2 run. instead, execute the server executable directly as following. This does not generate ros2 run wrapper process described above, and rclcpp default signal handler catches SIGTERM to shutdown the node server to call the destructor, which stores the all persistent parameter into the file system.

root@tomoyafujita:~/docker_ws/ros2_colcon# ./build/parameter_server/server
[WARN] [1725824318.350773940] [parameter_server]: Custom YAML file /tmp/parameter_server.yaml not exist
[INFO] [1725824318.350850234] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '6' parameters already!
tomoyafujita@~/DVT >docker exec -it 6670d8e2faf4 /bin/bash
root@tomoyafujita:~# cd docker_ws/ros2_colcon/
root@tomoyafujita:~/docker_ws/ros2_colcon# source install/local_setup.bsah
bash: install/local_setup.bsah: No such file or directory
root@tomoyafujita:~/docker_ws/ros2_colcon# source install/local_setup.bash 
root@tomoyafujita:~/docker_ws/ros2_colcon# ./build/parameter_server/server
[WARN] [1725824459.661133633] [parameter_server]: Custom YAML file /tmp/parameter_server.yaml not exist
[INFO] [1725824459.661248271] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '6' parameters already!
^C[INFO] [1725824826.253083838] [rclcpp]: signal_handler(signum=2)

[fujitatomoya]

As far as I see, there are two things here.

1. We need to start the container main process with ros2 process.

   A container's main running process is the ENTRYPOINT and/or CMD at the end of the Dockerfile. It's best practice to separate areas of concern by using one service per container. That service may fork into multiple processes (for example, Apache web server starts multiple worker processes).

   this is up to user configuration for container with docker run, it suggests that some wrapper script to start the application.

2. And if that main process is ros2run, it requires add signal handler SIGIN/SIGTERM to ros2run. ros2/ros2cli#899 to bypass the signal to the executable. (I believe that in this case, directly executing executable work-around can be applied.)

I think above is the proper approach for production or practical application, but if we can use host system and docker command, the following would be acceptable too?

### start parameter server
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 run parameter_server server 
[WARN] [1725825939.074215736] [parameter_server]: Custom YAML file /tmp/parameter_server.yaml not exist
[INFO] [1725825939.074269331] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '6' parameters already!
...

### set the persistent parameter
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.some_int 81
Set parameter successful
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param set /parameter_server persistent.a_string Konnichiwa
Set parameter successful

### check the persistent parameters
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param list
/parameter_server:
  persistent.a_string
  persistent.some_int
  qos_overrides./parameter_events.publisher.depth
  qos_overrides./parameter_events.publisher.durability
  qos_overrides./parameter_events.publisher.history
  qos_overrides./parameter_events.publisher.reliability
  start_type_description_service
  use_sim_time

### (host system) send the SIGINT to the process in the container before stopping the container
tomoyafujita@~/DVT >docker exec 6670d8e2faf4 ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 13:01 pts/0    00:00:00 /bin/bash
root          19       0  0 13:02 pts/1    00:00:00 /bin/bash
root         104      19  0 13:05 pts/1    00:00:00 /usr/bin/python3 /root/docker_ws/ros2_colcon/install/ros2cli/bin/ros2 run parameter_server server
root         107     104  0 13:05 pts/1    00:00:00 /root/docker_ws/ros2_colcon/install/parameter_server/lib/parameter_server/server
root         121       0  0 13:05 pts/2    00:00:00 /bin/bash
root         196       1  0 13:06 pts/2    00:00:00 /usr/bin/python3 -c from ros2cli.daemon.daemonize import main; main() --name ros2-daemon --ros-domain-id 0 --rmw-implementation rmw_fastrtps_cpp
root         262       0  0 13:07 ?        00:00:00 ps -ef
tomoyafujita@~/DVT >docker exec 6670d8e2faf4 kill -SIGINT 107

### restart the container
tomoyafujita@~/DVT >docker stop 6670d8e2faf4
6670d8e2faf4
tomoyafujita@~/DVT >docker start 6670d8e2faf4
6670d8e2faf4

root@tomoyafujita:~/docker_ws/ros2_colcon# tomoyafujita@~/DVT >
tomoyafujita@~/DVT >
tomoyafujita@~/DVT >docker exec -it 6670d8e2faf4 /bin/bash
root@tomoyafujita:/# cd 
root@tomoyafujita:~# cd docker_ws/ros2_colcon/
root@tomoyafujita:~/docker_ws/ros2_colcon# source install/local_setup.bash 
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 run parameter_server server
[INFO] [1725826222.347952874] [parameter_server]: Parameter Server node named: '/parameter_server' started and ready, and serving '8' parameters already!
...

### check the parameters
root@tomoyafujita:~/docker_ws/ros2_colcon# ros2 param list
/parameter_server:
  persistent.a_string
  persistent.some_int
  qos_overrides./parameter_events.publisher.depth
  qos_overrides./parameter_events.publisher.durability
  qos_overrides./parameter_events.publisher.history
  qos_overrides./parameter_events.publisher.reliability
  start_type_description_service
  use_sim_time

[fujitatomoya]

@tiesus for now i can only provides the answer with work-around how to manage the container from the host system. #26 takes over some action items here as Feature Request to support the container images. So i will go ahead to close this question.

[tiesus]

Hi,

thank you for the quick reply and the detailed answer. For now, I think the approach you describe above works for use, since we do not need to reconfigure the application in the process to often and can use host system and docker command. When ros2/ros2cli#899 is merged i will add a proper solution to our systems but for now the fix will also work for us.

[fujitatomoya]

@tiesus that is the idea too! thanks!