Persist state (and Station ID) across Docker upgrades

[bajtos]

A follow-up for our Slack discussion in https://space-meridian.slack.com/archives/C06RPCL6QGL/p1714047593651149.

I realised that our instructions for running Station Core don’t configure any persisted volume. All state files are kept inside the container. I think that means these state files can be lost, although I am not sure when exactly that would happen. When that happens, we lose the job count and Station ID keypair, everything will look like a fresh install of a new instance.

We have verified this is happening in our Station Core deployment running on Fly.io.

I'm just worried that we can't force them to implement a persistent volume
or can we? Otherwise we need to accept that there will be this noise in our data

We would have to detect whether we are running in Docker and whether STATE_ROOT is mounted as a persisted volume.

I don’t know how reliable this is, but if the STATE_DIR is a mounted volume, then it is listed in the list of all mounts inside the container.

❯ docker run -it --rm --network none -e STATE_ROOT=/state -v /tmp/state:/state debian /bin/bash
root@db797b6fc5a7:/# mount | grep $STATE_ROOT
/run/host_mark/private on /state type fakeowner (rw,nosuid,nodev,relatime,fakeowner)
root@db797b6fc5a7:/#station-public-dashboard 

For comparison:

❯ docker run -it --rm --network none -e STATE_ROOT=/tmp debian /bin/bash
root@171b4e474f30:/# mount | grep $STATE_ROOT
root@171b4e474f30:/#

Tasks:

• How to detect a mounted volume
• Update docker documentation
• Notify network, and set a deadline for migration

[bajtos]

Since we can't be 100% sure that the state is persisted outside of the running Docker container, I am not sure if Station Core can enforce this in a meaningful way. But! Maybe we can incentivise station instances that are old enough at the Meridian protocol level, e.g. by doubling the reward for instances older than 1 month. That way Station operators will have at least some motivation to comply.

[csmasterpath2023]

Has this feature already been implemented in the current version? I used the following command to create a container, but it failed. Can you help me？

docker run --name station --detach --env STATE_ROOT=/state --env FIL_WALLET_ADDRESS=0x000000000000000000000000000000000000dEaD ghcr.io/filecoin-station/core:20.4.1 -v /tmp/state1:/state

[juliangruber]

This hasn't been implemented yet. Can you please open a new issue, with this message and the error you received?

[bajtos]

Update docker documentation

We should explain to operators how to fix Error: EACCES: permission denied, mkdir errors they may encounter if the Docker daemon does not have permission to write to the state directory on the host.

See #456 (comment)

[bajtos]

Loosely related: space-meridian/roadmap#122