Explore the stability and recovery mechanism of sidechain

[Kailai-Wang]

As we are approaching staging/prod setup, it's important to ensure the sidechain is stable enough:

• it keeps running without problems for a considerable time duration (> 1 month), while validateers joining and leaving
• explore the possibility of recovery (DB recovery?) in case the sidechain stalls

It's crucial as all sensitive data is stored in sidechain storage.

[Kailai-Wang]

related: #1084

[github-actions]

❗ This issue is stale because it has been open for 60 days with no activity.
Remove Stale label or update it, otherwise this issue will be closed in 7 days.
@litentry/parachain

[github-actions]

🔐 This issue was closed because there has been no activity for 7 days since it became stale.

[Kailai-Wang]

The recovery mechanism is still important.