From 425ac66ddc517ec4309b1f264867c70976b89122 Mon Sep 17 00:00:00 2001 From: d-kuro Date: Thu, 19 Oct 2023 14:49:19 +0900 Subject: [PATCH] Update proposal to allow individual stopping of clustering and reconciliation (#576) * Update proposal to allow individual stopping of clustering and reconciliation Signed-off-by: d-kuro --- docs/designdoc/clustering_stop.md | 88 +++++++++++++++++++++++-------- 1 file changed, 67 insertions(+), 21 deletions(-) diff --git a/docs/designdoc/clustering_stop.md b/docs/designdoc/clustering_stop.md index c187c398a..04cd1bc2f 100644 --- a/docs/designdoc/clustering_stop.md +++ b/docs/designdoc/clustering_stop.md @@ -11,30 +11,33 @@ Therefore, we are implementing a function to temporarily stop MOCO's clustering ## Goals * Enable pause and resume of MySQL Cluster clustering +* Enable pausing and resuming the reconciliation loop for the MySQLCluster resource * Operational with the `kubectl moco` command * Check the presence or absence of a pause with `kubectl get mysqlcluster` ## Non-goals -* Changes to any MySQLCluster field will not be processed while paused - * Reconciliation loop will be stopped * Recovery is not guaranteed in the event that the MySQL Cluster is damaged by user operations when resuming from a pause ## ActualDesign -Users can stop the MySQL Cluster clustering with the following command: +Users can stop the MySQL Cluster clustering or reconciliation with the following command: ```console -$ kubectl moco clustering stop +$ kubectl moco stop clustering +$ kubectl moco stop reconciliation ``` -To resume clustering, use the following command: +To resume clustering or reconciliation, use the following command: ```console -$ kubectl moco clustering start +$ kubectl moco start clustering +$ kubectl moco start reconciliation ``` -When you execute `kubectl moco clustering stop `, +### Clustering + +When you execute `kubectl moco stop clustering `, the `moco.cybozu.com/clustering-stopped: "true"` is added to the `.metadata.annotation` of MySQLCluster. ```yaml @@ -43,25 +46,25 @@ metadata: moco.cybozu.com/clustering-stopped: "true" ``` -A MySQLCluster with this annotation will not undergo reconcile. -Also, the MySQL clustering is stopped because the `ClusterManager` is stopped. +This annotation will stop the `ClusterManager` of the annotated MySQLCluster. +Reconciliation processes due to changes in other MySQLCluster fields will continue to be performed. ```go r.ClusterManager.Stop(req.NamespacedName) ``` The process of stopping the `ClusterManager` is carried out only at the first reconcile -after the execution of `kubectl moco clustering stop `. +after the execution of `kubectl moco stop clustering `. -The `ClusteringStopped` gets added to `.status.conditions` of MySQLCluster. -The time when the `ClusterManager` was stopped is recorded in `.lastTransitionTime`. -It also updates the `.status` of `Available` and `Healthy` to `Unknown`. +The `ClusteringActive` is added to `.status.conditions` of MySQLCluster. +Normally `True` is recorded, but `False` is recorded when the clustering is stopped. +Also, when the clustering is stopped, the `Available` and `Healthy` statuses are also updated to `Unknown`. ```yaml status: conditions: - - type: ClusteringStopped - status: "True" + - type: ClusteringActive + status: "False" lastTransitionTime: 2018-01-01T00:00:00Z - type: Available status: "Unknown" @@ -71,27 +74,70 @@ status: lastTransitionTime: 2018-01-01T00:00:00Z ``` -Also, `STOPPED` will be added to the table displayed when you do `kubectl get mysqlcluster`. +Also, `CLUSTERING ACTIVE` will be added to the table displayed when you do `kubectl get mysqlcluster`. ```console $ kubectl get mysqlcluster -NAME AVAILABLE HEALTHY STOPPED PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP -test Unknown Unknown True 0 2 +NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS CLUSTERING ACTIVE RECONCILE ACTIVE LAST BACKUP +test Unknown Unknown 0 2 False True ``` -When you execute `kubectl moco clustering start `, +When you execute `kubectl moco start clustering `, `moco.cybozu.com/clustering-stopped: "true"` is removed from `.metadata.annotation` of MySQLCluster. -The controller resumes the reconcile process for `MySQLCluster` that does not have the `moco.cybozu.com/clustering-stopped: "true"` annotation. +The controller resumes the `ClusterManager` for `MySQLCluster` that does not have the `moco.cybozu.com/clustering-stopped: "true"` annotation. When the `ClusterManager` resumes operation, the `.status` of `ClusteringStopped` in `.status.conditions` is updated to `False`, and the `.status` of other `.status.conditions` is also updated from `Unknown`. +### Reconciliation + +When you execute `kubectl moco stop reconciliation `, +the `moco.cybozu.com/reconciliation-stopped: "true"` is added to the `.metadata.annotation` of MySQLCluster. + +```yaml +metadata: + annotations: + moco.cybozu.com/reconciliation-stopped: "true" +``` + +The reconcile process will not be executed for MySQLCluster resources annotated with this annotation. +Therefore, even if changes are made to the fields of the MySQLCluster resource, changes will not be propagated to other resources. + +The only exception is the stopping and restarting of clustering by the `moco.cybozu.com/clustering-stopped` annotation. +Also, the `ClusterManager` does not stop even if the reconcile process is stopped. + +The `ReconciliationActive` gets added to `.status.conditions` of MySQLCluster. +Normally `True` is recorded, but `False` is recorded when the reconciliation is stopped. + +```yaml +status: + conditions: + - type: ReconciliationActive + status: "False" + lastTransitionTime: 2018-01-01T00:00:00Z +``` + +Also, `RECONCILE ACTIVE` will be added to the table displayed when you do `kubectl get mysqlcluster`. + +```console +$ kubectl get mysqlcluster +NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS CLUSTERING ACTIVE RECONCILE ACTIVE LAST BACKUP +test Unknown Unknown 0 2 True False +``` + +When you execute `kubectl moco start reconciliation `, +`moco.cybozu.com/reconciliation-stopped: "true"` is removed from `.metadata.annotation` of MySQLCluster. + +The controller will resume the reconcile of MySQLCluster resources that do not have `moco.cybozu.com/clustering-stopped: "true"`. +If `ReconciliationStopped` in `.status.conditions` is `True` during this time, the controller will update `ReconciliationStopped` to `False`. + ### Metrics The following metrics are added to the moco-controller. ``` moco_cluster_clustering_stopped{name="mycluster", namespace="mynamesapce"} 1 +moco_cluster_reconciliation_stopped{name="mycluster", namespace="mynamesapce"} 1 ``` -1 if the cluster is clustering stopped, 0 otherwise. +1 if the cluster is clustering or reconciliation stopped, 0 otherwise.