Update proposal to allow individual stopping of clustering and reconc…

…iliation (#576)

* Update proposal to allow individual stopping of clustering and reconciliation

Signed-off-by: d-kuro <[email protected]>

docs/designdoc/clustering_stop.md

@@ -11,30 +11,33 @@ Therefore, we are implementing a function to temporarily stop MOCO's clustering
 ## Goals
 
 * Enable pause and resume of MySQL Cluster clustering
+* Enable pausing and resuming the reconciliation loop for the MySQLCluster resource
 * Operational with the `kubectl moco` command
 * Check the presence or absence of a pause with `kubectl get mysqlcluster`
 
 ## Non-goals
 
-* Changes to any MySQLCluster field will not be processed while paused
- * Reconciliation loop will be stopped
 * Recovery is not guaranteed in the event that the MySQL Cluster is damaged by user operations when resuming from a pause
 
 ## ActualDesign
 
-Users can stop the MySQL Cluster clustering with the following command:
+Users can stop the MySQL Cluster clustering or reconciliation with the following command:
 
 ```console
-$ kubectl moco clustering stop <CLSUTER_NAME>
+$ kubectl moco stop clustering <CLSUTER_NAME>
+$ kubectl moco stop reconciliation <CLSUTER_NAME>
 ```
 
-To resume clustering, use the following command:
+To resume clustering or reconciliation, use the following command:
 
 ```console
-$ kubectl moco clustering start <CLSUTER_NAME>
+$ kubectl moco start clustering <CLSUTER_NAME>
+$ kubectl moco start reconciliation <CLSUTER_NAME>
 ```
 
-When you execute `kubectl moco clustering stop <CLSUTER_NAME>`,
+### Clustering
+
+When you execute `kubectl moco stop clustering <CLSUTER_NAME>`,
 the `moco.cybozu.com/clustering-stopped: "true"` is added to the `.metadata.annotation` of MySQLCluster.
 
 ```yaml
@@ -43,25 +46,25 @@ metadata:
  moco.cybozu.com/clustering-stopped: "true"
 ```
 
-A MySQLCluster with this annotation will not undergo reconcile.
-Also, the MySQL clustering is stopped because the `ClusterManager` is stopped.
+This annotation will stop the `ClusterManager` of the annotated MySQLCluster.
+Reconciliation processes due to changes in other MySQLCluster fields will continue to be performed.
 
 ```go
 r.ClusterManager.Stop(req.NamespacedName)
 ```
 
 The process of stopping the `ClusterManager` is carried out only at the first reconcile
-after the execution of `kubectl moco clustering stop <CLSUTER_NAME>`.
+after the execution of `kubectl moco stop clustering <CLSUTER_NAME>`.
 
-The `ClusteringStopped` gets added to `.status.conditions` of MySQLCluster.
-The time when the `ClusterManager` was stopped is recorded in `.lastTransitionTime`.
-It also updates the `.status` of `Available` and `Healthy` to `Unknown`.
+The `ClusteringActive` is added to `.status.conditions` of MySQLCluster.
+Normally `True` is recorded, but `False` is recorded when the clustering is stopped.
+Also, when the clustering is stopped, the `Available` and `Healthy` statuses are also updated to `Unknown`.
 
 ```yaml
 status:
  conditions:
- - type: ClusteringStopped
- status: "True"
+ - type: ClusteringActive
+ status: "False"
  lastTransitionTime: 2018-01-01T00:00:00Z
  - type: Available
  status: "Unknown"
@@ -71,27 +74,70 @@ status:
  lastTransitionTime: 2018-01-01T00:00:00Z
 ```
 
-Also, `STOPPED` will be added to the table displayed when you do `kubectl get mysqlcluster`.
+Also, `CLUSTERING ACTIVE` will be added to the table displayed when you do `kubectl get mysqlcluster`.
 
 ```console
 $ kubectl get mysqlcluster
-NAME AVAILABLE HEALTHY STOPPED PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
-test Unknown Unknown True 0 2 <no value>
+NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS CLUSTERING ACTIVE RECONCILE ACTIVE LAST BACKUP
+test Unknown Unknown 0 2  False True <no value>
 ```
 
-When you execute `kubectl moco clustering start <CLSUTER_NAME>`,
+When you execute `kubectl moco start clustering <CLSUTER_NAME>`,
 `moco.cybozu.com/clustering-stopped: "true"` is removed from `.metadata.annotation` of MySQLCluster.
 
-The controller resumes the reconcile process for `MySQLCluster` that does not have the `moco.cybozu.com/clustering-stopped: "true"` annotation.
+The controller resumes the `ClusterManager` for `MySQLCluster` that does not have the `moco.cybozu.com/clustering-stopped: "true"` annotation.
 When the `ClusterManager` resumes operation, the `.status` of `ClusteringStopped` in `.status.conditions` is updated to `False`,
 and the `.status` of other `.status.conditions` is also updated from `Unknown`.
 
+### Reconciliation
+
+When you execute `kubectl moco stop reconciliation <CLSUTER_NAME>`,
+the `moco.cybozu.com/reconciliation-stopped: "true"` is added to the `.metadata.annotation` of MySQLCluster.
+
+```yaml
+metadata:
+ annotations:
+ moco.cybozu.com/reconciliation-stopped: "true"
+```
+
+The reconcile process will not be executed for MySQLCluster resources annotated with this annotation.
+Therefore, even if changes are made to the fields of the MySQLCluster resource, changes will not be propagated to other resources.
+
+The only exception is the stopping and restarting of clustering by the `moco.cybozu.com/clustering-stopped` annotation.
+Also, the `ClusterManager` does not stop even if the reconcile process is stopped.
+
+The `ReconciliationActive` gets added to `.status.conditions` of MySQLCluster.
+Normally `True` is recorded, but `False` is recorded when the reconciliation is stopped.
+
+```yaml
+status:
+ conditions:
+ - type: ReconciliationActive
+ status: "False"
+ lastTransitionTime: 2018-01-01T00:00:00Z
+```
+
+Also, `RECONCILE ACTIVE` will be added to the table displayed when you do `kubectl get mysqlcluster`.
+
+```console
+$ kubectl get mysqlcluster
+NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS CLUSTERING ACTIVE RECONCILE ACTIVE LAST BACKUP
+test Unknown Unknown 0 2 True False <no value>
+```
+
+When you execute `kubectl moco start reconciliation <CLSUTER_NAME>`,
+`moco.cybozu.com/reconciliation-stopped: "true"` is removed from `.metadata.annotation` of MySQLCluster.
+
+The controller will resume the reconcile of MySQLCluster resources that do not have `moco.cybozu.com/clustering-stopped: "true"`.
+If `ReconciliationStopped` in `.status.conditions` is `True` during this time, the controller will update `ReconciliationStopped` to `False`.
+
 ### Metrics
 
 The following metrics are added to the moco-controller.
 
 ```
 moco_cluster_clustering_stopped{name="mycluster", namespace="mynamesapce"} 1
+moco_cluster_reconciliation_stopped{name="mycluster", namespace="mynamesapce"} 1
 ```
 
-1 if the cluster is clustering stopped, 0 otherwise.
+1 if the cluster is clustering or reconciliation stopped, 0 otherwise.