[Feature]: CRD Versions Management

[iGxnon]

Description about the feature

It will be a good enhancement if the operator could automatically perform migration from old versions to latest.

When multiple versions are applied to the k8s cluster, conflicts arise with Kubernetes CRD regulations because the subsequent version also has the storage: true setting.

Current Version

If we try to apply a higher version of CRD, the operator will panic and refuse to start.

2023-08-07T12:00:09.337960Z DEBUG kube_client::client: Unsuccessful: ErrorResponse { status: "Failure", message: "CustomResourceDefinition.apiextensions.k8s.io \"xlineclusters.xlineoperator.xline.cloud\" is invalid: status.storedVersions[0]: Invalid value: \"v1alpha\": must appear in spec.versions", reason: "Invalid", code: 422 }
Error: ApiError: CustomResourceDefinition.apiextensions.k8s.io "xlineclusters.xlineoperator.xline.cloud" is invalid: status.storedVersions[0]: Invalid value: "v1alpha": must appear in spec.versions: Invalid (ErrorResponse { status: "Failure", message: "CustomResourceDefinition.apiextensions.k8s.io \"xlineclusters.xlineoperator.xline.cloud\" is invalid: status.storedVersions[0]: Invalid value: \"v1alpha\": must appear in spec.versions", reason: "Invalid", code: 422 })

Caused by:
    CustomResourceDefinition.apiextensions.k8s.io "xlineclusters.xlineoperator.xline.cloud" is invalid: status.storedVersions[0]: Invalid value: "v1alpha": must appear in spec.versions: Invalid

Code of Conduct

• I agree to follow this project's Code of Conduct

[iGxnon]

We need to use kube::core::crd::merge_crds to merge multiple versions or set the stored version to the latest by first patching down the previous stored version and then patch the latest version.

[iGxnon]

1. We should ensure that a new CRD version is backward compatible so that we can select the latest version as the stored version when merging CRDs.
2. When a breaking change occurs in CRD, we should not directly set the stored version to the latest one. This could potentially lead to inconsistent in the currently running CRs. We should ensure that all CRs are at the latest version before setting the stored value of the latest version to true.

[liangyuanpeng]

This is more of a feature or enhancement and not a bug. This is a problem that every operator needs to working for it.

[iGxnon]

This is more of a feature or enhancement and not a bug. This is a problem that every operator needs to working for it.

Thx, I'll change the title

[liangyuanpeng]

After a quick read of the project, I probably understand your motivation. I think https://github.com/xline-kv/xline-operator/tree/main/operator-k8s/src/crd/v1alpha can be removed, it may be a wrong version, the correct one is v1alpha1.

WDYT @iGxnon

xline-operator/operator-k8s/src/crd/mod.rs

Lines 1 to 12 in f3000b7

	/// v1alpha
	/// Features:
	/// 1. Basic deployment
	/// 2. Scale cluster
	/// 3. Xline data PV
	pub(crate) mod v1alpha;
	/// v1alpha1
	/// Features:
	/// 1. Xline sidecar
	/// 2. PV backup
	pub(crate) mod v1alpha1;

Seems like it's the right thing.

[iGxnon]

After a quick read of the project, I probably understand your motivation. I think https://github.com/xline-kv/xline-operator/tree/main/operator-k8s/src/crd/v1alpha can be removed, it may be a wrong version, the correct one is v1alpha1.

WDYT @iGxnon

xline-operator/operator-k8s/src/crd/mod.rs

Lines 1 to 12 in f3000b7

	/// v1alpha
	/// Features:
	/// 1. Basic deployment
	/// 2. Scale cluster
	/// 3. Xline data PV
	pub(crate) mod v1alpha;
	/// v1alpha1
	/// Features:
	/// 1. Xline sidecar
	/// 2. PV backup
	pub(crate) mod v1alpha1;

Seems like it's the right thing.

Not a big problem. As the alpha version is not quit stable, I plan to delete all previous alpha versions after releasing a new alpha/beta/stable version.
And the version v1alpha could be considered as v1alpha0 ?