Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: declarative publications and subscriptions #115

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
dafb56e
feat: declarative database management
leonardoce Aug 1, 2024
76d9770
feat: declarative publications and subscriptions
leonardoce Aug 20, 2024
3a6b698
chore: rebase on latest main
gabriele-wolfox Oct 10, 2024
0bd065b
Merge branch 'main' into declarative-pub-sub
gabriele-wolfox Oct 10, 2024
e6d6f43
fix: instance getter usage
gabriele-wolfox Oct 10, 2024
df10f9e
fix: manifests and linter
gabriele-wolfox Oct 10, 2024
a435207
fix: config CRD
gabriele-wolfox Oct 10, 2024
a0e8f4c
chore: add CRD examples for OLM build
gabriele-wolfox Oct 10, 2024
c0cfeb3
chore: split SQL functions in separated files
gabriele-wolfox Oct 14, 2024
3b8d353
test: add basic e2e
gabriele-wolfox Oct 14, 2024
afae552
test: add basic e2e
gabriele-wolfox Oct 14, 2024
0a8c831
Merge branch 'main' into declarative-pub-sub
gabriele-wolfox Oct 14, 2024
896213d
fix: update CRDs
gabriele-wolfox Oct 14, 2024
82c0177
Merge branch 'main' into declarative-pub-sub
gabriele-wolfox Oct 14, 2024
5467bcb
fix: owner set, e2e and conflict after rebase
gabriele-wolfox Oct 15, 2024
94629ca
fix: e2e manifests and code bug
gabriele-wolfox Oct 15, 2024
2aa6522
test: remove owners from manifests, relying on default
gabriele-wolfox Oct 16, 2024
28ae471
Merge branch 'main' into declarative-pub-sub
gabriele-wolfox Oct 16, 2024
a2508a9
test: add data inside table and assertion on replication
gabriele-wolfox Oct 16, 2024
6d3bffd
chore: simplify field type inside spec and add docs
gabriele-wolfox Oct 16, 2024
2f4b094
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 16, 2024
60ca9b6
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 17, 2024
f1f39d5
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 17, 2024
01974e7
chore: fix scaffolding
NiccoloFei Oct 18, 2024
6ff7194
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 18, 2024
b213ce2
test: review E2E
NiccoloFei Oct 18, 2024
520d4a0
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 18, 2024
17531ba
test: more e2e fixes
NiccoloFei Oct 18, 2024
8d54159
docs: update e2e.md
NiccoloFei Oct 18, 2024
576780c
test: patch finalizers before deletion
NiccoloFei Oct 18, 2024
bdd13a9
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 18, 2024
3d08386
test(e2e): handle object notFound
NiccoloFei Oct 21, 2024
b927fd6
Merge branch 'main' into declarative-pub-sub
NiccoloFei Oct 21, 2024
58d8b9f
review: improve doc, fix sample
jsilvela Oct 21, 2024
db0aa35
Update config/olm-samples/postgresql_v1_publication.yaml
jsilvela Oct 22, 2024
30d67bf
Update docs/src/samples/cluster-example-logical-source.yaml
jsilvela Oct 22, 2024
cee4d7b
chore: de-template text fixtures
jsilvela Oct 22, 2024
4665793
fix(deps): update github.com/cloudnative-pg/cnpg-i digest to 8d61352 …
renovate[bot] Oct 22, 2024
cd38220
chore: merge main branch changes
armru Oct 22, 2024
0dd54c9
chore: improve operation order of the publication_controller
armru Oct 22, 2024
cc5e9b2
chore: add reconcilefinalizer and remove nestif exception
armru Oct 22, 2024
cabb7f4
fix: mark publication as failed if we encounter `Get` errors while ob…
armru Oct 22, 2024
c7b9af9
refactor: uniform failed behaviour between subscription and publications
armru Oct 22, 2024
8b8d293
refactor: uniform mark as ready logic
armru Oct 22, 2024
9c27b04
refactor: uniform getCluster logic, fix regression about intervals
armru Oct 22, 2024
e16d388
refactor: centralize finalizer logic
armru Oct 23, 2024
7127602
chore: improve getCluster
armru Oct 23, 2024
8f0357e
refactor: prepare sql methods to be unit tested
armru Oct 23, 2024
a054120
fix: make alterSql handle parameters correctly
armru Oct 23, 2024
f0387d4
fix: create publication sql
armru Oct 23, 2024
6fc3b56
test(subscription): add basic coverage
armru Oct 23, 2024
fb8797c
chore: fixing order and linting
NiccoloFei Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .wordlist-en-custom.txt
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,12 @@ PrimaryUpdateStrategy
PriorityClass
PriorityClassName
ProjectedVolumeSource
PublicationReclaimPolicy
PublicationSpec
PublicationStatus
PublicationTarget
PublicationTargetAllTables
PublicationTargetObject
PullPolicy
QoS
Quaresima
Expand Down Expand Up @@ -423,6 +429,9 @@ StatefulSets
StorageClass
StorageConfiguration
Storages
SubscriptionReclaimPolicy
SubscriptionSpec
SubscriptionStatus
SuccessfullyExtracted
SwitchReplicaClusterStatus
SyncReplicaElectionConstraints
Expand Down Expand Up @@ -492,6 +501,7 @@ addons
affinityconfiguration
aks
albert
allTables
allnamespaces
alloc
allocator
Expand Down Expand Up @@ -736,6 +746,7 @@ executables
expirations
extensibility
externalCluster
externalClusterName
externalClusterSecretVersion
externalClusters
externalclusters
Expand Down Expand Up @@ -1056,6 +1067,10 @@ promotionTimeout
promotionToken
provisioner
psql
publicate
publicated
publicationName
publicationReclaimPolicy
pv
pvc
pvcCount
Expand Down Expand Up @@ -1200,6 +1215,7 @@ subcommand
subcommands
subdirectory
subresource
subscriptionReclaimPolicy
substatement
successfullyExtracted
sudo
Expand All @@ -1218,6 +1234,7 @@ syslog
systemd
sysv
tAc
tableExpression
tablespace
tablespaceClassName
tablespaceMapFile
Expand Down
18 changes: 18 additions & 0 deletions PROJECT
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,21 @@ resources:
kind: Database
path: github.com/cloudnative-pg/cloudnative-pg/api/v1
version: v1
- api:
crdVersion: v1
namespaced: true
controller: true
domain: cnpg.io
group: postgresql
kind: Publication
path: github.com/cloudnative-pg/cloudnative-pg/api/v1
version: v1
- api:
crdVersion: v1
namespaced: true
controller: true
domain: cnpg.io
group: postgresql
kind: Subscription
path: github.com/cloudnative-pg/cloudnative-pg/api/v1
version: v1
30 changes: 30 additions & 0 deletions api/v1/publication_funcs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
Copyright The CloudNativePG Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

// SetAsFailed sets the publication as failed with the given error
func (pub *Publication) SetAsFailed(err error) {
pub.Status.Ready = false
pub.Status.Error = err.Error()
}

// SetAsReady sets the subscription as working correctly
func (pub *Publication) SetAsReady() {
pub.Status.Error = ""
pub.Status.Ready = true
pub.Status.ObservedGeneration = pub.Generation
}
128 changes: 128 additions & 0 deletions api/v1/publication_types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
/*
Copyright The CloudNativePG Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// PublicationReclaimPolicy describes a policy for end-of-life maintenance of Publications.
// +enum
type PublicationReclaimPolicy string

const (
// PublicationReclaimDelete means the publication will be deleted from Kubernetes on release
// from its claim.
PublicationReclaimDelete PublicationReclaimPolicy = "delete"

// PublicationReclaimRetain means the publication will be left in its current phase for manual
// reclamation by the administrator. The default policy is Retain.
PublicationReclaimRetain PublicationReclaimPolicy = "retain"
)

// PublicationSpec defines the desired state of Publication
type PublicationSpec struct {
// The corresponding cluster
ClusterRef corev1.LocalObjectReference `json:"cluster"`

// The name inside PostgreSQL
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="name is immutable"
Name string `json:"name"`

// The name of the database
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="dbname is immutable"
DBName string `json:"dbname"`

// The owner
Owner string `json:"owner,omitempty"`

// Parameters
Parameters map[string]string `json:"parameters,omitempty"`

// Publication target
Target PublicationTarget `json:"target,omitempty"`

// The policy for end-of-life maintenance of this publication
// +kubebuilder:validation:Enum=delete;retain
// +kubebuilder:default:=retain
// +optional
ReclaimPolicy PublicationReclaimPolicy `json:"publicationReclaimPolicy,omitempty"`
}

// PublicationTarget is what this publication should publish
// +kubebuilder:validation:XValidation:rule="(has(self.allTables) && !has(self.objects)) || (!has(self.allTables) && has(self.objects))",message="allTables and objects are not compatible"
type PublicationTarget struct {
// All tables should be publicated
AllTables bool `json:"allTables,omitempty"`

// Just the following schema objects
Objects []PublicationTargetObject `json:"objects,omitempty"`
}

// PublicationTargetObject is an object to publicate
type PublicationTargetObject struct {
// The schema to publicate
Schema string `json:"schema,omitempty"`

// A list of table expressions
TableExpression []string `json:"tableExpression,omitempty"`
}

// PublicationStatus defines the observed state of Publication
type PublicationStatus struct {
// A sequence number representing the latest
// desired state that was synchronized
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

// Ready is true if the database was reconciled correctly
Ready bool `json:"ready,omitempty"`

// Error is the reconciliation error message
Error string `json:"error,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
// +kubebuilder:printcolumn:name="Cluster",type="string",JSONPath=".spec.cluster.name"
// +kubebuilder:printcolumn:name="PG Name",type="string",JSONPath=".spec.name"
// +kubebuilder:printcolumn:name="Ready",type="boolean",JSONPath=".status.ready"
// +kubebuilder:printcolumn:name="Error",type="string",JSONPath=".status.error",description="Latest error message"

// Publication is the Schema for the publications API
type Publication struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`

Spec PublicationSpec `json:"spec"`
Status PublicationStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// PublicationList contains a list of Publication
type PublicationList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Publication `json:"items"`
}

func init() {
SchemeBuilder.Register(&Publication{}, &PublicationList{})
}
30 changes: 30 additions & 0 deletions api/v1/subscription_funcs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
Copyright The CloudNativePG Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

// SetAsFailed sets the subscription as failed with the given error
func (sub *Subscription) SetAsFailed(err error) {
sub.Status.Ready = false
sub.Status.Error = err.Error()
}

// SetAsReady sets the subscription as working correctly
func (sub *Subscription) SetAsReady() {
sub.Status.Error = ""
sub.Status.Ready = true
sub.Status.ObservedGeneration = sub.Generation
}
113 changes: 113 additions & 0 deletions api/v1/subscription_types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
/*
Copyright The CloudNativePG Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// SubscriptionReclaimPolicy describes a policy for end-of-life maintenance of Subscriptions.
// +enum
type SubscriptionReclaimPolicy string

const (
// SubscriptionReclaimDelete means the subscription will be deleted from Kubernetes on release
// from its claim.
SubscriptionReclaimDelete SubscriptionReclaimPolicy = "delete"

// SubscriptionReclaimRetain means the subscription will be left in its current phase for manual
// reclamation by the administrator. The default policy is Retain.
SubscriptionReclaimRetain SubscriptionReclaimPolicy = "retain"
)

// SubscriptionSpec defines the desired state of Subscription
type SubscriptionSpec struct {
// The corresponding cluster
ClusterRef corev1.LocalObjectReference `json:"cluster"`

// The name inside PostgreSQL
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="name is immutable"
Name string `json:"name"`

// The owner
Owner string `json:"owner,omitempty"`

// The name of the database
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="dbname is immutable"
DBName string `json:"dbname"`

// Parameters
// +optional
Parameters map[string]string `json:"parameters,omitempty"`

// The name of the publication
PublicationName string `json:"publicationName"`

// The name of the external cluster with the publication
ExternalClusterName string `json:"externalClusterName"`

// The policy for end-of-life maintenance of this subscription
// +kubebuilder:validation:Enum=delete;retain
// +kubebuilder:default:=retain
// +optional
ReclaimPolicy SubscriptionReclaimPolicy `json:"subscriptionReclaimPolicy,omitempty"`
}

// SubscriptionStatus defines the observed state of Subscription
type SubscriptionStatus struct {
// A sequence number representing the latest
// desired state that was synchronized
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

// Ready is true if the database was reconciled correctly
Ready bool `json:"ready,omitempty"`

// Error is the reconciliation error message
Error string `json:"error,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
// +kubebuilder:printcolumn:name="Cluster",type="string",JSONPath=".spec.cluster.name"
// +kubebuilder:printcolumn:name="PG Name",type="string",JSONPath=".spec.name"
// +kubebuilder:printcolumn:name="Ready",type="boolean",JSONPath=".status.ready"
// +kubebuilder:printcolumn:name="Error",type="string",JSONPath=".status.error",description="Latest error message"

// Subscription is the Schema for the subscriptions API
type Subscription struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`

Spec SubscriptionSpec `json:"spec"`
Status SubscriptionStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// SubscriptionList contains a list of Subscription
type SubscriptionList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Subscription `json:"items"`
}

func init() {
SchemeBuilder.Register(&Subscription{}, &SubscriptionList{})
}
Loading
Loading