Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to automate and schedule backups #553

Merged
merged 45 commits into from
Jun 3, 2024
Merged

Add the ability to automate and schedule backups #553

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
7666e88
Add VitessBackupSchedule CRD
frouioui May 8, 2024
51f9653
Added the VitessBackupSchedule controller for the standalone CRD
frouioui May 10, 2024
eec5c02
Add VitessBackupSchedule to VitessCluster and clean up the CRD
frouioui May 13, 2024
4586474
Clean up VitessCluster controller for VitessBackupSchedule
frouioui May 13, 2024
fd30010
Use proper Docker Image in VitessBackupSchedule and re-add fields
frouioui May 13, 2024
b6c0150
Add backup strategies and fix VitessCluster reconcile loop for vbsc
frouioui May 13, 2024
bfe82b0
Allow for deletion of VitessBackupSchedule
frouioui May 14, 2024
2e4fd65
Enable backup for strategy 'backupShard'
frouioui May 14, 2024
33c23a0
Enable backup for strategy 'backupTablet'
frouioui May 14, 2024
49166e3
Refactor VitessBackupSchedule reconciling loop
frouioui May 14, 2024
b961d35
List Jobs using a label filter
frouioui May 15, 2024
9216b59
Support multiple schedules at the same time
frouioui May 15, 2024
4177d3d
Remove non-required logging
frouioui May 15, 2024
def8ac3
Allow for multiple backup strategies per schedule
frouioui May 15, 2024
9b1ce77
Remove non-required logging
frouioui May 15, 2024
dc2aa18
Merge remote-tracking branch 'origin/main' into scheduled-backups
frouioui May 16, 2024
062d7a0
Self-review
frouioui May 16, 2024
d8cec2a
Re-generate CRDs and operator.yaml
frouioui May 16, 2024
3347bed
Add timeout for stale jobs
frouioui May 16, 2024
cd77eb3
Add E2E test for scheduled backups
frouioui May 16, 2024
a129b4c
Modify backup strategy to use a flag map
frouioui May 16, 2024
614f31d
Revert debug code
frouioui May 16, 2024
95ff978
Fix the backup schedule test to expect 2 every-five-minute pods
frouioui May 17, 2024
ed10321
Add default to podJobTimeout
frouioui May 17, 2024
f45dc13
Fix clock skew issue
frouioui May 20, 2024
bd70ae6
Merge remote-tracking branch 'origin/main' into scheduled-backups
frouioui May 24, 2024
e6e0e45
Apply review suggestions
frouioui May 24, 2024
8cbfcad
Change CRD example to be every day instead of every minute
frouioui May 24, 2024
6e20613
Revert errexit change in test script
frouioui May 24, 2024
bc74ab4
Add BackupCluster and BackupKeyspace
frouioui May 24, 2024
b30aa09
Remove Cluster field in Strategy
frouioui May 28, 2024
4dc9ce0
Add release notes
frouioui May 28, 2024
9f7cd44
Fix 101 schedule test
frouioui May 28, 2024
54c9512
Use env variable for the golang version
frouioui May 28, 2024
e6946fb
Add custom annotations and affinity to backup pods
frouioui May 30, 2024
9739a94
Add more information in the release notes about extraFlags
frouioui May 30, 2024
46b6967
Fix metrics and add new metrics for timed out jobs
frouioui May 30, 2024
5809cbd
Revert unwanted change
frouioui May 30, 2024
70ba063
Revert BackupKeyspace and BackupCluster
frouioui May 30, 2024
6dbd28a
Merge remote-tracking branch 'origin/main' into scheduled-backups
frouioui May 30, 2024
f74592b
Revert panic issue
frouioui May 31, 2024
36d1b40
Removed Replace concurrency policy
frouioui Jun 3, 2024
54ec222
Fix review suggestions
frouioui Jun 3, 2024
1458329
Fix fmt
frouioui Jun 3, 2024
290b3a1
Fix out of bound
frouioui Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,25 @@ steps:
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"

- name: "Backup Schedule Test"
command:
- apk add g++ make bash gcompat curl mysql mysql-client libc6-compat
- wget https://golang.org/dl/go1.22.3.linux-amd64.tar.gz
frouioui marked this conversation as resolved.
Show resolved Hide resolved
- tar -C /usr/local -xzf go1.22.3.linux-amd64.tar.gz
- export PATH=$PATH:/usr/local/go/bin
- rm go1.22.3.linux-amd64.tar.gz
- ln -s /lib/libc.so.6 /usr/lib/libresolv.so.2
frouioui marked this conversation as resolved.
Show resolved Hide resolved
- make backup-schedule-test
concurrency: 1
concurrency_group: 'vtop/backup-schedule-test'
timeout_in_minutes: 60
plugins:
- docker#v3.12.0:
image: "docker:latest"
propagate-environment: true
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"

- name: "VTOrc and VTAdmin Test"
command:
- apk add g++ make bash gcompat curl mysql mysql-client libc6-compat chromium
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,5 @@ tags
### Intellij IDEs ###
.idea/*
# End of https://www.gitignore.io/api/go,vim,emacs,visualstudiocode

vtdataroot
frouioui marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ backup-restore-test: build e2e-test-setup
echo "Running Backup-Restore test"
test/endtoend/backup_restore_test.sh

backup-schedule-test: build e2e-test-setup
echo "Running Backup-Schedule test"
test/endtoend/backup_schedule_test.sh

vtorc-vtadmin-test: build e2e-test-setup
echo "Running VTOrc and VtAdmin test"
test/endtoend/vtorc_vtadmin_test.sh
162 changes: 162 additions & 0 deletions deploy/crds/planetscale.com_vitessbackupschedules.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.11.3
creationTimestamp: null
name: vitessbackupschedules.planetscale.com
spec:
group: planetscale.com
names:
kind: VitessBackupSchedule
listKind: VitessBackupScheduleList
plural: vitessbackupschedules
singular: vitessbackupschedule
scope: Namespaced
versions:
- name: v2
schema:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
allowedMissedRun:
minimum: 0
type: integer
concurrencyPolicy:
enum:
- Allow
- Forbid
- Replace
example: Forbid
type: string
failedJobsHistoryLimit:
format: int32
minimum: 0
type: integer
image:
type: string
imagePullPolicy:
type: string
jobTimeoutMinute:
default: 10
format: int32
minimum: 0
type: integer
name:
example: every-minute
minLength: 1
pattern: ^[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?$
type: string
resources:
properties:
claims:
items:
properties:
name:
type: string
required:
- name
type: object
type: array
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
type: object
type: object
schedule:
example: '* * * * *'
minLength: 0
type: string
startingDeadlineSeconds:
format: int64
minimum: 0
type: integer
strategies:
items:
properties:
extraFlags:
additionalProperties:
type: string
type: object
keyspaceShard:
example: commerce/-
type: string
name:
enum:
- BackupTablet
- BackupShard
type: string
tabletAlias:
example: zone1-0000000102
type: string
required:
- name
type: object
minItems: 1
type: array
successfulJobsHistoryLimit:
format: int32
minimum: 0
type: integer
suspend:
type: boolean
required:
- name
- resources
- schedule
- strategies
type: object
status:
properties:
active:
items:
properties:
apiVersion:
type: string
fieldPath:
type: string
kind:
type: string
name:
type: string
namespace:
type: string
resourceVersion:
type: string
uid:
type: string
type: object
x-kubernetes-map-type: atomic
type: array
lastScheduledTime:
format: date-time
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
102 changes: 102 additions & 0 deletions deploy/crds/planetscale.com_vitessclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,108 @@ spec:
type: object
minItems: 1
type: array
schedules:
items:
properties:
allowedMissedRun:
minimum: 0
type: integer
concurrencyPolicy:
enum:
- Allow
- Forbid
- Replace
example: Forbid
type: string
failedJobsHistoryLimit:
format: int32
minimum: 0
type: integer
jobTimeoutMinute:
default: 10
format: int32
minimum: 0
type: integer
name:
example: every-minute
minLength: 1
pattern: ^[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?$
type: string
resources:
properties:
claims:
items:
properties:
name:
type: string
required:
- name
type: object
type: array
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
type: object
type: object
schedule:
example: '* * * * *'
minLength: 0
type: string
startingDeadlineSeconds:
format: int64
minimum: 0
type: integer
strategies:
items:
properties:
extraFlags:
additionalProperties:
type: string
type: object
keyspaceShard:
example: commerce/-
type: string
name:
enum:
- BackupTablet
- BackupShard
type: string
tabletAlias:
example: zone1-0000000102
type: string
required:
- name
type: object
minItems: 1
type: array
successfulJobsHistoryLimit:
format: int32
minimum: 0
type: integer
suspend:
type: boolean
required:
- name
- resources
- schedule
- strategies
type: object
type: array
subcontroller:
properties:
serviceAccountName:
Expand Down
1 change: 1 addition & 0 deletions deploy/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ resources:
- crds/planetscale.com_vitessbackups.yaml
- crds/planetscale.com_vitessbackupstorages.yaml
- crds/planetscale.com_etcdlockservers.yaml
- crds/planetscale.com_vitessbackupschedules.yaml
9 changes: 9 additions & 0 deletions deploy/role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,5 +68,14 @@ rules:
- vitessbackupstorages
- vitessbackupstorages/status
- vitessbackupstorages/finalizers
- vitessbackupschedules
- vitessbackupschedules/status
- vitessbackupschedules/finalizers
verbs:
- '*'
- apiGroups:
- batch
resources:
- jobs
frouioui marked this conversation as resolved.
Show resolved Hide resolved
verbs:
- '*'
Loading
Loading