Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0/1 Completed Lingering Jobs #603

Open
coffee-time-design opened this issue Jul 18, 2024 · 0 comments
Open

0/1 Completed Lingering Jobs #603

coffee-time-design opened this issue Jul 18, 2024 · 0 comments

Comments

@coffee-time-design
Copy link

What steps did you take and what happened:
I updated the Helm charts across my k3s clusters, everything seemed to be fine, then I noticed a few days later that job pods were not going away after the job had run:
job.batch/cert-manager-default-kopia-j6thr-maintain-job-1721254904505 1/1 6s 3h59m
job.batch/cert-manager-default-kopia-j6thr-maintain-job-1721258868468 1/1 12s 173m
job.batch/cert-manager-default-kopia-j6thr-maintain-job-1721263194387 1/1 3m20s 101m

velero cert-manager-default-kopia-j6thr-maintain-job-17212588684659hdd 0/1 Completed 0 170m
velero cert-manager-default-kopia-j6thr-maintain-job-1721263194385q8f4 0/1 Completed 0 98m

What did you expect to happen:
I expected these objects to clear after completion.

Anything else you would like to add:
When I look at the logs of those pods, there are some long errors, that start with:
time="2024-07-17T23:27:55Z" level=warning msg="active indexes
and end with:
logSource="pkg/kopia/kopia_log.go:101" logger name="[index-blob-manager]" sublevel=error
But ultimately:
time="2024-07-17T23:27:57Z" level=info msg="Finished quick maintenance." logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:94" logger name="[shared-manager]"

But everything appears to be working just fine, backups are being taken.
The above errors also appear in the controller pod.
I am backing up to MinIO

Environment:
Helm Chart Values:

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.10.0 # Versions are here: https://github.com/vmware-tanzu/velero-plugin-for-aws
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins
configuration:
  backupStorageLocation:
  - name: default
    provider: aws
    bucket: cluster-velero
    credential:
      name: cluster-velero
      key: cloud
    config:
      region: us-east-1
      s3ForcePathStyle: "true"
      s3Url: "http://minio:9000"
  volumeSnapshotLocation:
  - name: default
    provider: aws
    config:
      region: us-east-1
  defaultVolumesToFsBackup: true
credentials:
  useSecret: true
  existingSecret: cluster-velero
deployNodeAgent: true
nodeAgent:
  resources:
    requests:
      cpu: 20m
      memory: 64Mi
  tolerations:
    - effect: "NoExecute"
      operator: "Equal"
      value: "true"
      key: "CriticalAddonsOnly"
schedules:
  cluster:
    disabled: false
    schedule: "0 0 * * *"
    template:
      ttl: "2160h"  # 90 days, expressed in hours
      storageLocation: default
      includedNamespaces:
      - "*" # Backup everything
  • helm version (use helm version): v3.13.1
  • helm chart version and app version (use helm list -n <YOUR NAMESPACE>): velero-7.1.1 1.14.0
  • Kubernetes version (use kubectl version):
    Client Version: v1.30.2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.5+k3s1
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: Baremetal
  • OS (e.g. from /etc/os-release): Ubuntu 23.10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant