Velero backup failing in my onprem kubernetes cluster

[ashoks27]

Hello Team ,

I have to installed Velero in my onprem Kubernetes cluster with IBM cloud as the storage, my Velero backups are getting failed with the below error.

I have attached the detailed log in the attachment. Please check and advise to move forward in it. Thanks.

Velero Pod logs :

time="2021-03-17T15:00:32Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
time="2021-03-17T15:00:32Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/1/0)" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:154"
time="2021-03-17T15:01:19Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://s3-api.us-geo.objectstorage.softlayer.net/ibm-iks-velro-bucket?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp: i/o timeout" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:175"
I0317 15:01:54.219893 1 request.go:621] Throttling request took 1.047265472s, request: GET:https://10.96.0.1:443/apis/autoscaling/v2beta1?timeout=32s
$

$ velero get backup
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup5 Failed 0 0 2021-03-17 14:55:18 +0000 GMT 29d default
$

velero_error_17Mar.txt

Regards,
Ashok Kumar S

[dsu-igeek]

We see this error: time="2021-03-17T14:53:48Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/1/0, Backup location "default" is unavailable: rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://s3-api.us-geo.objectstorage.softlayer.net/ibm-iks-velro-bucket?delimiter=%2F&list-type=2&prefix=: dial tcp: i/o timeout)" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:152"

Please check your connectivity to the object store. Are there firewall rules preventing you from reaching it?

[ashoks27]

@dsu-igeek Thanks for your response.. Currently the Cluster has internet access enabled on ports 80 and 443. Is there anyother specific FW requirement for Velero.. Please confirm

[ashoks27]

@dsu-igeek Could you please advise on the above query.

[ashoks27]

Hello @dsu-igeek
I have checked with my Network team and currently ports 443 and 80 enabled for oubound from the source cluster. Also as part of troubleshoot , we have opened all ports from source subnet and still it was not able to reach the Storage bucket.
Can you please suggest is there anything else we can try to fix this issue.

Firewall Logs after opening all ports :
Session ID: 490064, Policy name: Direct-WEB-Access/9, Timeout: 1790, Valid
In: xx.xx.xx.239/33769 --> 3.121.6.180/443;tcp, Conn Tag: 0x0, If: ae0.2107, Pkts: 4461, Bytes: 2224446,
Out: 3.121.6.180/443 --> 169.46.41.170/26592;tcp, Conn Tag: 0x0, If: ae1.0, Pkts: 3926, Bytes: 334097,
Session ID: 1143236, Policy name: Direct-WEB-Access/9, Timeout: 1778, Valid
In: xx.xx.xx.239/36562 --> 34.120.177.193/443;tcp, Conn Tag: 0x0, If: ae0.2107, Pkts: 13, Bytes: 1294,
Out: 34.120.177.193/443 --> 169.46.41.170/3478;tcp, Conn Tag: 0x0, If: ae1.0, Pkts: 13, Bytes: 4900,
Session ID: 1156694, Policy name: Direct-WEB-Access/9, Timeout: 1778, Valid
In: xx.xx.xx.239/48510 --> 185.199.109.133/443;tcp, Conn Tag: 0x0, If: ae0.2107, Pkts: 11, Bytes: 1160,
Out: 185.199.109.133/443 --> 169.46.41.170/28107;tcp, Conn Tag: 0x0, If: ae1.0, Pkts: 10, Bytes: 5281,
Session ID: 1214515, Policy name: Direct-WEB-Access/9, Timeout: 24, Valid
In: xx.xx.xx.239/56603 --> 193.33.30.39/123;udp, Conn Tag: 0x0, If: ae0.2107, Pkts: 1, Bytes: 76,
Out: 193.33.30.39/123 --> 169.46.41.170/28128;udp, Conn Tag: 0x0, If: ae1.0, Pkts: 1, Bytes: 76,
Session ID: 1410365, Policy name: Direct-WEB-Access/9, Timeout: 4, Valid
In: xx.xx.xx.239/54488 --> 171.66.97.126/123;udp, Conn Tag: 0x0, If: ae0.2107, Pkts: 1, Bytes: 76,
Out: 171.66.97.126/123 --> 169.46.41.170/2721;udp, Conn Tag: 0x0, If: ae1.0, Pkts: 1, Bytes: 76,
Total sessions: 5

Velero Error:

time="2021-03-24T11:45:55Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get http://s3-api.us-geo.objectstorage.softlayer.net/ibm-iks-velro-bucket?delimiter=%2F&list-type=2&prefix=backups%2F: dial tcp: i/o timeout" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:175"

[ashish-amarnath]

@ashoks27 here are some steps to troubleshoot:

1. kubectl -n velero exec -ti -- bash
2. Once in the Velero contaiener
3. try to resolve the name s3-api.us-geo.objectstorage.softlayer.net
nslookup s3-api.us-geo.objectstorage.softlayer.net
4. Try to ping s3-api.us-geo.objectstorage.softlayer.net to confirm network connectivity from the VeleroPod to the configured backup storage location.

From the logs it is very clear that Velero is not able to connect to the backup storage location.

[VickyWinner]

Was this issue resolved? I do have similar issue and the error is same. However, I can see the traffic under activity log for storage account.

I am using Azure blob storage for backup location.