cluster startup with couchbase gets 401 but values confirmed

[ksully-execute]

Using the github jans values file, altered for my existing cluster, and using couchbase.
Have the values adjusted, and verified that the superuser and restricted user values are correctly defined, and confirmed access through curl using the cnCouchbaseUrl and both cnCouchbaseUser/cncnCouchbasePassword and cnCouchbaseSuperUser/cnCouchbaseSuperUserPassword. The value work (converted to a basic auth header).
So why am I seeing messages like this?
WARNING - jans.pycloudlib.lock - 2023-12-01 19:50:08,275 - Backing off LockManager.check_connection(<LockRecord(adapter=CouchbaseLock)>) for 10.0 seconds; error='requests.exceptions.HTTPError: 40 │ │ 1 Client Error: Unauthorized for url: https://couchbasedev-0000.couchbase.svc.cluster.local:18091/pools/default/buckets'

│ WARNING - jans.pycloudlib.lock - 2023-12-01 19:50:28,123 - Giving up LockManager.check_connection(<LockRecord(adapter=CouchbaseLock)>) after 7 tries within 60.1 seconds; error='requests.exception │ │ s.HTTPError: 401 Client Error: Unauthorized for url: https://couchbasedev-0000.couchbase.svc.cluster.local:18091/pools/default/buckets'

and of course other components saying this, which I assume is because of the above?
WARNING - jans.pycloudlib.wait - 2023-12-01 19:53:23,671 - Config is not ready; reason=Config 'hostname' is not available; retrying in 10.0 seconds
Thanks for any thoughts

[ossdhaval]

Adding @misba7 for comments.

[misba7]

Config is not ready; reason=Config 'hostname' is not available shows that the config job didn't complete successfully. Since the config job populates hostname and other data in the configmap named cn, which didn't happen in your case.
This will subsequently cause the persistence job to fail too, and then the rest of the pods.
So try redeploying, and keep an eye on the config-job.

[ksully-execute]

Thanks @misba7 , I have watched the config job before, and the first thing that shows up is that warning. Is there a way to get further info (turn up debug or something)?
I've verified the couchbase credentials (cnCouchbaseUser/cnCouchbasePassword and cnCoubhaseSuperUser/cnCouchbaseSuperUserPassword) that I have the in yaml can access the cnCouchbaseURL that I have in there
cnCouchbaseUrl: couchbasedev-0000.couchbase.svc.cluster.local
So I'm not sure why it's getting a 401...any thoughts?

[misba7]

No, this warning is from other components, stating that the config-job pod didn't populate the configmap cn with its data(hostname being part of the data).

So that's why I am saying try to delete the jans namespace, do a fresh install and check the config-job pod.

[ksully-execute]

[ksully-execute]

also, just to confirm (not a k8s expert)
When I check the credentials against the couchbase pod, the name in chCouchbaseUrl isn't resolved from the command line of a control plane machine (which I believe is normal), but is resolvable through a dns lookup within a pod, like this:
`
Server: 10.96.0.10
Address: 10.96.0.10#53

Name: couchbasedev-0000.couchbase.svc.cluster.local
Address: 10.111.32.72
`

And then using that 10.111.32.71 address, the curl command like this work fine:
`curl -k --location https://10.111.32.72:18091/pools/default/buckets --header Content-Type: application/x-www-form-urlencoded --header 'Authorization: Basic ZGGV2MTpxZXdrajBwcXc5Z2EwOTRnOXF3aDRnYXdoZ2hQQVc0ODk='

curl -k --location https://10.111.32.72:18091/pools/default/buckets --header Content-Type: application/x-www-form-urlencoded --header 'Authorization: Basic ZGV2MTpxZXdrajBwcXc5Z2EwOTRnOXF3aDRnYXdoZ2hQQVc0ODk='
[{"name":"travel-sample","nodeLocator":"vbucket","bucketType":"membase","storageBackend":"couchstore","uuid":"f1da38c8faf3258a81047d293989b460","uri":"/pools/default/buckets/travel-sample?bucket_uuid=f1da38c8faf3258a81047d293989b460","streamingUri":"/pools/default/bucketsStreaming/travel-sample?bucket_uuid=f1da38c8faf3258a81047d293989b460","bucketCapabilitiesVer":"","bucketCapabilities":["collections","durableWrite","tombstonedUserXAttrs","couchapi","subdoc.ReplaceBodyWithXattr","subdoc.DocumentMacroSupport","subdoc.ReviveDocument","dcp.IgnorePurgedTombstones","dcp","cbhello","touch","cccp","xdcrCheckpointing","nodesExt","xattr"],"collectionsManifestUid":"1","ddocs":{"uri":"/pools/default/buckets/travel-sample/ddocs"},"vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":0,"serverList":["couchbasedev-0000.couchbasedev.couchbase.svc:11210"],"vBucketMap":[[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[0],[
`

[ksully-execute]

thanks for the point on the cert. My memory is not so good, appreciate the pointer to the prior discussion. Will change that

[ksully-execute]

cert done, and put testenvironment to true as well, no joy, same error on couchbase.
I guess I can try go get mysql or something just to get past this for now (couchbase being my persistence of choice).
I'll take any other suggestions though (see 2 posts up for notes on the verification on couchbase creds).
Thanks

[ksully-execute]

is there a way to get more debug into, specifically the basic auth head that it's trying to use?
Or perhaps a way to get the URL used for couchbase to be http instead of https?

[misba7]

There's appLoggers to configure log level for every component. For example you can configure global.auth-server.appLoggers here