test: Retry auth in checkClientCertAuthentication

[mvollmer]

No description provided.

[mvollmer]

I did see this locally with 50% chance until I seriously started looking for reasons, then it stopped failing, of course. Current theory: it is something intermediate that goes away with retrying.

[mvollmer]

Experiment results: The first password auth attempt in checkClientCertAuthentication sometimes fails with an internal error. Retrying it after five seconds always succeeds. Delaying the first try by ten seconds also always succeeds.

I have no idea why it fails the first time and why it succeeds the second time. I don't think we need to find out.

[martinpitt]

So perhaps the while ! id alice; do sleep 5; done check isn't sufficient, and we need to wait until password auth works? That might also explain some related flakes.

[mvollmer]

So perhaps the while ! id alice; do sleep 5; done check isn't sufficient, and we need to wait until password auth works?

During my testing, id alice either worked on first try, or would never succeed. The latter is probably samba-in-kubernetes/samba-container#160.

How would you wait until passwd auth works? I think just retrying inside do_test is ok.

[martinpitt]

@mvollmer Note that samba-in-kubernetes/samba-container#160 applied to the other container image that we used in the meantime. I went back to https://github.com/Fmstrat/samba-domain, and it at least made that problem better. You can check sssctl domain-status cockpit.lan if it's the same problem, but that was very binary: It either worked right away or never. So if you can "fix" this by retrying, it's definitively something else.

Not sure about checking if pwd auth works -- in that case we can just retry the whole curl thing I suppose. There's only so much yak that we can and want to shave 😁

[mvollmer]

Slightly amplified test run: https://cockpit-logs.us-east-1.linodeobjects.com/pull-19719-20231211-084650-1638835f-rhel-9-4-networking/log.html

[mvollmer]

It either worked right away or never. So if you can "fix" this by retrying, it's definitively something else.

No, the while ! id alice loop will time out and the test will fail right there. The password auth will fail after id alice has succeeded on first try, and the succeed after a retry.

[martinpitt]

Argh, the faulty flake8 unit-test container update does not affect just bots https://github.com/cockpit-project/bots/issues/5659 but also cockpit. That is urgent I suppose.

Note that this is still no-test.

Thanks!

[martinpitt]

Danke! I bow to the bots in their infinite wisdom for the final verdict, of course. 🙇

[martinpitt]

The "4" is funny. But we can keep this for the time being, and replace with .* if/when it becomes a problem.

[martinpitt]

Meh, TestAD failure. That's a different test, but possibly same root cause?

another case

Retried.

[martinpitt]

After 5 retries, I'll look the other way here.