Add an e2e tests that detects error logs during etcd bootstrap

[serathius]

What would you like to be added?

Etcd should not write any error logs during normal operation. We could add a test that starts etcd cluster, does some simple operations and validates if there is any error level log written.

This could be further improved by enabling this check in all e2e tests when cluster/server is being closed. As some test intentionally inject errors, we could make it configurable option that is enabled by default and disabled in all tests we expect error to be written.

Why is this needed?

Logs are used to provide useful debug information to users, however if mislabeled they could cause user to lose trust and ignore errors. For error logs to be useful they should only be emitted where there is something wrong.

We should prevent issues like #17245 where a change in code started generating a large number of error logs.

Proposed in #17249 (comment) with agreement between maintainers.

[nitishfy]

Hi, I'd like to take this up!

[serathius]

Thanks, feel free to reach out to me on slack (get an invite) if you need any help
/assign @nitishfy

[serathius]

#17423 covered a single node cluster, we should also cover:

• 3 node cluster
• different TLS configurations.

[nitishfy]

#17423 covered a single node cluster, we should also cover:

• 3 node cluster
• different TLS configurations.

we can raise a new PR for it. WDYT?

[serathius]

SG, thanks

[serathius]

Also, when we are done, we should consider backporting the test.

[serathius]

Still remaining to be completed:

• Extend the existing test to also validate 3 node cluster.
• Backport the test to release-3.5 and release-3.4 branches to prevent regressions there.

[ghouscht]

I think I can take this.

/assign

[ghouscht]

done 🙂 I think this can be closed now.

• test extension: chore(e2e): extend logging test to 3 node cluster #18806
• 3.4 backport: [3.4] chore(e2e): backport TestNoErrorLogsDuringNormalOperations test #18809
• 3.5 backport: [3.5] chore(e2e): backport TestNoErrorLogsDuringNormalOperations test #18807

[serathius]

Awesome, thanks @ghouscht !

[ahrtr]

It looks like the test doesn't enable TLS by default. Should we also extend the test to support the case of enabling TLS?

[serathius]

Good point, I missed #17329 (comment).

[ghouscht]

Oh I missed that as well, sorry! I'll have a look later on and will prepare another PR.

[ghouscht]

Something like this #18819? Or did you have something different in your mind?