Allow to run e2e tests on parallel

[wainersm]

Describe the bug

I never tried to run the e2e tests in parallel but @mkulke tried and there are some problems. In many occasions we have talked about it and proposed some change (for example, per-tests namespaces) however we don't have yet a decent TODO list to enable e2e in parallel.

Let's use this to discuss and come up with list of action items.

How to reproduce

N/A

CoCo version information

N/A

What TEE are you seeing the problem on

None

Failing command and relevant log output

No response

[mkulke]

namespace-per-test would be one solution, but that would make debugging very cumbersome when you run them locally, also there would be a lot to clean up, I wouldn't recommend it.

If the tests use the same resource names in the same namespaces, you have to be very diligent with cleanup. it's easy to introduce race conditions even if you run tests sequentally, e.g. when you cleanup in a test for a meta-resource like a deployment or a job, you have to wait for the underlying pod to be deleted by k8s, because the pod might carry a label that the next test is relying on.

I think the best solution is to introduce a naming helper that will produce names with a unique suffix (name := gen_name("nginx")) and convert all places that use hardcoded resource names to use it. if someone adds a test, they will hopefully follow the same pattern.