Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When running Argo Workflows we occasionally see that our workflows report surprisingly high CPU and Memory resource durations.

We have traced this back to an issue with Containerd which will sometimes fails a Pod and for some reason sets the startedAt date to "the epoch" (ie: 1970-01-01T00:00:00Z).

...
"initContainerStatuses": [
{
  "containerID": "containerd://b55ff9eb8687235ae0fecd4216522b75b6f7a0fab5eca18de8c06365bd2636fc",
  "image": "quay.io/argoproject/argoexec:v3.5.11",
  "imageID": "quay.io/argoproject/argoexec@sha256:bb3938480cbe7a7c0f053eb77a5d3edb22c868bf57407bf066fd105961f26c72",
  "lastState": {},
  "name": "init",
  "ready": false,
  "restartCount": 0,
  "started": false,
  "state": {
	"terminated": {
	  "containerID": "containerd://b55ff9eb8687235ae0fecd4216522b75b6f7a0fab5eca18de8c06365bd2636fc",
	  "exitCode": 128,
	  "finishedAt": "2024-09-13T13:28:03Z",
	  "message": "failed to create containerd task: failed to create shim task: context deadline exceeded: unknown",
	  "reason": "StartError",
	  "startedAt": "1970-01-01T00:00:00Z" <---- here
	}
  }
}
],
...

This results in the incorrect calculation of a duration due to the function below which assumes that a startedAt data of 1970 is expected.

I appreciate this feels like more of a containerd bug that an argo-workflows bug, however I think it is worth adding some logic to handle this case more gracefully.

For additional context, we are running on top of managed GKE when we see this bug. I am not attaching a reproducible workflow because... well... this bug is challenging to reproduce!

I will propose a fix very soon and will let the maintainers decide whether it is worth patching. From our perspective it's worth it because we use the Argo Workflows reported CPU/Memory usage to bill downstream customers... 😨

Version(s)

v3.5.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

N/A

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

N/A