Implementing a timeout on the runners for long-running pipelines

[daniddelrio]

In our organization, we have issues on some workflows intermittently and unintentionally running for a few hours. Since there's no option of globally setting timeouts on the workflows (see Discussion), we were wondering if there's a way to implement this timeout on the runner-level or in ARC.

Our first idea was to:

1. Using a pre-start hook, annotate our runners with the start time of the job
2. Create a cronjob that constantly checks for this running job and for the time elapsed since the annotated starting time. If it surpasses a certain amount of time, then we kill the runner

However, maybe there's an easier way that's already implemented natively? Something like a TTL controller or an existing process that can handle this.

I see that there was an issue #2360 posted about this, but it didn't get any traction