Is it possible to save events from trigger errors for later replay?

[dradetsky]

I can't tell if this is possible or not. I think it should be, since at least for some workflow cases it can be very important, but it's not totally obvious.

Scenario: I have a Sensor which triggers some process (such as an argo-workflow) which might persistently fail, either because I made a programming error or because the process depends on some external service which might be down. In this case, I don't want a simple retry-strategy, as I expect this just means it will fail 3 times in a row rather than just once. Instead, I would want all the events which triggered failures to be saved somewhere, presumably along with the error messages which they triggered. Then, once I believe I have solved the issue which caused the failures, I want to replay the events back into the EventBus so they are redelivered to the Sensor as normal, only this time they presumably will succeed.

I can't tell for sure if argo-events currently supports this. It certainly doesn't support it in a way which makes it easy and obvious for me to enable, so if it is currently supported, there should be documentation which explains how to do it.

If argo-events does not currently support it, is there some reason why this is considered out-of-scope for the project? I can see that for a lot of use-cases, it would be a needless complication. For example, the most common thing I've used it for in the past is CI workflows: a developer pushes a change to his PR branch, which triggers a webhook post from github, which triggers a CI build. In that case, I wouldn't want the developer to just poke the PR again once the issue was fixed.

However, for other use-cases, I think this is really important. And lots of other event services seem to support it.

If the reason argo-events doesn't support this is just because nobody has had time to implement it, all the developers have a ton of work, listen kid you try running a big open source project some time, etc etc, I think that's totally fine & I understand & make no judgment. I just want this to be made absolutely clear. Maybe I'll even try to implement it myself.

[dradetsky]

Just looking at the code, it looks like it would be necessary to make the actionFunc passed to subscribe return an error. Which means (I think) we would have to have SensorContext.triggerActions read the results of triggerWithRateLimit via a channel. I'm not sure what implications this would have, if any.