Propagate correlationId label also to the initial flow that triggered the subflow

[anna-geller]

Feature description

Let's say we have 3 flows, parent called flow1 and two subflows flow2 and flow3 triggered in a chain:

flow1 --> flow2 --> flow3

Currently, the executionId of flow1 will be propagated as system_correlationId label to flow2 and flow3 but I believe flow1 should also get the same label to clearly indicate all flows in the chain incl. the initial one

To reproduce:

id: flow1
namespace: company.team
tasks:
  - id: subflow
    type: io.kestra.plugin.core.flow.Subflow
    namespace: company.team
    flowId: flow2

id: flow2
namespace: company.team
tasks:
  - id: subflow
    type: io.kestra.plugin.core.flow.Subflow
    namespace: company.team
    flowId: flow3

id: flow3
namespace: company.team
tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: hello from the last flow

[loicmathieu]

This is not that easy as when we propagate the correlationId to the first subflow, we cannot update the parent execution.
We can add the correlationId to all execution but it would just be a duplicate of the executionId.

This needs investigation on how to do this the best.

[anna-geller]

interesting, I thought we could add this label (as below but automatically) to all executions that have at least 1 subflow task and don't have any upstream parent execution ID?

id: flow1
namespace: company.team

labels:
  system_correlationId: "{{ flow.executionId }}"

tasks:
  - id: subflow
    type: io.kestra.plugin.core.flow.Subflow
    namespace: company.team
    flowId: flow2

We can add the correlationId to all execution but it would just be a duplicate of the executionId.

This would run the risk that the subflow execution would get its own correlationId (equal to is own executionId) overriding the one it got from parent right?

[loicmathieu]

It may be a way to do it, a little brittle

[anna-geller]

Ludo suggests to explore the option that if there is no correlationId (e.g. no such ID passed from a parent execution), we should add correlationId equal to the current executionId

[loicmathieu]

It's easy if we add the label to all executions but this would clutter all executions with a useless label.

But as I said, the challenge is that at the point we will execute a subflow, we cannot set this label to the current execution.

[anna-geller]

feel free to discuss this with Ludo, no strong opinions, to me the most important part is that we add it to the first parent flow in the chain 👍

[loicmathieu]

Decision has been to add the system_correlationId field on all executions that didn't have one.