[bug] Dependency chains are not respected in moon ci

[kikones34]

Describe the bug

If you have a dependency chain task1 -> task2 -> task3, when task1 is touched, moon ci will only execute task2 and it will not propagate to task3.

Steps to reproduce

Configure three tasks as follows:

  dependency:
    command: echo "I'm the dependency"
    inputs:
      - dependency-input.txt
    options:
      runInCI: true

  dependency-checker:
    deps:
      - dependency
    command: echo "I'm the dependency checker"
    inputs:
      - dependency-checker-input.txt
    options:
      runInCI: true

  dependent:
    deps:
      - dependency-checker
    command: echo "I'm the dependent"
    inputs:
      - dependent-input.txt
    options:
      runInCI: true

Commit everything to the default git branch. moon ci shouldn't execute anything.
Now, modify the file dependency-input.txt and execute moon ci. The dependency-checker task is run, but the dependent is not:

▪▪▪▪ Gathering touched files
  services/trio-cloud-search/dependency-input.txt
▪▪▪▪ Gathering potential targets
  ...
  trio-cloud-search:dependency
  trio-cloud-search:dependency-checker
  trio-cloud-search:dependent
  ...
▪▪▪▪ Generating action graph
Target count: 335
Action count: 12
▪▪▪▪ Running targets
▪▪▪▪ trio-cloud-search:dependency (faf9f199)
        trio-cloud-search:dependency | I'm the dependency
▪▪▪▪ trio-cloud-search:dependency (faf9f199)
▪▪▪▪ trio-cloud-search:dependency-checker (7f147af2)
trio-cloud-search:dependency-checker | I'm the dependency checker
▪▪▪▪ trio-cloud-search:dependency-checker (7f147af2)

Expected behavior

The task dependent should also be executed since it depends on dependency-checker.

Additional context

Note that these dependency chains behave correctly regarding the cache logic (so if dependency is dirty, it will propagate the dirty state up to dependent). The issue only occurs in moon ci.

[milesj]

@kikones34 This is by design at the moment. The "run dependents" logic only applies to direct dependents of affected tasks, and not the entire tree. It was built this way as a simple regression testing mechanism and ensuring a change in a project didn't affect it's immediate consumers.

However, we can probably add an option to control the depth.

[kikones34]

@milesj Oh, I see. The problem is that this breaks the way we're managing inter-project dependencies. In the real-world use case, what we actually have is:

• A check-source-changed task on each project which gets dirty when the source code of the project changes.
• A check-deps-source-changed task on each project, which has dependencies such as proj1:check-source-changed, proj2:check-source-changed, etc. It gets dirty whenever one of those project's source is changed.
• Tasks that depend on check-deps-source-changed. For example, test only gets dirty if the project source or the source of the dependencies has changed.

This has worked perfectly as far as caching goes, but I'm finding it impossible to replicate with the CI feature.

[milesj]

@kikones34 Yah that makes sense. Are you able to use moon run in CI now to replicate what you want?

[kikones34]

Yeah for now we're using moon run, but managing the cache is tricky due to the pipelines being run by Bamboo agents, different PRs open on different branches at the same time, etc.
For now it's mostly working, although we are limiting the cache to be per-branch, otherwise many issues arise. That's why we were looking into leveraging the moon ci capabilities so that we can always run just the minimum necessary tasks given by the touched files, without having to worry so much about the cache state.

[milesj]

I'll look into supporting deep dependents in the next release.

[milesj]

I've made some changes in v1.29, can you give that a shot?

[kikones34]

I've been checking out the new tracker logs and CLI options upstream and downstream, but I think there's still no way to pass these to moon ci, right? It only seems available for moon query projects. I also cannot really test if it would behave as expected for this issue's example, because the options can't be passed to moon query tasks either, and I get no information regarding affected tasks using the projects query.
In any case, the introduction of these parameters looks promising, and the tracker logs definitely will help debug stuff moving forwards!

[milesj]

Yeah moon ci is hard-coded to how it works and probably won't change. I'd rather add this functionality to moon run.

[dudicoco]

@milesj can you please elaborate on why this is the intended behavior of moon ci?
I believe most use cases require downstream dependents to run when a task changes, otherwise you have to rely solely on task inputs in order to make them affected by such a change.

We could put the outputs of a dependency as an inputs for the task like so:

# app1
tasks
  build:
    command: build
    inputs:
    - 'src/**/*`
	outputs:
	- bin/main

# app2
tasks
  build:
    deps:
    - app1:build
    command: build
    inputs:
    - 'src/**/*`
	- /app1/bin/main

# app3
tasks
  build:
    deps:
    - app2:build
    command: build
    inputs:
    - 'src/**/*`
	- /app2/bin/main

However, moon generates the graph before the outputs are generated so this doesn't work.

So it seems that at the current state moon ci is actually not suitable to run in CI since a lot of dependent tasks will not be triggered.