HDDS-10468. CI: Make unit depend on build

[smengcl]

What changes were proposed in this pull request?

To avoid unit from using old cached builds in PR CIs, hopefully.
At the cost of lengthening the CI by ~8 minutes.

For the background, see #6286 (comment)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10468

How was this patch tested?

• CI dependency tree

[adoroszlai]

a race can unintentionally cause the unit job to use an old cached ozone build to run the actual test.

Thanks @smengcl for finding this issue. However, Ozone build should not be cached at all.

build job caches Maven repository with dependencies but not Ozone.

It also stores via upload action:

• Ozone binary tarball (for acceptance tests)
• Ozone source tarball (for compile check)
• Maven repo with only Ozone jars (for integration tests)

unit job uses Maven repo (with only dependencies) from cache, but compiles Ozone itself.

integration job uses both Maven repos (dependencies via cache, Ozone via artifact download).

It turns out the cached repo with dependencies also contains Ozone, probably due to cache path exclusion being broken: actions/toolkit#713

I think we'll need to restore explicit deletion of Ozone jars at the end of build:

      - name: Delete temporary build artifacts before caching
        run: |
          rm -fr ~/.m2/repository/org/apache/ozone
        if: always()

[adoroszlai]

POC: adoroszlai@aecdde0

[smengcl]

Thanks @adoroszlai for the comment.

Thanks @smengcl for finding this issue. However, Ozone build should not be cached at all.

build job caches Maven repository with dependencies but not Ozone.

Aha! That makes sense.

unit job uses Maven repo (with only dependencies) from cache, but compiles Ozone itself.

integration job uses both Maven repos (dependencies via cache, Ozone via artifact download).

It turns out the cached repo with dependencies also contains Ozone, probably due to cache path exclusion being broken: actions/toolkit#713

I think we'll need to restore explicit deletion of Ozone jars at the end of build:

      - name: Delete temporary build artifacts before caching
        run: |
          rm -fr ~/.m2/repository/org/apache/ozone
        if: always()

That sounds good. Would you like to open a new PR/jira?

[adoroszlai]

Would you like to open a new PR/jira?

OK, I'll post a PR.

BTW, I realized this affects unit check only if you restrict the build to specific modules by using -pl (e.g. -pl :ozone-client). In this case dependencies within Ozone (e.g. hdds-common) for that submodule are taken from Maven repo. Normally unit tests all Ozone modules, compiling everything from scratch.

[smengcl]

Would you like to open a new PR/jira?

OK, I'll post a PR.

BTW, I realized this affects unit check only if you restrict the build to specific modules by using -pl (e.g. -pl :ozone-client). In this case dependencies within Ozone (e.g. hdds-common) for that submodule are taken from Maven repo. Normally unit tests all Ozone modules, compiling everything from scratch.

You are right. I was not fully aware of the implication of -pl when I was using it. It obviously skips all other modules that would be compiled. Let me close this one. Thanks!