[BREAKING CHANGE] Use Quarkus Maven plugin for JVM mode tests and isolate tested Quarkus application, simplify creating of reproducers

[michalvavrik]

Summary

TL;DR;

• if you let QE FW build native executable, the framework prepares for you standalone application with all the changes (forced dependencies, build-time properties)
  • than you can use same command to start application as you saw in logs (so you know exactly how to reproduce the failure)
  • please remember that we have Quarkus Maven plugin present in our projects, so when custom build is not required naturally (e.g. forced dependencies, custom build-time properties in native mode or Quarkus build is explicitly skipped with quarkus.build.skip=true), you need to enforce it, e.g. set ts.global.custom-build.required to use this
• use Quarkus Maven plugin to test JVM mode, because that is what customers do, they don't use Quarkus bootstrap directly

Practical example:

• https://github.com/quarkus-qe/quarkus-test-framework/blob/main/examples/blocking-reactive-model/src/test/java/io/quarkus/qe/GreetingResourceIT.java has forced dependencies
• before this PR for JVM mode, we created augmented model in memory with Quarkus bootstrap and run it
• after this PR, you have actual application with augmented app properties (added build-time props only, runtime props are still passes as -D)
• most importantly, checkout parent POM, there is none

cd examples/blocking-reactive-model
mvn clean verify -Dts.global.delete.folder.on.exit=false
## Quarkus REST
cd target/GreetingResourceIT/rest/mvn-build/
# now you are in actual tested Quarkus REST application, now you can run it with
# if you set runtime properties, you better check log for what QE FW set as system properties
java -jar target/quarkus-app/quarkus-run.jar
## RESTEasy Classic
cd target/GreetingResourceIT/resteasyClassic/mvn-build/
# now you are in actual tested RESTEasy Classic application, now you can run it with
java -jar target/quarkus-app/quarkus-run.jar

Longer version:

Before this PR, we build JVM move tests without forced dependencies directly with Quarkus boostrap. However:

• this is not what users do
• we are not certain we use the bootstrap in a samw way as Quarkus does
• we don't have actual application that we can run separately from our framework as one of following could happen:
  • application properties were augmented
  • dependencies were added
  • only some classes were tested, not all the application classes on path

So this PR should make creating reproducers easier, because you will have independent application (no parent, no test framework):

• app that you can copy without adjustments
• command used to run application including all system properties set as -Dproperty-key=property-value

Cons:

• file R/W operations and CLI commands are slower than running everthing in one Java process
• merging profiles and plugins and dependency managements between parent and tested module is complex; IMO it can only be done properly with effective POM, which would slower tests down; so I only tested what we actually do ATM and that is what is supported and nothing more

Breaking changes:

• you need to add io.quarkus.qe:quarkus-test-preparer to your project and run the prepare-pom-mojo goal

Other changes this PR contains:

• replaced busy waiting in LogHandler with a Timer
  • maybe it will also help with the Possible data race during deletion of logs #612
• @Mojo annotation cannot be shorter and formatter drops any line break I add, while checkstyle plugin keeps complaining about line length, so I added checkstyle suppression rule
  • the suppression rule can be removed when Let annotations span several lines #644 is implemented
• makes sure that Maven settings are propageted, therefore implements for baremetal and native Custom settings file is not reflected in the tests #943

What this PR doesn't do:

• I leave OpenShift as is, current way how we build OpenShift can be really cumbersome when it comes to determine when is custom artifact build required, I'd need much more than few days to get this right.

Please check the relevant options

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Dependency update
• Refactoring
• Release (follows conventions described in the RELEASE.md)
• Breaking change (fix or feature that would cause existing functionality to change)
• This change requires a documentation update
• This change requires execution against OCP (use run tests phrase in comment)

Checklist:

• Example scenarios has been updated / added
• Methods and classes used in PR scenarios are meaningful
• Commits are well encapsulated and follow the best practices

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

There is no chance I'll be able to finish this, I just don't have time. I'll reduce this PR to what works well and improves current state - use Maven plugin helper for all baremetal runs.

[michalvavrik]

run tests

[michalvavrik]

run tests

[michalvavrik]

run tests

[mjurc]

This works nicely and I cannot really find anything wrong about the PR itself.

Thanks for the enhancement.

[mjurc]

About the

merging profiles and plugins and dependency managements between parent and tested module is complex; IMO it can only be done properly with effective POM, which would slower tests down; so I only tested what we actually do ATM and that is what is supported and nothing more

I think this would require a deeper change in framework (e.g. making applications more isolated) cos I don't really think making workarounds by figuring out the effective pom and the excluding the framework deps gives us much value

[michalvavrik]

Thanks @mjurc , I'll rerun TS with this FW version (both baremetal and OCP) to re-check there are no issues before I make release.

[michalvavrik]

Added note to the Wiki https://github.com/quarkus-qe/quarkus-test-framework/wiki/1.-Getting-Started#standalone-reproducer.

[michalvavrik]

As there is a release in progress, I run daily build of this FW project and it is green https://github.com/quarkus-qe/quarkus-test-framework/actions/runs/11934859467 so that means K8 and Windows native are fine as well.

[michalvavrik]

For record TS daily build with bumped FW was green quarkus-qe/quarkus-test-suite#2208 and OCP looked good as well.