Add tests for analysis using the Clang and GCC analyzers

[lzaoral]

No description provided.

[lzaoral]

This PR is blocked since it requires a newer release of csdiff and Testing Farm tries to install all packages from the given COPR build:

Error: 
 Problem: conflicting requests
  - nothing provides csdiff > 2.8.0 needed by csmock-plugin-snyk-3.3.5.20230206.145923.g5d0b606.pr_95-1.fc38.noarch

[kdudka]

@lzaoral Could we please configure csmock CI to use csdiff from the Packit-managed COPR?

[kdudka]

@lzaoral Could we please configure csmock CI to use csdiff from the Packit-managed COPR?

[kdudka]

Another option would be not to install csmock-plugin-snyk when we only need csmock-plugin-clang.

csmock-plugin-clang depends on csmock-common, which installs everything what is needed. There is no need to explicitly install the csmock metapackage, which pulls in all the plug-ins.

[lzaoral]

Hmm, it looks like Testing Farm does that regardless of the contents in the given fmf files.

[lzaoral]

Looks like this is not possible at the moment: https://packit.dev/docs/testing-farm/#running-tests-with-builds-from-another-pull-request

[kdudka]

@lzaoral We do not need to use builds from another pull request. I meant to use csdiff from https://copr.fedorainfracloud.org/coprs/g/codescan/csutils/ which corresponds to the main branch of csdiff upstream. Anyway, in case it helped, I can roll out a new release of csdiff today.

[lzaoral]

Yes, I know. But this seems to be the only way to install the contents of the csdiff and csmock COPR repositories at the same time. Other methods like 3330b2a install them too late...

[kdudka]

I see. csdiff-2.9.0 is being built...

[lzaoral]

/packit retest-failed

[thrix]

hmm, we should somehow disable those unsupported combos on Testing Farm ... it is hitting our error rate :)

[TomasTomecek]

Folks, could you please drop the testing targets for architectures that are not available in the Testing Farm? https://docs.testing-farm.io/general/0.1/test-environment.html#_supported_architectures

So for tests, please keep only x86_64 and aarch64

We are working in Packit and Testing Farm to account for this, but right now all requests are sent to TF as Miro points out.

[lzaoral]

Sorry @thrix and @TomasTomecek! I've deleted everything except x86_64 as that's what we are mainly interested in anyway.

[lzaoral]

The latest release of Testing Farm should support fix the unjustifiable dependence on a newer version of csdiff.

[lzaoral]

I've dropped the integration with TF completely for now. It can be added in the future and the tests themselves can be merged even in the current state.

[kdudka]

So the tests we are about to merge will not run automatically at all?

Then we should at least mention in README how to run the tests manually. If nobody knows about the tests, they will break sooner or later.

[lzaoral]

Hmm, everything except CS8 passed but the table below still shows all Fedora tests as running. 🤔

[lzaoral]

Current state: The @csutils copr repository is properly installed, however, the packages are not properly updated on stable Fedora releases for an unknown reason.

[lzaoral]

@kdudka, the PR should be ready to be reviewed. The test is modular, therefore, if you want to test a new tool, you only need to create corresponding .fmf file and set proper environment variables. Also, the tar.xz archive with results should be provided by the Testing Farm for each test (open the log dir link for given test and them select the data directory).

edit: add instructions to get the logs

[kdudka]

Nice work. Thank you!