Document procedure for developing new DSS tests

[BenjaminPelletier]

          @BenjaminPelletier, following our discussion at the last community meeting about maintaining and updating the qualifier configuration used in the CI for the DSS repo, should we maybe change the way we release scenarios that 'break' the current DSS implementation?

So far we have been doing this:

1. Add scenario (or fix existing one) that reveals an issue with the DSS
2. Fix the issue on the DSS side, manually test the PR against the new scenario
3. Release new DSS version
4. update DSS version on the monitoring repo, obtain a green CI
5. merge new scenario
6. release monitoring image
7. update monitoring image on the DSS repo
8. update CI qualifier configuration on the DSS repo

This works, but is a bit ad-hoc and requires trusting the manual step in 2) above. The main reason we do this is that we usually directly update the qualifier configurations when we add a scenario, which means CI runs will necessarily break if the DSS has an open issue.

An alternative process could be:

1. Add scenario that reveals issue with the DSS, but do not update the default configurations yet
2. Open PR, get it reviewed and merged
3. Release new version of the monitoring image
4. Within a PR on the DSS:

• Fix the issue on the DSS,
• update the qualifier config used in the DSS CI to use the new scenario,
• get a green CI that confirms the issue is fixed and merge

5. release the DSS image
6. Open a PR on the monitoring repo that:

• updates to the latest released DSS
• updates the configurations to test the newly fixed features

The second process is technically cleaner, albeit a little bit more involved, and introduces the risk to forget to add a relevant configuration to our defaults in step 6). On the other hand, it gets us closer to properly using the qualifier and forces us to go through the gymnastics of properly updating configurations on the DSS side. (Under the current process, we risk forgetting updating the monitoring image or the qualifier configuration on the DSS repo, which is guaranteed to be impossible with the alternative process)

I will try the second approach for the set of scenarios that check that OVNs are being verified to see if it is even practical: for entirely new scenarios it should be easy, but for updates to existing scenarios it will be less so.

Note that there will still be situations that require an ad-hoc approach.

Originally posted by @Shastick in interuss/dss#1083 (comment)