Implement a Fuzzy CI to catch ocamlmerlin regressions

[pitag-ha]

This PR implements a Fuzzy CI, that catches end-to-end regressions in Merlin by checking the responses of a fixed set of ocamlmerlin queries. That set of queries is determined by a deterministic but randomly chosen set of file location samples on the Irmin code base. The CI passes automatically if the responses on all samples remain the same before and after the PR. If there's a diff in the responses, the CI fails and makes the diff available as a GH action artifact. There's a high-level description of the CI in much more detail on a new wiki page about the CI.

Motivation for the CI

Merlin already has quite a few end-to-end tests (which are also run in CI). All of those are behavior-specific: We want to test one concrete behavior and write a tailored test for it. Additionally to those end-to-end tests, we now also want to test non-specifically on a large randomized input set.

Some implementation details

There are two reasons why the data of the ocamlmerlin responses is generated in the CI instead of living in the repo: 1. The data is over 20 MB big. 2. The generation of the data takes over 10 minutes, and we don't want our contributors to have to re-generate it for PRs that change the ocamlmerlin responses.

With the current implementation, the CI takes about 14-15 min to run. That's about 2 min longer than the mac-os CI (edit: Actually nvm, the mac-os CI is only one job of the main.yml workflow, which in total takes over 20 min, so clearly longer than this workflow! :)), which is the currently longest CI. There'd be a straightforward way to reduce the running time of this CI by another ~30 seconds. Let me know if I should do that.

I also think that we should observe over the coming months if a given ocamlmerlin response change will always be duplicated among multiple samples. If that's the case, we can reduce the sample size, which would speed up the CI.

We can also discuss whether we want to reduce the number of query types we test. Currently, it's type-enclosing, occurrences,locate, complete-prefix, and errors. I think all of them are useful to test. What do you think?

[pitag-ha]

Btw, the CI is expected to fail on this PR: The first commit changes the ocamlmerlin behavior by adding a query_num value to its response. That's necessary for the simplified/"distilled" diff generated by the CI (there's more info about the distilled diff on the new wiki page). Let me know if you'd like me to factor out the query_num addition into a separate PR, e.g. to discuss whether the query_num should always form part of the response or only on an opt-in basis via a CLI arg.

[github-actions]

This PR changes the response of some of the ocamlmerlin queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analysis of this PR has been approved in its following state:

• URL to download the generated data sets and their diffs between PR base branch and merge branch (at the moment of approval): https://github.com/ocaml/merlin/actions/runs/7281268850
• 256-sha of full responses diff: ad6b69213dc31fe5be9d6a9b48138daff64c8a0eaf3f47ecdfb0c8e792d6d352

[voodoos]

We can also discuss whether we want to reduce the number of query types we test. Currently, it's type-enclosing, occurrences,locate, complete-prefix, and errors. I think all of them are useful to test. What do you think?

I think that's a good starting point.

I made a review pass on the PR. My main worry is future maintenance: the actions are quite complex. Do you know what, in day-to-day Merlin work might require changes to the CI ? I guess upgrading to a new version of OCaml is one, but are there other things we should take in consideration?

[voodoos]

Maybe we could take advantage of the monorepo to also run on more diverse projects and not only Irmin ?

[pitag-ha]

Yes, adding more diverse projects to the workflow should be relatively easy (I'd then decrease the number of samples per file, though, to avoid having the CI run for too long). Do you have projects in mind?

[voodoos]

I would target something that is part of Irmin's dependencies but didn't have a look yet (and probably won't before next year...).

[pitag-ha]

We'd need to have a look if anything in the Irmin dependency cone adds additional value to the test base. As mentioned in the comment about the CI execution time, it's better for the execution time to have few files with lots of samples than to have lots of files with few samples.

(and probably won't before next year...).

(same for me :) )

[pitag-ha]

My main worry is future maintenance: the actions are quite complex. Do you know what, in day-to-day Merlin work might require changes to the CI ?

I'd say that there are the following main areas of attention in "day-to-day" Merlin work:

1. Whenever a non-deterministic field is added to the ocamlmerlin responses, those fields need to get filtered by the CI. That filtering already needs to be done for the merlin end-to-end tests, where it's done in tests/merlin-wrapper. If we merge this CI, it will also need to be done for the CI in its dedicated step for it.
2. Upgrading to a new version of OCaml will require several adaptations. Among others, it requires re-generating Irmin's lock file. I've decided to use opam-monorepo to generate the lock file rather than opam lock for several reasons, such as it supports depexts, and a universe is a good base for caching. We can talk about the point of adapting the CI when upgrading OCaml more, if you want to, since it will be the most significant point of the CI's maintenance, and it's currently not ideally supported by the CI.
3. For proactive maintenance, the CI can be improved in the future by adding support for new queries / queries that become more important. Also, the sample set could possibly be improved. That will be easier once we have a good overview of what the CI is capable of with the current sample size and current sample code base choice, and in which cases it fails to catch a regression.

Like every CI, there are also a few Merlin-external factors that might need maintenance:

1. The workflow uses a couple of external actions. If external actions get out of date, they'll need to be upgraded. I've made sure to only use official external actions from ocaml and from actions: ocaml/setup-ocaml, actions/checkout, actions/cache, actions/download-artifact and actions/upload-artifact.
2. A few things are done manually, e.g. some manual GH API calls. In the case of GH API breaking changes, the CI would need to adapt.
3. There's a few clumsy details, e.g.: When reading the comments on the PR to find and parse the comment containing information about the last approved diff state, only the last page with 100 comments is read. So if there were ever more than 99 comments after the approval comment, the CI wouldn't work. I can fix that, if it happens. Or I can fix it rn if you prefer. (not the case anymore)
4. Some things rely on current GH actions primitives: I share a binary between different runners, assuming that all runners are hosted on x86 machines (currently true); I've split the workflow for permission-handling (currently recommended and possible).

[pitag-ha]

I've just updated the comment above, crossing out one maintenance point (after implementing a fix). I think maintenance of this CI would be quite ok, but do let me know if you think differently! The main maintenance burden (I hope) will be updating the CI when Merlin bumps the compiler. As you've mentioned, that happens only twice a year. I'm also very happy to update the CI myself those two times per year. What will need to be done will be to edit an env variable in the CI and to generate a lock file for the test code base Irmin.

If you think similarly as me about maintenance, this might be ready to be merged.

[voodoos]

Thanks a lot Sonja !

Let's merge it and see how it goes :-)

[voodoos]

Could rebase and squash commits into relevant steps ?

[pitag-ha]

Cool, thanks! I've just squashed the commits.

As a heads-up: I'm planning to transfer the merl-an repo, which this CI uses, from my GH account to the Tarides GH org this week. If you want, we can wait for that to happen before you merge. Otherwise, you can merge now and I open a one-line PR to adapt the CI once merl-an is transferred.

[voodoos]

As a heads-up: I'm planning to transfer the merl-an repo, which this CI uses, from my GH account to the Tarides GH org this week. If you want, we can wait for that to happen before you merge.

If your confident this will happen during the next few weeks then let's wait for it 🙂

[pitag-ha]

In the end, I'm not prioritizing transferring merl-an at all. If that sounds good to you, @voodoos , it might be best to merge this PR now, and once I transfer merl-an, I open a 1-liner PR. Wdyt?