Add search and sort capabilities to dnf-json

[bcl]

In osbuild/cockpit-composer#1462 @teg mentioned that it would be a good idea to implement searching and pagination in dnf-json instead of the current behavior which has it dumping all packages into json and processing that in osbuild-composer. This is to help speed up the responsiveness of the cockpit-composer ui, but composer-cli cmdline use will also see the same benefit.

My current 'plan' is to add a search command to dnf-json which ties into the dnf query parameters (https://dnf.readthedocs.io/en/latest/api_queries.html) and then add the ability to use them to the weldr api. I'll work with @jkozol to figure out exactly what the UI needs from the API.

[bcl]

In addition to adding the search command to dnf-json I've been experimenting with ways to speed up the listing of all packages. See pr #2918 for what I've come up with as the 'best' solution so far.

Currently, on my 4G/2core VM running Fedora 35, it takes around 5-6s to list all packages with /api/v0/modules/list/**. I experimented with changes to dnf-json and was able to shave off maybe 2s from that by limiting it to the latest NEVRA and by filtering on x86_64 + noarch packages. The timing there is fuzzy though, dnf isn't very consistent in how long it takes to return results. The other problem with this is that it changes the package list so that it doesn't include other builds or i686 versions of the packages.

So I added a 60s cache to the dnf-json results in osbuild-composer. This works really well in my testing, with results being returned in under 1s at the API level. The UI takes longer, either due to my browser or some other factors that I cannot measure. The result is that the UI feels much more responsive while you are using it. And yes, you will get a slow response every 60s when it refreshes the results, but every other page transition is fast so I think the improvement is worth it.

[teg]

@jkozol do we still list all packages in the new UI, or is it all search based now?

If we do list them, the additional speedup possibility I had in mind was pagination all the way from dnf-json. I think we would need to change the return structs a bit to collect all versions of a package together in one object and paginate at that level.

Something like: give me package 10 to 20, each version of each package.

Not sure how much is gained by this given the other optimisations, but my thinking was what we would never pass tens of thousands of entries from dnf-json to composer.

[teg]

Another idea, I don't know why we never did this:

Could we add a call to just download metadata for a repo? On start (or repo add), we could download all metadata in the background so that when someone starts using composer the metadata has most likely been downloaded. We still need to deal with expiration, but it would become the exception rather than the rule.

[bcl]

I don't think it's possible to speed things up by paging in dnf-json, the dnf API does't have any way to return a subset of the results so you end up needing to get all of them. In my experiments I tried doing things like counting total number of packages and returning a range of them and it ran at the same speed, or not enough difference to notice. Currently we combine multiple builds for the same package into a different format in osbuild-composer so this would have to be moved down to dnf-json (or remove returning multiple versions) -- otherwise dnf-json's idea of a page and the UI idea of a page will be different. The same applies for arches, on x86_64 it is currently also returning i686 results on Fedora.

Funny you should ask about downloading metadata at startup :) I had that on my list to look at today. lorax-composer used to do a depsolve of all the image types with an empty blueprint. This was to check to make sure they could all be used, and to pre-load the metadata cache since that delays the first request by a couple minutes on my setup.

I don't think we need to do much more than a simple depsolve call to dnf-json for each supported distro (including user added sources). That will load the metadata cache and then the 20s timeout you added will apply to all future requests and the current cache size limits will be applied.

[teg]

Thanks for the explanation, that all makes sense to me!