Unification: Package search result pagination

[lucasgarfield]

We would like to eliminate pagination from the package search feature. This would allow us to perform our own (better) sorting logic similar to what is done in Image Builder Front End.

Currently, searching for packages is done by calling the Weldr API, which supports* pagination. *In a sense it requires pagination. On my machine, requests for > ~1000 packages fail.

Therefore, changes to the Weldr API are probably necessary to implement this effectively as fetching the full package list would require a very large number of API calls (consider that Fedora has ~35k packages).

[teg]

This is related to a comment I made on your PR: I think we should consider doing the searching and pagination in dnf-json. Then we have the logic in one place and pass up the minimal amount of data (I didn't measure, but dumping the whole package database over json for every search as we do now in composer is probably sub optimal).

[bcl]

Therefore, changes to the Weldr API are probably necessary to implement this effectively as fetching the full package list would require a very large number of API calls (consider that Fedora has ~35k packages).

FWIW it only requires two, but yes it's a giant amount of data. eg. If you look at the output from composer-cli --json packages list it does an initial request with the limit set to 0 to get the total number of projects:

{
    "method": "GET",
    "path": "/projects/list?limit=0",
    "status": 200,
    "body": {
        "limit": 0,
        "offset": 0,
        "projects": [],
        "total": 6829
    }
}

And then requests all of them in one query with limit set to the total count:

{
    "method": "GET",
    "path": "/projects/list?limit=6829",
    "status": 200,
    "body": {
        "limit": 6829,
        "offset": 0,
        "projects": [
            {

...

It would be much more efficient to implement the searching on the server side like Tom suggests.

[jkozol]

@lucasgarfield Any additional information about this issue?