[Bug][GitLab] GitLab Pagination not working again

[kkc-tonywu]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The number of GitLab Repo MRs、tag、issue、deployment... in our system exceeds 10,000.

According to the official GitLab documentation:

For performance reasons, if a query returns more than 10,000 records, GitLab excludes some headers.
ref: GitLab Documentation

As a result, the x-total-page header is not returned, which causes Develake to only fetch 100 records for the subtask.

What do you expect to happen

No matter the settings, it is necessary to retrieve the complete data.

How to reproduce

When MR, tag, issue, and Deployment are executed tens of thousands of times, attempting to retrieve the data afterwards will only return 100 records.

Anything else

No response

Version

1.0.1

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello, @kkc-tonywu! I'm here to help you with any bugs, questions, or contributions. Let me know how I can assist you while you wait for a human maintainer.

I found a similar closed issue that might be relevant to your situation: [Bug][Gitlab Plugin] Missing X-Total and X-Total-Pages headers cause data losing in gitlab plugin](#4093). This issue was closed, and it involved the GitLab plugin handling pagination when certain headers are missing ^[1].

In the current implementation of the GitLab plugin in DevLake, pagination is handled by checking the x-next-page header. If this header is missing, the function assumes there are no more pages to fetch. This means that if both x-total-page and x-next-page headers are missing, the plugin might not fetch all available records ^[2].

Unfortunately, I couldn't find specific logic in the codebase to handle cases where GitLab's API returns more than 10,000 records or when both headers are missing. You might need to consider implementing a workaround or custom logic to handle such scenarios.

To continue talking to Dosu, mention @dosu.

[klesh]

Hi, Please upgrade to https://github.com/apache/incubator-devlake/releases/tag/v1.0.2-beta5 and see if the problem is fixed. Thanks.