[Bug][Azure DevOps] PR data is capped at 101 for one of our Projects

[vincentpatella]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

PR data is capped at 101 for one of our Projects. We have a project that is pulling data from Azure DevOps to collect metrics on the repo. One of the metrics, Number of New Pull Requests, always shows 101 entries for the month. However, it has showed this same number for the past month. When we investigated it, it looks like earlier entries are being deleted or not counted by DevLake, capping the amount of PRs. This is preventing us from getting accurate data for this repo. We have updated our Blueprint to collect from the last 6 months and run 'Collect Data in Full Refresh Mode', but neither helped get the older data. Is there another setting we're missing that will allow us to collect this data, or is this a bug within DevLake?

What do you expect to happen

We expect to see PR data from the past 6 months for our repo show up in our Grafana dashboard using the stock Azure DevOps dashboard.

How to reproduce

1. Set up a Project
2. Add an Azure DevOps Data Connection
3. Select a repo that gets lots of PRs (over 100/ month)
4. Set the sync policy for the past 6 months
5. Collect data
6. Check default Azure DevOps dashboard in Grafana to see if all PRs appear

Anything else

We have tried this on two versions of DevLake, but got the same error

• v1.0.1-beta8
• v1.0.2-beta4

Version

v1.0.2-beta4

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

I found a similar issue related to data collection limits in DevLake. There was an issue with the Enterprise GitHub Plugin where only a limited number of pull requests were collected. The solution suggested was to change the "Time After" setting in the Sync policy section to resolve the issue of collecting only the most recent data ^[1].

For your specific case with Azure DevOps, you might want to check the default limit for pull requests, which is set to 50 in the SQL query configuration ^[2]. Additionally, ensure that your Sync Policy settings are correctly configured to cover the desired time range and that the "Collect Data in Full Refresh Mode" is functioning as expected, as it should delete existing data within the designated time range and re-collect it ^[3].

If these suggestions do not resolve the issue, it might be worth investigating further if there are any specific limitations or bugs in the versions you are using, although I couldn't find any known issues for those specific versions ^[2].

To continue talking to Dosu, mention @dosu.

[vincentpatella]

1. I don't see an option to edit the "Time After" field, but I did specify the date range to specifically look for the past 6 months
2. Setting that value to 5000 in the dashboard and saving that change in a copy of the default dashboard did not make any difference
3. It looks like my Sync Policy settings are correct

I believe this needs more investigation, as these solutions did not help.