Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to fetch all data from Apis that implement pagination mechanisms #302

Open
dpsenner opened this issue Aug 29, 2022 · 2 comments
Open
Labels
type/feature-request Requested new feature or enhancement

Comments

@dpsenner
Copy link

dpsenner commented Aug 29, 2022

There are many api's out there that implement pagination mechanisms. To mention some:

  1. Atlassian Jira
  2. Teamviewer Api

It would be great if the plugin would support these apis better by leveraging the pagination and fetching more data. This is not as easy as it might sound. I can't propose a sound technical spec on how to implement this. Therefore I'm highlighting two concrete examples:

Teamviewer Api endpoint https://webapi.teamviewer.com/api/v1/reports/devices

This api sends an optional next_offset item in the json response that should be sent as http argument offset_id to fetch more data. For example the first response to /reports/devices?from_date=2021-08-01T05:14:00Z json may look like:

{
   ...,
   "next_offset": "eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0="
}

And the following request to fetch more data should now be sent with the parameter /reports/devices?offset_id=eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0=.

A smart guy would be able to base64 decode eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0= to:

{"Start":"2021-08-01T05:14:00+00:00","TableContinuationToken":"8efb4267-2355-4c31-b2bf-48319d32394a"}

Atlassian Jira Api endpoint https://jira.example.com/rest/api/2/search

This api endpoint expects numeric http parameters maxResults and startAt while returning startAt, maxResults, total in the json response. For example requesting ?maxResults=2000&startAt=100 the following is returned:

{
    "startAt": 100,
    "maxResults": 1000,
    "total": 1521,
    ...
}

Note that maxResults is capped at 1000.

@vdufloth
Copy link

Hi @dpsenner where you able to connect to Jira using JSON API at all, or did this very issue prevented it from working?

With curl and postman I can fetch the data, but on Grafana it just says: "JSON API: Cannot connect to API"

image

@dpsenner
Copy link
Author

dpsenner commented May 11, 2023

@vdufloth Yes, I could fetch the data but ended up with partial results. This is not at all useful and confusing at best. At the moment I can't recall how I managed to get it working.

@gabor gabor added the type/feature-request Requested new feature or enhancement label May 24, 2024
@gabor gabor moved this to Backlog in OSS Big Tent May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature-request Requested new feature or enhancement
Projects
Status: Backlog
Development

No branches or pull requests

3 participants