Data fetching on client-side directly from GitHub API

[AlexxNica]

Working on getting better loading times for roadmaps with suggestions from #33.
This is still a draft and it needs some cleanup and syncing before moving to ready for review.

Changes

• Decoupled functions from our API to a new module we can use from the client-side.
• The new module doesn't require GitHub API tokens and instead relies on the client's rate limiting for public GitHub API calls.
  • In the future we can have a fail-safe mode to fetch roadmaps using other methods if clients hit their limit, although we should be cautious due to possible abuse.
• Calling GitHub APIs directly from the client without having to jump through our endpoints is expected to improve loading times and operational overhead by a lot, I still need to do some testing but seems like a good assumption.

[vercel]

@AlexxNica is attempting to deploy a commit to the ipfs-ignite Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
starmaps	❌ Failed (Inspect)			Feb 17, 2023 at 4:26AM (UTC)

[whizzzkid]

Some thoughts around following your approach on how it works and improves performance:

• the public access limits are not published by github. This article suggests the calls are rate limited to 60req/hour. The bacalhau roadmap sends 63 requests and it can grow larger, so one load is going to exhaust that limit and it would need to fallback to the implemented proxy api anyways, traversing through the roadmap would need not be able to call public apis and we’ll spend time waiting for the calls to fail before falling back.
• We’ll also need to compute responses on the client side instead of server side, that’s ok but just another cost to think about.
• Caching responses in service worker won’t be reliable either because responses from other origins would be marked as opaque responses which would would not allow us to compare the network responses and cached responses, which means every update is basically new update. We’ll also might approach the service worker storage limit, because each opaque response would be counted as 7.5mb response, so 63 calls, = 472.5mb of storage for one roadmap.

[AlexxNica]

Preliminary tests with cache disabled

Before

After

[AlexxNica]

Caching responses in service worker won’t be reliable either because responses from other origins would be marked as opaque responses which would would not allow us to compare the network responses and cached responses, which means every update is basically new update. We’ll also might approach the service worker storage limit, because each opaque response would be counted as 7.5mb response, so 63 calls, = 472.5mb of storage for one roadmap.

@whizzzkid are there other ways to cache responses from the client without the downside of having to store them as opaque responses?

What do you think of using standard HTTP caching mixed with the service worker in the background to check for changes/updates?

We could use conditional requests to check if the content has changed before calls.