Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the GetTipsetByHeight in-memory skip list cache persistent for finalised tipsets #12568

Open
aarshkshah1992 opened this issue Oct 9, 2024 · 3 comments
Labels
kind/feature Kind: Feature

Comments

@aarshkshah1992
Copy link
Contributor

aarshkshah1992 commented Oct 9, 2024

When a Lotus node starts up, it kicks-off a "prefilling" of the in-memory GetTipsetByHeight cache in a go-routine. See here.

On Archival nodes, this pre-filling can take hours (RPC queries for tipsets way in the past will take a really long time/fail till this cache is pre-filled). However, because the cache is in-memory, the pre-filling needs to be done all over again once the node restarts.

Once the ChainIndexer ships -> we should re-engineer the GetTipsetByHeight cache to use the persistent ChainIndexer atleast for finalised tipsets. We should also investigate if there's a strong reason to not do this for non-finalised tipsets.

This will improve UX for Archival RPC providers and also speed up RPC queries for finalised tipsets immediately after node boot up.

@aarshkshah1992 aarshkshah1992 added the kind/feature Kind: Feature label Oct 9, 2024
@github-project-automation github-project-automation bot moved this to 📌 Triage in FilOz Oct 9, 2024
@aarshkshah1992 aarshkshah1992 changed the title Make the GetTipsetByHeight in-memory skip list cache persistent for finalised tipsets Make the GetTipsetByHeight in-memory skip list cache persistent for finalised tipsets Oct 9, 2024
@aarshkshah1992
Copy link
Contributor Author

Glif's archival node only took 3 hours for the hydration all the way upto genesis. So it's bad but not that bad.

@BigLep
Copy link
Member

BigLep commented Oct 9, 2024

@aarshkshah1992
Copy link
Contributor Author

Note that we need the ChainIndexer to be deployed and backfilled on nodes before we can use it to replace the skiplist in the tipset cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Kind: Feature
Projects
Status: 🐱 Todo
Development

No branches or pull requests

2 participants