Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firehose chart spec #46

Open
calinah opened this issue Jan 19, 2023 · 0 comments
Open

Firehose chart spec #46

calinah opened this issue Jan 19, 2023 · 0 comments

Comments

@calinah
Copy link
Contributor

calinah commented Jan 19, 2023

Simple setup (non-HA)
Firehose single binary that does :

  • reader - will spun geth by itself (polygon as target to read data):
  • merger - merge blocks into chunks (reader outputs 1 block at the time - merge block gets rid of duplicates). Every merge block in Ethereum has 100 blocks
  • relayer -
  • firehose - does substreams calculations. Tier 1 firehose config is the main config, tier 2 config is needed for running substreams.

Storage:
*1st version store on local filesystem

  • GCS or S3
  • ext4 might not work that well

If blocks are missing rewind.

Firehose HA
Firehose single binary that does :

  • 2 reader - will spun geth by itself (polygon as target to read data):
  • 1 merger - merge blocks into chunks (reader outputs 1 block at the time - merge block gets rid of duplicates). Every merge block in Ethereum has 100 blocks
  • 2 relayer -
  • 2 or more replicas firehose - based on number of users and subgraphs (1000 subgraphs per firehose)

Storage:

  • Object storage or shared filesystem or both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant