Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idea/Proposal: EDGE-URID #29

Open
alvin-reyes opened this issue Jun 18, 2023 · 1 comment
Open

Idea/Proposal: EDGE-URID #29

alvin-reyes opened this issue Jun 18, 2023 · 1 comment

Comments

@alvin-reyes
Copy link
Contributor

Edge-urid

image

Functional / features

  • Content creator
  • File aggregator
    • Aggregates small files (> BUCKET_AGG_SIZE) to bucket
    • The last file to be added can exceed the BUCKET_AGG_SIZE
    • Creates a CAR file for the bucket
  • File splitter
    • Splits a large file into (SPLIT_SIZE configuration)
    • Applies to files that are larger than the BUCKET_AGG_SIZE.
    • The large file will be associated to a new bucket with each split being added as a content
    • Creates a CAR file for the bucket
  • Bucket-creator
    • If the content has a miner, a bucket will be created for the miner and for the content. This bucket will need to be filled based on BUCKET_AGG_SIZE.
  • Deal-checker
    • If bucket is already replicated more than (MAX_REP) times, it can be deleted by the ADMIN
  • Gateway for serving content

System objects

  • Content
  • Content_Deals
  • Buckets

Endpoints

To serve file

  • gw/

To add a file

  • add
    • File, bucket_uuid (optional), miner (optional)
  • gw
    • File, bucket_uuid (optional), miner (optional)
  • fetch-from-url
    • File, bucket_uuid (optional), miner (optional)

To get buckets (pull model from SP)

  • get-available-open-buckets
    • Return list of CAR files and COMMP and deal request metadata
  • Get-open-bucket
    • Return the CAR file and COMMP and deal request metadata
  • get-private-bucket (private)
    • Return list of CAR files and COMMP of each for a specific bucket

To manage buckets

  • create-bucket (admin or anyone)
    • create a bucket with specific meta/keys
    • bucket_uuid
    • miner
  • delete-buckets

To get status

  • status/content
  • status/bucket
  • status/cid

To get stats

  • stats
    • Number of content
    • Number of buckets
    • Number of deals attempted
    • Number of deals made
@alvin-reyes alvin-reyes changed the title Proposal: EDGE-URID Idea/Proposal: EDGE-URID Jun 19, 2023
@jcace
Copy link

jcace commented Jun 20, 2023

From our chat today @alvin-reyes , quick outline of the tag idea:

  • Admin endpoints to create and delete tags (+ modify ? )
  • When user uploads content, they can optionally specify a tag. If unspecified, uses default tag
  • tag is where we specify the BUCKET_AGG_SIZE and SPLIT_SIZE parameters
  • buckets are a lower level abstraction,tag is also automatically to the resulting bucket where contents end up
  • a single bucket will never mix different tag s -> resulting carfiles will only contain contents with the same tag
  • Thus, tag:bucket is 1:many , tag:content is also 1:many
  • When user calls get-available-open-buckets, return the tag param for each bucket too

Open questions:

  • Not entirely sure how to handle the delete case. If a tag has already generated a bunch of carfiles then what happens to them?
  • Allowing tag renaming would also be nice too. Easy to do if tag are indexed by ID. Just need to make sure we use the id when referencing from DDM.
  • Could a tag's BUCKET_AGG_SIZE and SPLIT_SIZE be modified after it's set?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants