Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Existing PSA_Requests should be available to Elastic Provider #794

Open
flea89 opened this issue Dec 15, 2021 · 8 comments · May be fixed by #1844
Open

Existing PSA_Requests should be available to Elastic Provider #794

flea89 opened this issue Dec 15, 2021 · 8 comments · May be fixed by #1844
Labels
P1 High: Likely tackled by core team if no one steps up pi/psa-follow-up topic/pot Issues handled by PT. topic/psa

Comments

@flea89
Copy link
Contributor

flea89 commented Dec 15, 2021 •
edited
Loading

Overarching user needs / context:

  • When migrating pinning to Elastic Provider (pick-up service), we will need the existing pinned pins to be on S3 so the Elastic Provider can provide them.
  • Also, we need to make sure they are stored in 2 systems rather than 1, so we will still be able to retrieve pinned files from IPFS once we migrate the PSA to use picup.

Scope of this ticket:

  • A backup job that copies existing pins that are marked as pinned to S3. It should be written in a way that it can be run repeatedly from cron, to keep backing up pinned pins.

Acceptance criteria:

  • All pinned pins, processed through PSA to cluster should appear in S3
  • The script/job can be run multiple times
@flea89 flea89 mentioned this issue Dec 17, 2021
Merged
@mbommerez mbommerez added the topic/pot Issues handled by PT. label Jan 20, 2022
@mbommerez
Copy link

Removing this from the PSA Epic and adding as a separate item to the backlog.

@mbommerez mbommerez added P4 kind/enhancement A net-new feature or improvement to an existing feature labels Mar 17, 2022
@dchoi27 dchoi27 added P2 Medium: Good to have, but can wait until someone steps up potential/support-pi and removed P4 labels Apr 7, 2022
@mbommerez
Copy link

Discussed with @alanshaw @flea89 @francois-potato .

  • Existing CRON job could be repurposed.
  • We cannot back up pins straight away, as we don't have the data.
  • As soon as we get the pin request we could fetch it in the background and send it to S3.
  • Makes sense to put it in Elastic Provider.
  • Regardless of the above, we should be backing up files we have pinned, as we do for uploads.
  • It needs to be built in a way that it can be used in NFT.storage as well

@flea89 flea89 changed the title Pinning - Backups CIDs pinned through the PSA pinning APIs should be backup on S3 May 11, 2022
@flea89 flea89 changed the title CIDs pinned through the PSA pinning APIs should be backup on S3 CIDs pinned through the Pinning Service APIs should be backup to S3 May 11, 2022
@flea89 flea89 added the blocked label Jul 13, 2022
@olizilla
Copy link
Contributor

We should implement a backup job that copies existing pins that are marked as pinned to S3. It should be written in a way that it can be run repeatedly from cron, to keep backing up pinned pins. We need to unblock that part of the work asap. Is anything else needed here @flea89 @mbommerez ?

@flea89
Copy link
Contributor Author

flea89 commented Jul 13, 2022

@alanshaw wrote something back in the days, I can definitively have a look and re-purpose/adapt if needed.

@olizilla I'd like to understand how this relates to Elastic Provider work....
I'm wondering if, with the rise of EP, this task can be translated to "let's make existing PSA_REQUESTS available to EP, rather than back up pins from cluster to a "cold storage".
Is this the right assumption?
Or that will come after?

@olizilla
Copy link
Contributor

We need the existing pinned pins to be on S3 so the elastic provider can provide them, and so that they are stored in 2 systems rather than 1. So yes, you can reframe it as let's make existing PSA_REQUESTS available to EP if that helps.

@flea89
Copy link
Contributor Author

flea89 commented Jul 13, 2022

Yup, that makes sense, thanks for validating my assumptions!

That been clarified, the reason for blocking it was to understand with you if this had any dependency with your service.
Given atm we don't have a huge amount of context on how you're approaching it and we haven't discussed yet how w3 and service will integrate, I was keen to get your POV before approaching this.

Anyway with the assumption psa state is kept in w3 (not in your service) it shouldn't be a big deal.
But please advise if I'm missing anything.
I'll unblock.

@flea89 flea89 removed the blocked label Jul 13, 2022
@mbommerez
Copy link

mbommerez commented Jul 21, 2022 •
edited
Loading

Repurposing this ticket as a migration script ticket.

@mbommerez mbommerez changed the title CIDs pinned through the Pinning Service APIs should be backup to S3 Existing PSA_REQUESTS should be available to Elastic Provider Jul 21, 2022
@mbommerez mbommerez added P1 High: Likely tackled by core team if no one steps up and removed P2 Medium: Good to have, but can wait until someone steps up kind/enhancement A net-new feature or improvement to an existing feature stack/write-services labels Jul 21, 2022
@mbommerez mbommerez changed the title Existing PSA_REQUESTS should be available to Elastic Provider Existing PSA_Requests should be available to Elastic Provider Jul 22, 2022
@mbommerez mbommerez linked a pull request Sep 6, 2022 that will close this issue
feat: new job to backup pins to s3 #1844
Open
@mbommerez mbommerez assigned flea89 and unassigned joshghent Sep 12, 2022
@flea89 flea89 removed their assignment Sep 12, 2022
@mbommerez
Copy link

  • Most issues have been fixed, this is close to be done.
  • New end-2-end tests have been added (try to reach cluster) on top of the previous mocked ones.
  • We cannot reach the cluster peer locally with Dagula - this could be to do with cluster setup with docker, rather than a code problem.

@flea89 flea89 removed their assignment Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P1 High: Likely tackled by core team if no one steps up pi/psa-follow-up topic/pot Issues handled by PT. topic/psa
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: new job to backup pins to s3 web3-storage/web3.storage
5 participants
@olizilla @joshghent @flea89 @dchoi27 @mbommerez