Skip to content
This repository has been archived by the owner on Jan 12, 2024. It is now read-only.

Set up requester pays #22

Closed
5 tasks done
zaneselvans opened this issue May 17, 2022 · 1 comment · Fixed by #23
Closed
5 tasks done

Set up requester pays #22

zaneselvans opened this issue May 17, 2022 · 1 comment · Fixed by #23
Assignees
@zaneselvans
Labels
cloud intake Intake data catalogs

Comments

@zaneselvans
Copy link
Member

zaneselvans commented May 17, 2022 •
edited
Loading

After having a few $20 days, we've decided to limit our exposure to unexpected data egress fees by turning on Requester Pays for the storage buckets containing the pudl-catalog.

  • Enable requester pays on gs://intake.catalyst.coop
  • Make the tests supply a billing project so they can access the storage buckets.
  • Check that we're only downloading a minimal amount of data (a couple of state-years) in the tests.
  • Update the example notebook to use requester_pays and new dd.read_parquet() args.
  • Provide documentation / links for setting up a user billing project in the README.
@zaneselvans zaneselvans added intake Intake data catalogs cloud labels May 17, 2022
@zaneselvans zaneselvans self-assigned this May 17, 2022
@zaneselvans
Copy link
Member Author

See also comments on this issue

@zaneselvans zaneselvans linked a pull request May 17, 2022 that will close this issue
Switch to using requester pays on storage buckets #23
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@zaneselvans zaneselvans

Labels
cloud intake Intake data catalogs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Switch to using requester pays on storage buckets catalyst-cooperative/pudl-catalog
1 participant
@zaneselvans