Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(s3): Don't hardcode endpoint and region #2159

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

cogk
Copy link
Contributor

@cogk cogk commented Sep 17, 2024

Don't hardcode endpoint and region for S3 buckets

There were many references to the ap-south-1 region and a global implicit reference to AWS. This PR contains refactors of the S3 bucket systems for "offsite backups", "dashboard uploads" and "Backup Bucket doctype" buckets, to avoid vendor lock-in.

I tried my best to make it backwards compatible with what I assume is the configuration on Frappe Cloud, but I can't check.

Details:

  • Added region and endpoint fields in Press Settings for the "offsite backups" and "dashboard uploads" buckets
    • Fields are: offsite_backups_endpoint, remote_uploads_region, remote_uploads_endpoint
  • Dropped unused fields: offsite_backups_provider (replaced by endpoint), data_40 (typo probably)
  • There might be some agent stuff left that is not handled correctly (e.g. retrieve the ENDPOINT_URL param here)
  • I did not add unit tests

cc: @gavindsouza 👋
re: #19 #22


For the Frappe team

  • There was a hardcoded uploads.frappe.cloud URL somewhere, I just kept it unchanged, but the refactor won't be truly finished until a migration path can be found for you.
  • I'm not so sure about the other buckets (Backup Bucket doctype), they already had the Endpoint and Region fields, but these were not used everywhere (e.g. not in delete_s3_file)

Typical configuration values for the fields

AWS S3 (Fallback/Legacy)

  • Region: leave empty for ap-south-1
  • Endpoint: leave empty for automatic endpoint generation by boto3 = AWS S3 (I believe https://s3.amazonaws.com)

AWS S3

  • Region: ap-south-1
  • Endpoint: https://s3.ap-south-1.amazonaws.com (possibly a custom DNS name such as uploads.example.com)

OVH

  • Region: de
  • Endpoint: https://storage.de.cloud.ovh.net

@@ -1854,28 +1852,26 @@ def get_upload_link(file, parts=1):

@frappe.whitelist()
def multipart_exit(file, id, action, parts=None):
press_settings: "PressSettings" = frappe.get_single("Press Settings") # type: ignore
bucket_name = "uploads.frappe.cloud" # TODO: Use press_settings.remote_uploads_bucket
Copy link
Contributor Author

@cogk cogk Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the Frappe team can confirm that the current config on Frappe Cloud is indeed
remote_uploads_bucket = "uploads.frappe.cloud"

Which I believe is the case:

Suggested change
bucket_name = "uploads.frappe.cloud" # TODO: Use press_settings.remote_uploads_bucket
bucket_name = press_settings.remote_uploads_bucket

@cogk cogk force-pushed the refactor-endpoint-region-for-s3-buckets branch from 0d3cb6c to 28a422e Compare September 17, 2024 14:16
@NagariaHussain
Copy link
Member

A patch could be written to set the value back in Press Settings if migration needs to be automatic.

@cogk
Copy link
Contributor Author

cogk commented Oct 17, 2024

  • Need to migrate boto3_offsite_backup_session for audit

@cogk cogk force-pushed the refactor-endpoint-region-for-s3-buckets branch from 88b64f3 to aa12651 Compare October 17, 2024 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants