S3 to GCS

s3-to-gcs is a lambda function that allows you to perform a one way file copy from an existing Amazon S3 bucket to an existing Google Cloud Storage Bucket.

Prerequisites

AWS Setup

1. Create IAM credentials in order to deploy the lambda function by following this guide
2. Ensure that the S3 bucket is created

Google Cloud Platform setup

1. Create a service account for the lambda function by following this guide
2. Create a set of access keys for the service account by following this guide. Make sure the key type is JSON.
3. Save the access key json file as service-account.json in the root folder of this repository (don't worry, it's already gitignored)
4. Assign "Legacy Bucket Writer" role to the GCS bucket for this service account

Installation

Install serverless and dependencies

Install the serverless framework by running the following:

npm install serverless -g

Install the other dependencies by running

npm install

sls plugin install -n serverless-python-requirements --aws_bucket <aws_bucket_name> --gcs_bucket <gcs_bucket_name>
sls plugin install -n serverless-plugin-existing-s3 --aws_bucket <aws_bucket_name> --gcs_bucket <gcs_bucket_name>

Deploy the lambda function

# This deploys the lambda function, the IAM role,
sls deploy --aws_bucket <aws_bucket_name> --gcs_bucket <gcs_bucket_name> --aws-profile <aws_profile>

# This deploys the S3 event trigger for an existing S3 bucket
sls s3deploy --aws_bucket <aws_bucket_name> --aws-profile <aws_profile>

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

MIT