diff --git a/README.md b/README.md index 90539e1b..dae72d9f 100644 --- a/README.md +++ b/README.md @@ -1,87 +1,81 @@ -## GCS CLI +## bosh-gcscli -A CLI for uploading, fetching and deleting content to/from -the [GCS blobstore](https://cloud.google.com/storage/). This is **not** -an official Google Product. +[![GoDoc](https://godoc.org/github.com/cloudfoundry/bosh-gcscli?status.svg)](https://godoc.org/github.com/cloudfoundry/bosh-gcscli) -## Installation -``` -go get github.com/cloudfoundry/bosh-gcscli -``` +A Golang CLI for uploading, fetching and deleting content to/from [Google Cloud Storage](https://cloud.google.com/storage/). +This tool exists to work with the [bosh-cli](https://github.com/cloudfoundry/bosh-cli) and [director](https://github.com/cloudfoundry/bosh). -## Usage - -Given a JSON config file (`config.json`)... - -``` json -{ - "bucket_name": "name of GCS bucket (required)", - - "credentials_source": "flag for credentials - (optional, defaults to Application Default Credentials) - (can be "static" for json_key), - (can be "none" for explicitly no credentials)" - "storage_class": "storage class for objects - (optional, defaults to bucket settings)", - "json_key": "JSON Service Account File - (optional, required for static credentials)", - "encryption_key": "Base64 encoded 32 byte Customer-Supplied - encryption key used to encrypt objects - (optional)" -} -``` +This is **not** an official Google Product. +## Installation -Empty `credentials_source` implies attempting to use Application Default -Credentials. `none` as `credentials_source` specifies no read-only scope -with explicitly no credentials. `static` as `credentials_source` specifies to -use the [Service Account File](https://developers.google.com/identity/protocols/OAuth2ServiceAccount) included -in `json_key`. +```bash +go get github.com/cloudfoundry/bosh-gcscli +``` -Empty `storage_class` implies using the default for the bucket. +## Commands -``` bash -# Usage +### Usage +```bash bosh-gcscli --help - -# Command: "put" -# Upload a blob to the GCS blobstore. +``` +### Upload an object +```bash bosh-gcscli -c config.json put - -# Command: "get" -# Fetch a blob from the GCS blobstore. -# Destination file will be overwritten if exists. +``` +### Fetch an object +```bash bosh-gcscli -c config.json get - -# Command: "delete" -# Remove a blob from the GCS blobstore. +``` +### Delete an object +```bash bosh-gcscli -c config.json delete - -# Command: "exists" -# Checks if blob exists in the GCS blobstore. -bosh-gcscli -c config.json exists +``` +### Check if an object exists +```bash +bosh-gcscli -c config.json exists ``` ``` -Alternatively, this package's underlying client can be used to access GCS, -see the [godoc](https://godoc.org/github.com/cloudfoundry/bosh-gcscli) -for more information. - -## Tooling - -A Makefile is provided for ease of development. Targets are annotated -with descriptions. - -gvt is used for vendoring. For full usage, see the [manual at godoc](https://godoc.org/github.com/FiloSottile/gvt). - -Integration tests expect to be run from a host with [Application Default -Credentials](https://developers.google.com/identity/protocols/application-default-credentials) -available which has permissions to create and delete buckets. -Application Default Credentials are present on any GCE instance and inherit -the permisions of the [service account](https://cloud.google.com/iam/docs/service-accounts) -assigned to the instance. +## Configuration +The command line tool expects a JSON configuration file. Run `bosh-gcscli --help` for details. + +### Authentication Methods (`credentials_source`) +* `static`: A [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) key will be provided via the `json_key` field. +* `none`: No credentials are provided. The client is reading from a public bucket. +* <empty>: [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials) + will be used if they exist (either through `gcloud auth application-default login` or a [service account](https://cloud.google.com/iam/docs/understanding-service-accounts)). + If they don't exist the client will fall back to `none` behavior. + +## Running Integration Tests + +1. Ensure [gcloud](https://cloud.google.com/sdk/downloads) is installed and you have authenticated (`gcloud auth login`). + These credentials will be used by the Makefile to create/destroy Google Cloud Storage buckets for testing. +1. Set the Google Cloud project: `gcloud config set project ` +1. Generate a service account with the `Storage Admin` role for your project and set the contents as + the environment variable `GOOGLE_APPLICATION_CREDENTIALS`, for example: + ```bash + export project_id=$(gcloud config get-value project) + + export service_account_name=bosh-gcscli-integration-tests + export service_account_email=${service_account_name}@${project_id}.iam.gserviceaccount.com + credentials_file=$(mktemp) + + gcloud config set project ${project_id} + gcloud iam service-accounts create ${service_account_name} --display-name "Integration Test Access for bosh-gcscli" + gcloud iam service-accounts keys create ${credentials_file} --iam-account ${service_account_email} + gcloud project add-iam-policy-binding ${project_id} --member serviceAccount:${service_account_email} --role roles/storage.admin + + export GOOGLE_APPLICATION_CREDENTIALS="$(cat ${credentials_file})" + ``` +1. Run the unit and fast integration tests: `make test-fast-int` +1. Clean up buckets: `make clean-gcs` + +## Development + +* A Makefile is provided that automates integration testing. Try `make help` to get started. +* [gvt](https://godoc.org/github.com/FiloSottile/gvt) is used for vendoring. ## License -This library is licensed under Apache 2.0. Full license text is -available in [LICENSE](LICENSE). \ No newline at end of file +This tool is licensed under Apache 2.0. Full license text is available in [LICENSE](LICENSE). \ No newline at end of file diff --git a/main.go b/main.go index 0daaab5e..2ebbe3d6 100644 --- a/main.go +++ b/main.go @@ -27,29 +27,23 @@ import ( "golang.org/x/net/context" ) -var version string +var version = "dev" // usageExample provides examples of how to use the CLI. -// -// This is used when printing the help text. const usageExample = ` # Usage bosh-gcscli --help -# Command: "put" # Upload a blob to the GCS blobstore. bosh-gcscli -c config.json put -# Command: "get" # Fetch a blob from the GCS blobstore. # Destination file will be overwritten if exists. bosh-gcscli -c config.json get -# Command: "delete" # Remove a blob from the GCS blobstore. bosh-gcscli -c config.json delete -# Command: "exists" # Checks if blob exists in the GCS blobstore. bosh-gcscli -c config.json exists ` @@ -58,30 +52,27 @@ var ( shortHelp = flag.Bool("h", false, "Print this help text") longHelp = flag.Bool("help", false, "Print this help text") configPath = flag.String("c", "", - `JSON config file (ie, config.json). + `path to a JSON file with the following contents: { - "bucket_name": "name of GCS bucket (required)", - - "credentials_source": "flag for credentials - (optional, defaults to Application Default Credentials) - (can be "static" for json_key), - (can be "none" for explicitly no credentials)" + "bucket_name": "name of Google Cloud Storage bucket (required)", + "credentials_source": "Optional, defaults to Application Default Credentials or none) + (can be 'static' for a service account specified in json_key), + (can be 'none' for explicitly no credentials)" + "json_key": "JSON Service Account File + (optional, required for 'static' credentials)", "storage_class": "storage class for objects (optional, defaults to bucket settings)", - "json_key": "JSON Service Account File - (optional, required for static credentials)", "encryption_key": "Base64 encoded 32 byte Customer-Supplied - encryption key used to encrypt objects - (optional)" + encryption key used to encrypt objects + (optional, defaults to GCS controlled key)" } storage_class is one of MULTI_REGIONAL, REGIONAL, NEARLINE, or COLDLINE. - See the docs for characteristics and location compatibility. - https://cloud.google.com/storage/docs/storage-classes + For more information on characteristics and location compatibility: + https://cloud.google.com/storage/docs/storage-classes - For more information on Customer-Supplied encryption keys, - see the docs. - https://cloud.google.com/storage/docs/encryption + For more information on Customer-Supplied encryption keys: + https://cloud.google.com/storage/docs/encryption `) ) @@ -93,7 +84,7 @@ func main() { os.Exit(0) } - if *shortHelp || *longHelp { + if *shortHelp || *longHelp || len(flag.Args()) == 0 { flag.Usage() fmt.Println(usageExample) os.Exit(0) @@ -129,7 +120,7 @@ func main() { switch cmd { case "put": if len(nonFlagArgs) != 3 { - log.Fatalf("Put method expected 3 arguments got %d\n", len(nonFlagArgs)) + log.Fatalf("put method expected 3 arguments got %d\n", len(nonFlagArgs)) } src, dst := nonFlagArgs[1], nonFlagArgs[2] @@ -144,7 +135,7 @@ func main() { fmt.Println(err) case "get": if len(nonFlagArgs) != 3 { - log.Fatalf("Get method expected 3 arguments got %d\n", len(nonFlagArgs)) + log.Fatalf("get method expected 3 arguments got %d\n", len(nonFlagArgs)) } src, dst := nonFlagArgs[1], nonFlagArgs[2] @@ -158,13 +149,13 @@ func main() { err = blobstoreClient.Get(src, dstFile) case "delete": if len(nonFlagArgs) != 2 { - log.Fatalf("Delete method expected 2 arguments got %d\n", len(nonFlagArgs)) + log.Fatalf("delete method expected 2 arguments got %d\n", len(nonFlagArgs)) } err = blobstoreClient.Delete(nonFlagArgs[1]) case "exists": if len(nonFlagArgs) != 2 { - log.Fatalf("Exists method expected 2 arguments got %d\n", len(nonFlagArgs)) + log.Fatalf("exists method expected 2 arguments got %d\n", len(nonFlagArgs)) } var exists bool