Skip to content

Commit

Permalink
docs: usability/doc clean up
Browse files Browse the repository at this point in the history 
- if the user doesn't specify any params to bosh-gcscli show help text
- clean up README a bit, add notes on how to run integration tests
  • Loading branch information
@johnsonj
johnsonj committed Oct 25, 2017
1 parent 7a2a1ad commit 4affb91
Show file tree
Hide file tree
Showing 2 changed files with 83 additions and 98 deletions.
134 changes: 64 additions & 70 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,87 +1,81 @@
## GCS CLI
## bosh-gcscli

A CLI for uploading, fetching and deleting content to/from
the [GCS blobstore](https://cloud.google.com/storage/). This is **not**
an official Google Product.
[![GoDoc](https://godoc.org/github.com/cloudfoundry/bosh-gcscli?status.svg)](https://godoc.org/github.com/cloudfoundry/bosh-gcscli)

## Installation

```
go get github.com/cloudfoundry/bosh-gcscli
```
A Golang CLI for uploading, fetching and deleting content to/from [Google Cloud Storage](https://cloud.google.com/storage/).
This tool exists to work with the [bosh-cli](https://github.com/cloudfoundry/bosh-cli) and [director](https://github.com/cloudfoundry/bosh).

## Usage

Given a JSON config file (`config.json`)...

``` json
{
"bucket_name": "name of GCS bucket (required)",

"credentials_source": "flag for credentials
(optional, defaults to Application Default Credentials)
(can be "static" for json_key),
(can be "none" for explicitly no credentials)"
"storage_class": "storage class for objects
(optional, defaults to bucket settings)",
"json_key": "JSON Service Account File
(optional, required for static credentials)",
"encryption_key": "Base64 encoded 32 byte Customer-Supplied
encryption key used to encrypt objects
(optional)"
}
```
This is **not** an official Google Product.

## Installation

Empty `credentials_source` implies attempting to use Application Default
Credentials. `none` as `credentials_source` specifies no read-only scope
with explicitly no credentials. `static` as `credentials_source` specifies to
use the [Service Account File](https://developers.google.com/identity/protocols/OAuth2ServiceAccount) included
in `json_key`.
```bash
go get github.com/cloudfoundry/bosh-gcscli
```

Empty `storage_class` implies using the default for the bucket.
## Commands

``` bash
# Usage
### Usage
```bash
bosh-gcscli --help

# Command: "put"
# Upload a blob to the GCS blobstore.
```
### Upload an object
```bash
bosh-gcscli -c config.json put <path/to/file> <remote-blob>

# Command: "get"
# Fetch a blob from the GCS blobstore.
# Destination file will be overwritten if exists.
```
### Fetch an object
```bash
bosh-gcscli -c config.json get <remote-blob> <path/to/file>

# Command: "delete"
# Remove a blob from the GCS blobstore.
```
### Delete an object
```bash
bosh-gcscli -c config.json delete <remote-blob>

# Command: "exists"
# Checks if blob exists in the GCS blobstore.
bosh-gcscli -c config.json exists <remote-blob>
```
### Check if an object exists
```bash
bosh-gcscli -c config.json exists <remote-blob>```
```

Alternatively, this package's underlying client can be used to access GCS,
see the [godoc](https://godoc.org/github.com/cloudfoundry/bosh-gcscli)
for more information.

## Tooling

A Makefile is provided for ease of development. Targets are annotated
with descriptions.

gvt is used for vendoring. For full usage, see the [manual at godoc](https://godoc.org/github.com/FiloSottile/gvt).

Integration tests expect to be run from a host with [Application Default
Credentials](https://developers.google.com/identity/protocols/application-default-credentials)
available which has permissions to create and delete buckets.
Application Default Credentials are present on any GCE instance and inherit
the permisions of the [service account](https://cloud.google.com/iam/docs/service-accounts)
assigned to the instance.
## Configuration
The command line tool expects a JSON configuration file. Run `bosh-gcscli --help` for details.

### Authentication Methods (`credentials_source`)
* `static`: A [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) key will be provided via the `json_key` field.
* `none`: No credentials are provided. The client is reading from a public bucket.
* &lt;empty&gt;: [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials)
will be used if they exist (either through `gcloud auth application-default login` or a [service account](https://cloud.google.com/iam/docs/understanding-service-accounts)).
If they don't exist the client will fall back to `none` behavior.
## Running Integration Tests
1. Ensure [gcloud](https://cloud.google.com/sdk/downloads) is installed and you have authenticated (`gcloud auth login`).
These credentials will be used by the Makefile to create/destroy Google Cloud Storage buckets for testing.
1. Set the Google Cloud project: `gcloud config set project <your project>`
1. Generate a service account with the `Storage Admin` role for your project and set the contents as
the environment variable `GOOGLE_APPLICATION_CREDENTIALS`, for example:
```bash
export project_id=$(gcloud config get-value project)
export service_account_name=bosh-gcscli-integration-tests
export service_account_email=${service_account_name}@${project_id}.iam.gserviceaccount.com
credentials_file=$(mktemp)
gcloud config set project ${project_id}
gcloud iam service-accounts create ${service_account_name} --display-name "Integration Test Access for bosh-gcscli"
gcloud iam service-accounts keys create ${credentials_file} --iam-account ${service_account_email}
gcloud project add-iam-policy-binding ${project_id} --member serviceAccount:${service_account_email} --role roles/storage.admin
export GOOGLE_APPLICATION_CREDENTIALS="$(cat ${credentials_file})"
```
1. Run the unit and fast integration tests: `make test-fast-int`
1. Clean up buckets: `make clean-gcs`
## Development
* A Makefile is provided that automates integration testing. Try `make help` to get started.
* [gvt](https://godoc.org/github.com/FiloSottile/gvt) is used for vendoring.
## License
This library is licensed under Apache 2.0. Full license text is
available in [LICENSE](LICENSE).
This tool is licensed under Apache 2.0. Full license text is available in [LICENSE](LICENSE).
47 changes: 19 additions & 28 deletions main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,29 +27,23 @@ import (
"golang.org/x/net/context"
)

var version string
var version = "dev"

// usageExample provides examples of how to use the CLI.
//
// This is used when printing the help text.
const usageExample = `
# Usage
bosh-gcscli --help
# Command: "put"
# Upload a blob to the GCS blobstore.
bosh-gcscli -c config.json put <path/to/file> <remote-blob>
# Command: "get"
# Fetch a blob from the GCS blobstore.
# Destination file will be overwritten if exists.
bosh-gcscli -c config.json get <remote-blob> <path/to/file>
# Command: "delete"
# Remove a blob from the GCS blobstore.
bosh-gcscli -c config.json delete <remote-blob>
# Command: "exists"
# Checks if blob exists in the GCS blobstore.
bosh-gcscli -c config.json exists <remote-blob>`

Expand All @@ -58,30 +52,27 @@ var (
shortHelp = flag.Bool("h", false, "Print this help text")
longHelp = flag.Bool("help", false, "Print this help text")
configPath = flag.String("c", "",
`JSON config file (ie, config.json).
`path to a JSON file with the following contents:
{
"bucket_name": "name of GCS bucket (required)",
"credentials_source": "flag for credentials
(optional, defaults to Application Default Credentials)
(can be "static" for json_key),
(can be "none" for explicitly no credentials)"
"bucket_name": "name of Google Cloud Storage bucket (required)",
"credentials_source": "Optional, defaults to Application Default Credentials or none)
(can be 'static' for a service account specified in json_key),
(can be 'none' for explicitly no credentials)"
"json_key": "JSON Service Account File
(optional, required for 'static' credentials)",
"storage_class": "storage class for objects
(optional, defaults to bucket settings)",
"json_key": "JSON Service Account File
(optional, required for static credentials)",
"encryption_key": "Base64 encoded 32 byte Customer-Supplied
encryption key used to encrypt objects
(optional)"
encryption key used to encrypt objects
(optional, defaults to GCS controlled key)"
}
storage_class is one of MULTI_REGIONAL, REGIONAL, NEARLINE, or COLDLINE.
See the docs for characteristics and location compatibility.
https://cloud.google.com/storage/docs/storage-classes
For more information on characteristics and location compatibility:
https://cloud.google.com/storage/docs/storage-classes
For more information on Customer-Supplied encryption keys,
see the docs.
https://cloud.google.com/storage/docs/encryption
For more information on Customer-Supplied encryption keys:
https://cloud.google.com/storage/docs/encryption
`)
)

Expand All @@ -93,7 +84,7 @@ func main() {
os.Exit(0)
}

if *shortHelp || *longHelp {
if *shortHelp || *longHelp || len(flag.Args()) == 0 {
flag.Usage()
fmt.Println(usageExample)
os.Exit(0)
Expand Down Expand Up @@ -129,7 +120,7 @@ func main() {
switch cmd {
case "put":
if len(nonFlagArgs) != 3 {
log.Fatalf("Put method expected 3 arguments got %d\n", len(nonFlagArgs))
log.Fatalf("put method expected 3 arguments got %d\n", len(nonFlagArgs))
}
src, dst := nonFlagArgs[1], nonFlagArgs[2]

Expand All @@ -144,7 +135,7 @@ func main() {
fmt.Println(err)
case "get":
if len(nonFlagArgs) != 3 {
log.Fatalf("Get method expected 3 arguments got %d\n", len(nonFlagArgs))
log.Fatalf("get method expected 3 arguments got %d\n", len(nonFlagArgs))
}
src, dst := nonFlagArgs[1], nonFlagArgs[2]

Expand All @@ -158,13 +149,13 @@ func main() {
err = blobstoreClient.Get(src, dstFile)
case "delete":
if len(nonFlagArgs) != 2 {
log.Fatalf("Delete method expected 2 arguments got %d\n", len(nonFlagArgs))
log.Fatalf("delete method expected 2 arguments got %d\n", len(nonFlagArgs))
}

err = blobstoreClient.Delete(nonFlagArgs[1])
case "exists":
if len(nonFlagArgs) != 2 {
log.Fatalf("Exists method expected 2 arguments got %d\n", len(nonFlagArgs))
log.Fatalf("exists method expected 2 arguments got %d\n", len(nonFlagArgs))
}

var exists bool
Expand Down

0 comments on commit 4affb91

Please sign in to comment.