Using with AWS Elasticsearch Service

When creating an Elastic client, you can specify an *http.Client to use instead of the default client.

Since the release of v1.2.0 of the AWS SDK for Go (June 23rd, 2016), signing generic *http.Request instances has become much easier. A wrapper for standard Go *http.Client instances has been written that allows outgoing requests to Amazon Elasticsearch Service (or any other service) to be signed before the request is sent.

To install the package, simply go get github.com/sha1sum/aws_signing_client

As of Jan 19th, 2017 the aws_signing_client has an issue which causes 403 errors. This PR fixes the problem: https://github.com/sha1sum/aws_signing_client/pull/3

For now, you may wish to use https://github.com/deoxxa/aws_signing_client

Example Usage

import (
  "github.com/aws/aws-sdk-go/aws/signer/v4"
  "github.com/aws/aws-sdk-go/aws/credentials"
  "github.com/sha1sum/aws_signing_client"
  "gopkg.in/olivere/elastic.v3"
)

func newElasticClient(creds *credentials.Credentials) (*elastic.Client, error) {
  signer := v4.NewSigner(creds)
  awsClient, err := aws_signing_client.New(signer, nil, "es", "us-east-1")
  if err != nil {
    return nil, err
  }
  return elastic.NewClient(
    elastic.SetURL("https://my-aws-endpoint.us-east-1.es.amazonaws.com"),
    elastic.SetScheme("https"),
    elastic.SetHttpClient(awsClient),
    elastic.SetSniff(false), // See note below
  )
}

Sniffing

AWS Elasticsearch Service occasionally does not return the node details needed by Elastic for sniffing purposes. You can try leaving sniffing enabled, but if you find that ErrNoClient errors are cropping up unexpectedly upon client creation, you may want to disable it. Health checks for the cluster should still operate as normal.

Alternatives

Note: These may no longer work with more recent versions of Elasticsearch and changes in AWS.

Roll your own http.Transport

Thanks to @mthenw who wrote his own http.Transport (see #317) based on https://github.com/smartystreets/go-aws-auth:

import (
	awsauth "github.com/smartystreets/go-aws-auth"
)

...

type AWSSigningTransport struct {
	HTTPClient  *http.Client
	Credentials awsauth.Credentials
}

// RoundTrip implementation
func (a AWSSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	return a.HTTPClient.Do(awsauth.Sign4(req, a.Credentials))
}

Usage:

signingTransport := AWSSigningTransport{
	Credentials: awsauth.Credentials{
		AccessKeyID:     os.Getenv("AWS_ACCESS_KEY"),
		SecretAccessKey: os.Getenv("AWS_SECRET_KEY"),
	},
	HTTPClient: http.DefaultClient,
}
signingClient := &http.Client{Transport: http.RoundTripper(signingTransport)}

return elastic.NewClient(
	elastic.SetURL(...),
	elastic.SetScheme("https"),
	elastic.SetHttpClient(signingClient),
	elastic.SetSniff(false),
)

Apex - AWS Lambda integration

Another library by @edoardo849 uses a RoundTripper that can be tweaked by adding custom timeouts etc... The library is meant to be used within the Apex.run framework in a Lambda function. It also adds an optional feature to log the http request and response for debugging purposes.

Usage:

import (
    "github.com/edoardo849/apex-aws-signer"
    "github.com/apex/log"
    "github.com/aws/aws-sdk-go/service/elasticsearchservice"
    "github.com/aws/aws-sdk-go/aws/session"
    "gopkg.in/olivere/elastic.v3"
)

// Example For ElasticSearch
transport := signer.NewTransport(session.New(&aws.Config{Region: aws.String("aws-region")}), elasticsearchservice.ServiceName)

// This is optional, "ctx" is the *apex.Context
transport.Logger = log.WithField("requestID", ctx.RequestID)

httpClient := &http.Client{
    Transport: transport,
}
// Use the client with Olivere's elastic client
client, err := elastic.NewClient(
    elastic.SetSniff(false),
    elastic.SetURL("your-aws-es-endpoint"),
    elastic.SetScheme("https"),
    elastic.SetHttpClient(httpClient),
)