Skip to content

Latest commit

 

History

History
274 lines (210 loc) · 16 KB

README.md

File metadata and controls

274 lines (210 loc) · 16 KB
Raw

Supports two main uses cases:

  • Creates and configures a single private S3 bucket for storing logs from various AWS services, which are nested as bucket prefixes. Logs will expire after a default of 90 days, with option to configure retention value.
  • Creates and configures a single private S3 bucket for a single AWS service. Logs will expire after a default of 90 days, with option to configure retention value.

Logging from the following services is supported for both cases as well as in AWS GovCloud:

Terraform Versions

Terraform 0.13. Pin module version to ~> 10.X Submit pull-requests to main branch.

Terraform 0.12. Pin module version to ~> 8.X . Submit pull-requests to terraform12 branch.

Usage for a single log bucket storing logs from all services

# Allows all services to log to bucket
module "aws_logs" {
  source         = "trussworks/logs/aws"
  s3_bucket_name = "my-company-aws-logs"
}

Usage for a single log bucket storing logs from a single service (ELB in this case)

module "aws_logs" {
  source         = "trussworks/logs/aws"
  s3_bucket_name = "my-company-aws-logs-elb"
  default_allow  = false
  allow_elb      = true
}

Usage for a single log bucket storing logs from multiple specified services (ALB and ELB in this case)

module "aws_logs" {
  source         = "trussworks/logs/aws"
  s3_bucket_name = "my-company-aws-logs-lb"
  default_allow  = false
  allow_alb      = true
  allow_elb      = true
}

Usage for a single log bucket storing CloudTrail logs from multiple accounts

module "aws_logs" {
  source              = "trussworks/logs/aws"
  s3_bucket_name      = "my-company-aws-logs-cloudtrail"
  default_allow       = false
  allow_cloudtrail    = true
  cloudtrail_accounts = [data.aws_caller_identity.current.account_id, aws_organizations_account.example.id]
}

Usage for a single log bucket storing logs from multiple application load balancers (ALB) and network load balancers (NLB)

module "aws_logs" {
  source            = "trussworks/logs/aws"
  s3_bucket_name    = "my-company-aws-logs-lb"
      default_allow     = false
      allow_alb         = true
      allow_nlb         = true
      alb_logs_prefixes = [
       "alb/hello-world-prod",
       "alb/hello-world-staging",
       "alb/hello-world-experimental",
      ]
      nlb_logs_prefixes = [
       "nlb/hello-world-prod",
       "nlb/hello-world-staging",
       "nlb/hello-world-experimental",
      ]
    }

Requirements

Name Version
terraform >= 0.13.0
aws >= 3.0, < 4.0

Providers

Name Version
aws >= 3.0, < 4.0

Modules

No modules.

Resources

Name Type
aws_s3_bucket.aws_logs resource
aws_s3_bucket_public_access_block.public_access_block resource
aws_caller_identity.current data source
aws_elb_service_account.main data source
aws_iam_policy_document.main data source
aws_partition.current data source
aws_redshift_service_account.main data source
aws_region.current data source

Inputs

Name Description Type Default Required
alb_account Account for ALB logs. By default limits to the current account. string "" no
alb_logs_prefixes S3 key prefixes for ALB logs. list(string) 
[
  "alb"
]
 no
allow_alb Allow ALB service to log to bucket. bool false no
allow_cloudtrail Allow Cloudtrail service to log to bucket. bool false no
allow_cloudwatch Allow Cloudwatch service to export logs to bucket. bool false no
allow_config Allow Config service to log to bucket. bool false no
allow_elb Allow ELB service to log to bucket. bool false no
allow_nlb Allow NLB service to log to bucket. bool false no
allow_redshift Allow Redshift service to log to bucket. bool false no
cloudtrail_accounts List of accounts for CloudTrail logs. By default limits to the current account. list(string) [] no
cloudtrail_logs_prefix S3 prefix for CloudTrail logs. string "cloudtrail" no
cloudtrail_org_id AWS Organization ID for CloudTrail. string "" no
cloudwatch_logs_prefix S3 prefix for CloudWatch log exports. string "cloudwatch" no
config_accounts List of accounts for Config logs. By default limits to the current account. list(string) [] no
config_logs_prefix S3 prefix for AWS Config logs. string "config" no
create_public_access_block Whether to create a public_access_block restricting public access to the bucket. bool true no
default_allow Whether all services included in this module should be allowed to write to the bucket by default. Alternatively select individual services. It's recommended to use the default bucket ACL of log-delivery-write. bool true no
elb_accounts List of accounts for ELB logs. By default limits to the current account. list(string) [] no
elb_logs_prefix S3 prefix for ELB logs. string "elb" no
enable_mfa_delete A bool that requires MFA to delete the log bucket. bool false no
enable_versioning A bool that enables versioning for the log bucket. bool false no
force_destroy A bool that indicates all objects (including any locked objects) should be deleted from the bucket so the bucket can be destroyed without error. bool false no
logging_target_bucket S3 Bucket to send S3 logs to. Disables logging if omitted. string null no
logging_target_prefix Prefix for logs going into the log_s3_bucket. string "s3/" no
nlb_account Account for NLB logs. By default limits to the current account. string "" no
nlb_logs_prefixes S3 key prefixes for NLB logs. list(string) 
[
  "nlb"
]
 no
noncurrent_version_retention Number of days to retain non-current versions of objects if versioning is enabled. string 30 no
redshift_logs_prefix S3 prefix for RedShift logs. string "redshift" no
s3_bucket_acl Set bucket ACL per AWS S3 Canned ACL list. string "log-delivery-write" no
s3_bucket_name S3 bucket to store AWS logs in. string n/a yes
s3_log_bucket_retention Number of days to keep AWS logs around. string 90 no
tags A mapping of tags to assign to the logs bucket. Please note that tags with a conflicting key will not override the original tag. map(string) {} no

Outputs

Name Description
aws_logs_bucket ID of the S3 bucket containing AWS logs.
configs_logs_path S3 path for Config logs.
elb_logs_path S3 path for ELB logs.
redshift_logs_path S3 path for RedShift logs.

Upgrade Paths

Upgrading from 9.0.0 to 10.x.x

Version 10.x.x removes the region variable as it will pull from the region that your AWS session is associated with.

Upgrading from 6.0.0 to 7.x.x

This release simplifies nlb_logs_prefixes and alb_logs_prefixes to no longer need to pass in a formatted list and instead can be referenced as

nlb_logs_prefixes = [
 "nlb/hello-world-prod",
 "nlb/hello-world-staging",
 "nlb/hello-world-experimental",
]

This release defines more restrictive bucket policies for ALB and NLB logs to include the AWS account id to the allowed path. Terraform plans with this version of the module will look something like

~ Resource  = "arn:aws:s3:::bucket-a-us-west-2/nlb/*" -> "arn:aws:s3:::bucket-a-us-west-2/nlb/AWSLogs/480766629331/*"

Upgrading from 5.0.0 to 5.1.x

Version 5.1.0 removed the nlb_logs_prefix and nlb_accounts variables and now uses one nlb_logs_prefixes list as input. If you had not set the nlb_logs_prefix or nlb_accounts variables, then the default behavior does not change. If you had set nlb_logs_prefix, then simply pass the original value as a 1 item list to nlb_logs_prefixes (while watching that path separators are not duplicated). For example, nlb_logs_prefixes = ["logs/nlb"].

Use the format and formatlist functions in the caller module to support more complex logging that does limit by account id. For example:

    nlb_logs_prefixes = formatlist(format("nlb/%%s/AWSLogs/%s", data.aws_caller_identity.current.account_id), [
      "hello-world-prod",
      "hello-world-staging",
      "hello-world-experimental",
    ])

Upgrading from 4.0.0 to 4.1.x

Version 4.1.0 removed the aws_s3_bucket_policy resource and now applies the bucket policy directly to the aws_s3_bucket resource to address an operation ordering issue when creating a cloudtrail and logs bucket in the same terraform apply. Upgrading a bucket to use version 4.1.0 of the module will update the bucket in-place, but will destroy and recreate the bucket policy.

4.0.0

Version 4.0.0 upgraded to Terraform 12 syntax.

Upgrading from 3.4.0 to 3.5.x

Version 3.5.0 removed the alb_logs_prefix and alb_accounts variables and now uses one alb_logs_prefixes list as input. If you had not set the alb_logs_prefix or alb_accounts variables, then the default behavior does not change. If you had set alb_logs_prefix, then simply pass the original value as a 1 item list to alb_logs_prefixes (while watching that path separators are not duplicated). For example, alb_logs_prefixes = ["logs/alb"].

Use the format and formatlist functions in the caller module to support more complex logging that does limit by account id. For example:

    alb_logs_prefixes = formatlist(format("alb/%%s/AWSLogs/%s", data.aws_caller_identity.current.account_id), [
      "hello-world-prod",
      "hello-world-staging",
      "hello-world-experimental",
    ])

Upgrading from 2.1.X to 3.X.X

Before upgrading you will want to make sure you are on the latest version of 2.1.X.

The variable allow_s3 has been removed. If you were using the variable allow_s3 to manage the bucket ACL or policy creation you'll want to make changes as the variable has been removed. For the bucket ACL you will now use s3_bucket_acl which is set to log-delivery-write by default. If you had default_allow=false and allow_s3=false you'll want to set s3_bucket_acl="private".

If you are using default_allow=true you can skip the rest of this upgrade guide.

As for policy creation, all policies are now turned on or off via the allow_* variables. By setting these to true the effect block in the bucket policy for that resource will be modified to Allow whereas by default it will be set to Deny. Previously this module used a template to add or remove JSON text from the policy before rendering. The new module explicitly adds all resource policies as Deny and leaves it up to you to enable them.

Developer Setup

Install dependencies (macOS)

brew install pre-commit go terraform terraform-docs

Testing

Terratest is being used for automated testing with this module. Tests in the test folder can be run locally by running the following command:

make test

Or with aws-vault:

AWS_VAULT_KEYCHAIN_NAME=login aws-vault exec YOUR-AWS-PROFILE -- make test

A Note About NLB Access Logs

NLB Access logs are created only if the load balancer has a client request-based TLS listener. Also, the logs will only contain information about TLS requests. See the AWS Documentation on Access Logs for further details.

If you're using mTLS to exchange a mutually-trusted Certificate Authority, you may require a TCP listener. While it's true that TLS runs over TCP, for mTLS each new successive connection requires two roundtrips to complete the "full handshake." No NLB access logs will be created in this case.