Skip to content

Latest commit

 

History

History
472 lines (367 loc) · 34.2 KB

README.md

File metadata and controls

472 lines (367 loc) · 34.2 KB
Raw

terraform-opensearch

opensearch

Terraform module to setup all resources needed for setting up an AWS OpenSearch Service domain.

Requirements

Name Version
terraform >= 1.3.9, < 1.6.0
aws ~> 5.0

Providers

Name Version
aws ~> 5.0

Modules

No modules.

Resources

Name Type
aws_cloudwatch_log_group.cwl_application resource
aws_cloudwatch_log_group.cwl_index resource
aws_cloudwatch_log_group.cwl_search resource
aws_cloudwatch_log_resource_policy.cwl_resource_policy resource
aws_elasticsearch_domain.es resource
aws_security_group.sg resource
aws_iam_policy_document.cwl_policy data source
aws_region.current data source
aws_subnet.private data source

Inputs

Name Description Type Default Required
instance_type Instance type to use for the OpenSearch domain string n/a yes
name Name to use for the OpenSearch domain string n/a yes
volume_size EBS volume size (in GB) to use for the OpenSearch domain number n/a yes
application_logging_enabled Whether to enable OpenSearch application logs (error) in Cloudwatch bool false no
availability_zone_count Number of Availability Zones for the domain to use with zone_awareness_enabled.Valid values: 2 or 3. Automatically configured through number of instances/subnets available if not set. number null no
cognito_enabled Whether to enable Cognito for authentication in Kibana bool false no
cognito_identity_pool_id Required when cognito_enabled is enabled: ID of the Cognito Identity Pool to use string null no
cognito_role_arn Required when cognito_enabled is enabled: ARN of the IAM role that has the AmazonESCognitoAccess policy attached string null no
cognito_user_pool_id Required when cognito_enabled is enabled: ID of the Cognito User Pool to use string null no
custom_endpoint The domain name to use as custom endpoint for Elasicsearch string null no
custom_endpoint_certificate_arn ARN of the ACM certificate to use for the custom endpoint. Required when custom endpoint is set along with enabling endpoint_enforce_https string null no
dedicated_master_count Number of dedicated master nodes in the domain (can be 3 or 5) number 3 no
dedicated_master_enabled Whether dedicated master nodes are enabled for the domain. Automatically enabled when warm_enabled = true bool false no
dedicated_master_type Instance type of the dedicated master nodes in the domain string "t3.small.search" no
encrypt_at_rest Whether to enable encryption at rest for the cluster. Changing this on an existing cluster will force a new resource! bool true no
encrypt_at_rest_kms_key_id The KMS key id to encrypt the OpenSearch domain with. If not specified then it defaults to using the aws/es service KMS key string null no
endpoint_enforce_https Whether or not to require HTTPS bool true no
endpoint_tls_security_policy The name of the TLS security policy that needs to be applied to the HTTPS endpoint. Valid values: Policy-Min-TLS-1-0-2019-07 and Policy-Min-TLS-1-2-2019-07 string "Policy-Min-TLS-1-2-2019-07" no
ephemeral_list m3 and r3 are supported by aws using ephemeral storage but are a legacy instance type list(string) 
[
  "i3.large.search",
  "i3.xlarge.search",
  "i3.2xlarge.search",
  "i3.4xlarge.search",
  "i3.8xlarge.search",
  "i3.16xlarge.search",
  "r6gd.large.search",
  "r6gd.xlarge.search",
  "r6gd.2xlarge.search",
  "r6gd.4xlarge.search",
  "r6gd.8xlarge.search",
  "r6gd.12xlarge.search",
  "r6gd.16xlarge.search",
  "i3.large.elasticsearch",
  "i3.xlarge.elasticsearch",
  "i3.2xlarge.elasticsearch",
  "i3.4xlarge.elasticsearch",
  "i3.8xlarge.elasticsearch",
  "i3.16xlarge.elasticsearch",
  "r6gd.large.elasticsearch",
  "r6gd.xlarge.elasticsearch",
  "r6gd.2xlarge.elasticsearch",
  "r6gd.4xlarge.elasticsearch",
  "r6gd.8xlarge.elasticsearch",
  "r6gd.12xlarge.elasticsearch",
  "r6gd.16xlarge.elasticsearch"
]
 no
instance_count Size of the OpenSearch domain number 1 no
logging_enabled Whether to enable OpenSearch slow logs (index & search) in Cloudwatch bool false no
logging_retention How many days to retain OpenSearch logs in Cloudwatch number 30 no
node_to_node_encryption Whether to enable node-to-node encryption. Changing this on an existing cluster will force a new resource! bool true no
options_indices_fielddata_cache_size Sets the indices.fielddata.cache.size advanced option. Specifies the percentage of heap space that is allocated to fielddata number null no
options_indices_query_bool_max_clause_count Sets the indices.query.bool.max_clause_count advanced option. Specifies the maximum number of allowed boolean clauses in a query number 1024 no
options_override_main_response_version Whether to enable compatibility mode when creating an OpenSearch domain. Because certain Elasticsearch OSS clients and plugins check the cluster version before connecting, compatibility mode sets OpenSearch to report its version as 7.10 so these clients continue to work bool true no
options_rest_action_multi_allow_explicit_index Sets the rest.action.multi.allow_explicit_index advanced option. When set to false, OpenSearch will reject requests that have an explicit index specified in the request body bool true no
search_version Version of the OpenSearch domain string "OpenSearch_1.1" no
security_group_ids Extra security group IDs to attach to the OpenSearch domain. Note: a default SG is already created and exposed via outputs list(string) [] no
snapshot_start_hour Hour during which an automated daily snapshot is taken of the OpenSearch indices number 3 no
subnet_ids Required if vpc_id is specified: Subnet IDs for the VPC enabled OpenSearch domain endpoints to be created in list(string) [] no
tags Optional tags map(string) {} no
volume_iops Required if volume_type="io1" or "gp3": Amount of provisioned IOPS for the EBS volume number 0 no
volume_type EBS volume type to use for the OpenSearch domain string "gp2" no
vpc_id VPC ID where to deploy the OpenSearch domain. If set, you also need to specify subnet_ids. If not set, the module creates a public domain string null no
warm_count Number of warm nodes (2 - 150) number 2 no
warm_enabled Whether to enable warm storage bool false no
warm_type Instance type of the warm nodes string "ultrawarm1.medium.search" no
zone_awareness_enabled Whether to enable zone_awareness or not, if not set, multi az is enabled by default and configured through number of instances/subnets available bool null no

Outputs

Name Description
arn ARN of the OpenSearch domain
domain_id ID of the OpenSearch domain
domain_name Name of the OpenSearch domain
domain_region Region of the OpenSearch domain
endpoint DNS endpoint of the OpenSearch domain
kibana_endpoint DNS endpoint of Kibana
sg_id ID of the OpenSearch security group

Example

module "opensearch" {
  source = "github.com/skyscrapers/terraform-opensearch//opensearch?ref=11.3.0"

  name           = "logs-${terraform.workspace}-es"
  instance_count = 3
  instance_type  = "m5.large.elasticsearch"
  volume_size    = 100
  vpc_id         = data.terraform_remote_state.networking.outputs.vpc_id
  subnet_ids     = data.terraform_remote_state.networking.outputs.private_db_subnets
}

data "aws_iam_policy_document" "opensearch" {
  statement {
    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["${aws_iam_user.es_user.arn}"]
    }

    actions   = ["es:*"]
    resources = ["${module.elasticsearch.arn}/*"]
  }
}

resource "aws_elasticsearch_domain_policy" "opensearch" {
  domain_name     = module.opensearch.domain_name
  access_policies = data.aws_iam_policy_document.opensearch.json
}

Logging

This module by default creates Cloudwatch Log Groups & IAM permissions for ElasticSearch slow logging (search & index), but we don't enable these logs by default. You can control logging behavior via the logging_enabled and logging_retention parameters. When enabling this, make sure you also enable this on Elasticsearch side, following the AWS documentation.

You can also enable Elasticsearch error logs via application_logging_enabled = true.

Monitoring

For a CloudWatch based solution, check out our terraform-cloudwatch modules.

For a Kubernetes & Prometheus based solution, see the elasticsearch_k8s_monitoring module below.

NOTES

This module will not work without the ES default role AWSServiceRoleForAmazonElasticsearchService. This service role needs to be created per-account so you will need to add it if not present (just once per AWS account).

Here is a code sample you can use:

resource "aws_iam_service_linked_role" "es" {
  aws_service_name = "es.amazonaws.com"
}

opensearch-backup

This module can be used to create your own snapshots of Opensearch to S3, using Snapshot Management. It can also deploy a PrometheusRule for monitoring snapshot success.

Important

This requires Opensearch >= 2.5!

Requirements

Name Version
terraform >= 1.3.9
aws ~> 5.0
kubernetes ~> 2.23
opensearch ~> 2.2

Providers

Name Version
aws ~> 5.0
kubernetes ~> 2.23
opensearch ~> 2.2

Modules

Name Source Version
s3_snapshot terraform-aws-modules/s3-bucket/aws ~> 3.15

Resources

Name Type
aws_iam_role.snapshot_create resource
aws_iam_role_policy.snapshot_create resource
kubernetes_manifest.prometheusrule resource
opensearch_sm_policy.snapshot resource
opensearch_snapshot_repository.repo resource
aws_iam_policy_document.s3_snapshot_bucket data source
aws_iam_policy_document.snapshot_create data source
aws_iam_policy_document.snapshot_create_assume data source

Inputs

Name Description Type Default Required
name Name for the snapshot system, S3 bucket, etc. string n/a yes
aws_kms_key_arn ARN of the CMK used for S3 Server Side Encryption. When specified, we'll use the aws:kms SSE algorithm. When not specified, falls back to using AES256 string null no
bucket_key_enabled Whether to use Amazon S3 Bucket Keys for encryption, which reduces API costs bool false no
create_cron_expression The cron schedule used to create snapshots string "0 0 * * *" no
create_time_limit Sets the maximum time to wait for snapshot creation to finish. If time_limit is longer than the scheduled time interval for taking snapshots, no scheduled snapshots are taken until time_limit elapses. For example, if time_limit is set to 35 minutes and snapshots are taken every 30 minutes starting at midnight, the snapshots at 00:00 and 01:00 are taken, but the snapshot at 00:30 is skipped string "1h" no
custom_sm_policy Set this variable when you want to override the generated SM policy JSON with your own. Make sure to correctly set snapshot_config.repository to the same value as var.name (the bucket name) string null no
delete_cron_expression The cron schedule used to delete snapshots string "0 2 * * *" no
delete_time_limit Sets the maximum time to wait for snapshot deletion to finish string "1h" no
extra_bucket_policy Extra bucket policy to attach to the S3 bucket (JSON string formatted) string null no
indices The names of the indexes in the snapshot. Multiple index names are separated by ,. Supports wildcards (*) string "*" no
max_age The maximum time a snapshot is retained in S3 string "14d" no
max_count The maximum number of snapshots retained in S3 number 400 no
min_count The minimum number of snapshot retained in S3 number 1 no
prometheusrule_alert_labels Additional labels to add to the PrometheusRule alert map(string) {} no
prometheusrule_enabled Whether to deploy a PrometheusRule for monitoring the snapshots. Requires the prometheus-operator and elasticsearch-exporter to be deployed bool true no
prometheusrule_labels Additional K8s labels to add to the PrometheusRule map(string) 
{
  "prometheus": "opensearch-backup"
}
 no
prometheusrule_namespace Namespace where to deploy the PrometheusRule string "infrastructure" no
prometheusrule_query_period Period to apply to the PrometheusRule queries. Make sure this is bigger than the create_cron_expression interval string "32h" no
prometheusrule_severity Severity of the PrometheusRule alert. Usual values are: info, warning and critical string "warning" no
s3_force_destroy Whether to force-destroy and empty the S3 bucket when destroying this Terraform module. WARNING: Not recommended! bool false no
s3_replication_configuration Replication configuration block for the S3 bucket. See https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/tree/v3.15.1/examples/s3-replication for an example any {} no

Outputs

No outputs.

Example

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
    opensearch = {
      source = "opensearch-project/opensearch"
    }
  }
}

provider "opensearch" {
  url                 = module.opensearch.endpoint
  aws_region          = var._aws_provider_region
  aws_profile         = var._aws_provider_profile
  aws_assume_role_arn = "arn:aws:iam::${var._aws_provider_account_id}:role/${var._aws_provider_assume_role}"
}

module "opensearch_snapshots" {
  source = "github.com/skyscrapers/terraform-opensearch//opensearch-backup?ref=11.3.0"
  name   = "${module.opensearch.domain_name}-snapshots"
}

elasticsearch_k8s_monitoring

This module deploys our elasticsearch/monitoring chart on Kubernetes.

Requirements

Name Version
terraform >= 1.3.9, < 1.6.0
aws ~> 5.0
helm ~> 2.11
kubernetes ~> 2.23

Providers

Name Version
helm ~> 2.11

Modules

No modules.

Resources

Name Type
helm_release.elasticsearch_monitoring resource

Inputs

Name Description Type Default Required
cloudwatch_exporter_role_arn IAM role ARN to use for the CloudWatch exporter. Used via either IRSA or kube2iam (see var.irsa_enabled) string n/a yes
elasticsearch_domain_name Domain name of the AWS Elasticsearch domain string n/a yes
elasticsearch_domain_region Region of the AWS Elasticsearch domain string n/a yes
elasticsearch_endpoint Endpoint of the AWS Elasticsearch domain string n/a yes
kubernetes_namespace Kubernetes namespace where to deploy the skyscrapers/elasticsearch-monitoring chart string n/a yes
cw_exporter_memory Memory request and limit for the prometheus-cloudwatch-exporter pod string "160Mi" no
elasticsearch_monitoring_chart_version elasticsearch-monitoring Helm chart version to deploy string "1.11.2" no
es_exporter_memory Memory request and limit for the prometheus-elasticsearch-exporter pod string "48Mi" no
force_helm_update Modify this variable to trigger an update on all Helm charts (you can set any value). Due to current limitations of the Helm provider, it doesn't detect drift on the deployed values string "1" no
irsa_enabled Whether to use IAM Roles for Service Accounts. When true, the Cloudwatch exporter's SA is appropriately annotated. If false a kube2iam Pod annotation is set instead bool true no
sla SLA of the monitored Elasticsearch cluster. Will default to the k8s cluster SLA if omited string null no
system_nodeSelector nodeSelector to add to the kubernetes pods. Set to null to disable. map(map(string)) 
{
  "nodeSelector": {
    "role": "system"
  }
}
 no
system_tolerations Tolerations to add to the kubernetes pods. Set to null to disable. any 
{
  "tolerations": [
    {
      "effect": "NoSchedule",
      "key": "role",
      "operator": "Equal",
      "value": "system"
    }
  ]
}
 no

Outputs

No outputs.

kibana_k8s_auth_ingress

This module deploys an Ingress with external authentication on Kubernetes to reach the AWS Elasticsearch Kibana endpoint.

Requirements

Name Version
terraform >= 0.12

Providers

Name Version
kubernetes n/a

Inputs

Name Description Type Default Required
elasticsearch_domain_name Domain name of the AWS Elasticsearch domain string n/a yes
elasticsearch_endpoint Endpoint of the AWS Elasticsearch domain string n/a yes
ingress_auth_signin Value to set for the nginx.ingress.kubernetes.io/auth-signin annotation string n/a yes
ingress_auth_url Value to set for the nginx.ingress.kubernetes.io/auth-url annotation string n/a yes
ingress_host Hostname to use for the Ingress string n/a yes
kubernetes_namespace Kubernetes namespace where to deploy the Ingress string n/a yes
ingress_configuration_snippet Value to set for the nginx.ingress.kubernetes.io/configuration-snippet annotation string null no

Outputs

No output.

kibana_k8s_auth_proxy

This module is no longer maintained!

This module deploys keycloack-gatekeeper as OIDC proxy on Kubernetes to reach the AWS Elasticsearch Kibana endpoint.

Inputs

Name Description Type Default Required
elasticsearch_endpoint Endpoint of the AWS Elasticsearch domain string n/a yes
elasticsearch_domain_name Domain name of the AWS Elasticsearch domain string n/a yes
kubernetes_namespace Kubernetes namespace where to deploy the Keycloack-gatekeeper proxy chart string n/a yes
gatekeeper_image Docker image to use for the Keycloack-gatekeeper deployment string "keycloak/keycloak-gatekeeper:6.0.1" no
gatekeeper_ingress_host Hostname to use for the Ingress string n/a yes
gatekeeper_discovery_url URL for OpenID autoconfiguration string n/a yes
gatekeeper_client_id Client ID for OpenID server string n/a yes
gatekeeper_client_secret Client secret for OpenID server string n/a yes
gatekeeper_oidc_groups Groups that will be granted access. When using Dex with GitHub, teams are defined in the form <gh_org>:<gh_team>, for example skyscrapers:k8s-admins string n/a yes
gatekeeper_timeout Upstream timeouts to use for the proxy string "500s" no
gatekeeper_extra_args Additional keycloack-gatekeeper command line arguments list(string) [] no

Outputs

Name Description
callback_uri Callback URI. You might need to register this to your OIDC provider (like CoreOS Dex)

Upgrading

Version 10.0.0 to 11.0.0

We removed the custom S3 backup mechanism (via Lambda) from the opensearch module. As an alternative we now offer a new opensearch-backup module, which relies on the OpenSearch Snapshot Management API to create snapshots to S3.

If you want to upgrade, without destroying your old S3 snapshot bucket, we recommend to remove the bucket from Terraform's state and re-import it into the new backup module. For example, consider your code like this:

module "opensearch" {
  source = "github.com/skyscrapers/terraform-opensearch//opensearch?ref=11.3.0"
  ...
}

module "opensearch_backup" {
  source = "github.com/skyscrapers/terraform-opensearch//opensearch-backup?ref=11.3.0"
  name   = "${module.opensearch.domain_name}-snapshot"
}

Then you can migrate your snapshots S3 bucket like this:

terraform state rm module.opensearch.aws_s3_bucket.snapshot[0]
terraform import module.opensearch_backup.module.s3_snapshot.aws_s3_bucket.this[0] "<opensearch_domain_name>-snapshot"

Also make sure to set var.name of this module to <opensearch_domain_name>-snapshot!

Alternatively you can just let the module create a new bucket.

Version 9.1.4 to 10.0.0

In the elasticsearch_k8s_monitoring module, the variables system_tolerations and system_nodeSelector have been added to isolate the monitoring on a dedicated system node pool. If you don't want this you can override these variables to null to disable.

Version 8.0.0 to 8.2.0

In the opensearch module, the s3_snapshots_schedule_expression variable has been replaced with s3_snapshots_schedule_period. Instead of a cron expression, we only allow to specify a period in hours, which will be used as a rate(x hours).

Version 7.0.0 to 8.0.0

This change migrates the elasticsearch module to opensearch. This is mostly a cosmetic change, however there's several breaking things to note:

  • Security Group description is updated, which would normally trigger a destroy/recreate. However existing setups won't be affected due to an ignore lifecycle
  • Variables project and environment have been removed. Only the name variable is now used. For existing setups, you can set name = "<myproject>-<myenvironment>-<oldname>" to retain the original "name".
  • CloudWatch Log Groups will be destroyed and recreated using the new name. If you wish to keep your older logs, it's best to remove the existing Log Groups from the TF state:
terraform state rm module.elasticsearch.aws_cloudwatch_log_group.cwl_index
terraform state rm module.elasticsearch.aws_cloudwatch_log_group.cwl_search
terraform state rm module.elasticsearch.aws_cloudwatch_log_group.cwl_application
  • Variable elasticsearch_version has been renamed to search_version, with default value OpenSearch_1.1
  • We no longer merge the tags variable with our own hardcoded defaults (Environment, Project, Name) , all tags need to be passed through the tags variable and/or through the default_tags provider setting
  • Updated list of instance types with NVMe SSD storage

Version 6.0.0 to 7.0.0

Behavior of this module in function of backups has changed much between versions 6.0.0 and 7.0.0:

  • Replace the snapshot_bucket_enabled variable with s3_snapshots_enabled
    • Note: This will also enable the Lambda for automated backups
    • If you just want to keep the bucket, you can remove it from the terraform state and manage it outside the module: terraform state rm aws_s3_bucket.snapshot[0]
  • The IAM role for taking snapshots has been renamed. If you want to keep the old role too, you should remove it from the terraform state: terraform state rm module.registrations.aws_iam_role.role[0]
    • Otherwise just let it destroy the old role and it will create a new one

Also note that some default values for variables has beem changed, mostly related to encryption. If this triggers an unwanted change, you can override this by explicitly setting the variable with it's old value.