Skip to content

clouddrove/terraform-aws-elasticsearch

Folders and files

NameName
Last commit message
Last commit date
Apr 8, 2024
Jan 11, 2024
May 25, 2023
Jul 24, 2023
Jul 31, 2023
Jun 22, 2021
May 15, 2023
May 18, 2022
Jun 22, 2021
Jul 31, 2023
Jul 24, 2023
Mar 11, 2024
Jul 31, 2023
Oct 24, 2023
Jan 11, 2024

Repository files navigation

Terraform AWS Elasticsearch

Terraform module to create an Elasticsearch resource on AWS.

Latest Release tfsec Licence


We eat, drink, sleep and most importantly love DevOps. We are working towards strategies for standardizing architecture while ensuring security for the infrastructure. We are strong believer of the philosophy Bigger problems are always solved by breaking them into smaller manageable problems. Resonating with microservices architecture, it is considered best-practice to run database, cluster, storage in smaller connected yet manageable pieces within the infrastructure.

This module is basically combination of Terraform open source and includes automatation tests and examples. It also helps to create and improve your infrastructure with minimalistic code instead of maintaining the whole infrastructure code yourself.

We have fifty plus terraform modules. A few of them are comepleted and are available for open source usage while a few others are in progress.

Prerequisites

This module has a few dependencies:

Examples

IMPORTANT: Since the master branch used in source varies based on new modifications, we suggest that you use the release versions here.

Here are examples of how you can use this module in your inventory structure:

Multi Node non vpc

  module "elasticsearch" {
  source      = "clouddrove/elasticsearch/aws"

  name        = "es"
  environment = "test"
  label_order = ["name", "environment"]
  domain_name = "clouddrove"

  #IAM
  enable_iam_service_linked_role = false
  iam_actions                    = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]

  #Networking
  vpc_enabled             = false
  availability_zone_count = 2
  zone_awareness_enabled  = true
  allowed_cidr_blocks     = ["51.79.69.69"]


  #ES
  elasticsearch_version = "7.8"
  instance_type         = "c5.large.elasticsearch"
  instance_count        = 2

  # Volumes
  volume_size = 30
  volume_type = "gp2"

  #DNS
  dns_enabled     = false
  es_hostname     = "es"
  kibana_hostname = "kibana"
  dns_zone_id     = false

  advanced_options = {
  "rest.action.multi.allow_explicit_index" = "true"
  }

  #Cognito
  cognito_enabled  = false
  user_pool_id     = ""
  identity_pool_id = ""

  #logs
  log_publishing_index_enabled       = true
  log_publishing_search_enabled      = true
  log_publishing_application_enabled = true
  log_publishing_audit_enabled       = false
    }
  }

Multi Node vpc

    module "elasticsearch" {
    source      = "clouddrove/elasticsearch/aws"

    name        = "es"
    environment = "test"
    label_order = ["name", "environment"]
    domain_name = "clouddrove"

    #IAM
    enable_iam_service_linked_role = false
    iam_actions                    = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]

    #Networking
    vpc_enabled             = true
    security_group_ids      = [module.security_group.security_group_ids]
    subnet_ids              = tolist(module.public_subnets.public_subnet_id)
    availability_zone_count = length(module.public_subnets.public_subnet_id)
    zone_awareness_enabled  = true


    #ES
    elasticsearch_version = "7.8"
    instance_type         = "c5.large.elasticsearch"
    instance_count        = 2

    # Volumes
    volume_size = 30
    volume_type = "gp2"

    #DNS
    dns_enabled     = false
    es_hostname     = "es"
    kibana_hostname = "kibana"
    dns_zone_id     = false

    advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
    }

    #Cognito
    cognito_enabled  = false
    user_pool_id     = ""
    identity_pool_id = ""

    #logs
    log_publishing_index_enabled       = true
    log_publishing_search_enabled      = true
    log_publishing_application_enabled = true
    log_publishing_audit_enabled       = false
   }

Single Node Non vpc

   module "elasticsearch" {
   source   = "clouddrove/elasticsearch/aws"

   name        = "es"
   environment = "test"
   label_order = ["name", "environment"]

   #IAM

   enable_iam_service_linked_role = false
   iam_actions                    = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]

   #Networking

   vpc_enabled         = false
   allowed_cidr_blocks = ["51.79.69.69"]

   #Es
   elasticsearch_version = "7.8"
   instance_type         = "c5.large.elasticsearch"
   instance_count        = 1

  #Volume
  volume_size = 30
  volume_type = "gp2"

  #Logs
  log_publishing_application_enabled             = true
  log_publishing_search_cloudwatch_log_group_arn = true
  log_publishing_index_cloudwatch_log_group_arn  = true

  #Cognito
  cognito_enabled  = false
  user_pool_id     = ""
  identity_pool_id = ""

  #DNS
  kibana_hostname = "kibana"
  dns_zone_id     = "Z1XJD7SSBKXLC1"
  dns_enabled     = false
  es_hostname     = "es"


  advanced_options = {
  "rest.action.multi.allow_explicit_index" = "true"
  }

  enforce_https       = true
  tls_security_policy = "Policy-Min-TLS-1-0-2019-07"
  public_enabled      = false

  }

Single Node vpc

  module "elasticsearch" {
  source = "clouddrove/elasticsearch/aws"

  name        = "es"
  environment = "test"
  label_order = ["name", "environment"]

  #IAM
  enable_iam_service_linked_role = false
  iam_actions                    = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]

  #Networking

  vpc_enabled        = true
  security_group_ids = [module.security_group.security_group_ids]
  subnet_ids         = tolist(module.public_subnets.public_subnet_id)


  #Es
  elasticsearch_version = "7.8"
  instance_type         = "c5.large.elasticsearch"
  instance_count        = 1

  #Volume
  volume_size = 30
  volume_type = "gp2"

  #Logs
  log_publishing_application_enabled             = true
  log_publishing_search_cloudwatch_log_group_arn = true
  log_publishing_index_cloudwatch_log_group_arn  = true

  #Cognito
  cognito_enabled  = false
  user_pool_id     = ""
  identity_pool_id = ""

  #DNS
  kibana_hostname = "kibana"
  dns_zone_id     = "Z1XJD7SSBKXLC1"
  dns_enabled     = false
  es_hostname     = "es"

  advanced_options = {
  "rest.action.multi.allow_explicit_index" = "true"
  }

  enforce_https       = true
  tls_security_policy = "Policy-Min-TLS-1-0-2019-07"
  public_enabled      = false

  }

Note: There are some type of instances which not support encryption and EBS option, Please read about this here. Also, there are some limitation for instance type, Please read here

Inputs

Name Description Type Default Required
advanced_options Key-value string pairs to specify advanced configuration options. map(string) {} no
advanced_security_options_enabled AWS Elasticsearch Kibana enchanced security plugin enabling (forces new resource) bool false no
advanced_security_options_internal_user_database_enabled Whether to enable or not internal Kibana user database for ELK OpenDistro security plugin bool false no
advanced_security_options_master_user_arn ARN of IAM user who is to be mapped to be Kibana master user (applicable if advanced_security_options_internal_user_database_enabled set to false) string "" no
advanced_security_options_master_user_name Master user username (applicable if advanced_security_options_internal_user_database_enabled set to true) string "" no
advanced_security_options_master_user_password Master user password (applicable if advanced_security_options_internal_user_database_enabled set to true) string "" no
allowed_cidr_blocks List of CIDR blocks to be allowed to connect to the cluster list(string) [] no
attributes Additional attributes (e.g. 1). list(any) [] no
auto_tune_desired_state Desired state of Auto-Tune for the domain. Valid values are ENABLED, DISABLED. string "DISABLED" no
automated_snapshot_start_hour Hour at which automated snapshots are taken, in UTC. number 0 no
availability_zone_count Number of Availability Zones for the domain to use. number 2 no
cloudwatch_kms_key_id The KMS key ID to encrypt the Cloudwatch logs. string "" no
cognito_enabled Set to false to prevent enable cognito. bool true no
custom_endpoint Fully qualified domain for custom endpoint. string "" no
custom_endpoint_certificate_arn ACM certificate ARN for custom endpoint. string "" no
custom_endpoint_enabled Whether to enable custom endpoint for the Elasticsearch domain. bool false no
dedicated_master_count Number of dedicated master nodes in the cluster. number 0 no
dedicated_master_enabled Indicates whether dedicated master nodes are enabled for the cluster. bool false no
dedicated_master_type Instance type of the dedicated master nodes in the cluster. string "t2.small.elasticsearch" no
dns_enabled Flag to control the dns_enable. bool false no
dns_zone_id Route53 DNS Zone ID to add hostname records for Elasticsearch domain and Kibana. string "" no
domain_name Domain name. string "" no
elasticsearch_version Version of Elasticsearch to deploy. string "6.5" no
enable_iam_service_linked_role Whether to enabled service linked with role. bool false no
enable_logs enable logs bool true no
enabled Set to false to prevent the module from creating any resources. bool true no
encrypt_at_rest_enabled Whether to enable encryption at rest. bool true no
encryption_enabled Whether to enable node-to-node encryption. bool true no
enforce_https Whether or not to require HTTPS. bool true no
environment Environment (e.g. prod, dev, staging). string "" no
es_hostname The Host name of elasticserch. string "" no
iam_actions List of actions to allow for the IAM roles, e.g. es:ESHttpGet, es:ESHttpPut, es:ESHttpPost. list(string) [] no
identity_pool_id ID of the Cognito Identity Pool to use. string "" no
instance_count Number of data nodes in the cluster. number 4 no
instance_type Elasticsearch instance type for data nodes in the cluster. string "t2.small.elasticsearch" no
iops The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Applicable only for the Provisioned IOPS EBS volume type. number 0 no
kibana_hostname The Host name of kibana. string "" no
kms_key_id The KMS key ID to encrypt the Elasticsearch domain with. If not specified, then it defaults to using the AWS/Elasticsearch service KMS key. string "" no
label_order Label order, e.g. name,application. list(any) [] no
log_publishing_application_enabled Specifies whether log publishing option for ES_APPLICATION_LOGS is enabled or not. bool false no
log_publishing_audit_enabled Specifies whether log publishing option for AUDIT_LOGS is enabled or not. bool true no
log_publishing_index_enabled Specifies whether log publishing option for INDEX_SLOW_LOGS is enabled or not. bool false no
log_publishing_search_enabled Specifies whether log publishing option for SEARCH_SLOW_LOGS is enabled or not. bool false no
managedby ManagedBy, eg 'CloudDrove'. string "[email protected]" no
name Name (e.g. app or cluster). string "" no
repository Terraform current module repo string "https://github.com/clouddrove/terraform-aws-elasticsearch" no
retention_in_days Days of retention of cloudwatch. number 90 no
rollback_on_disable Whether to roll back to default Auto-Tune settings when disabling Auto-Tune. Valid values: DEFAULT_ROLLBACK or NO_ROLLBACK. string "DEFAULT_ROLLBACK" no
security_group_ids Security Group IDs. list(string) [] no
subnet_ids Subnet IDs. list(string) [] no
tls_security_policy The name of the TLS security policy that needs to be applied to the HTTPS endpoint. string "Policy-Min-TLS-1-0-2019-07" no
ttl The TTL of the record to add to the DNS zone to complete certificate validation. string "300" no
type Type of DNS records to create. string "CNAME" no
user_pool_id ID of the Cognito User Pool to use. string "" no
volume_size EBS volumes for data storage in GB. number 0 no
volume_type Storage type of EBS volumes. string "gp2" no
vpc_enabled Set to false if ES should be deployed outside of VPC. bool true no
warm_count Number of UltraWarm nodes number 2 no
warm_enabled Whether AWS UltraWarm is enabled bool false no
warm_type Type of UltraWarm nodes string "ultrawarm1.medium.elasticsearch" no
zone_awareness_enabled Enable zone awareness for Elasticsearch cluster. bool false no

Outputs

Name Description
domain_arn ARN of the Elasticsearch domain.
tags A mapping of tags to assign to the resource.

Testing

In this module testing is performed with terratest and it creates a small piece of infrastructure, matches the output like ARN, ID and Tags name etc and destroy infrastructure in your AWS account. This testing is written in GO, so you need a GO environment in your system.

You need to run the following command in the testing folder:

  go test -run Test

Feedback

If you come accross a bug or have any feedback, please log it in our issue tracker, or feel free to drop us an email at [email protected].

If you have found it worth your time, go ahead and give us a ★ on our GitHub!

About us

At CloudDrove, we offer expert guidance, implementation support and services to help organisations accelerate their journey to the cloud. Our services include docker and container orchestration, cloud migration and adoption, infrastructure automation, application modernisation and remediation, and performance engineering.

We are The Cloud Experts!


We ❤️ Open Source and you can check out our other modules to get help with your new Cloud ideas.