About

Terraform libvirt module to provision a node in a distributed minio cluster with tls, optional server side encryption integration (to encrypt data at rest), optional remote automation for cluster-wide minio binary upgrades using ferio and etcd.

For server side encryption integration, the currently supported topology is that each minio server has a kes proxy (that it talks to via encrypted localhost traffic on 127.0.0.1... to protect the data in the event that component other than minio is compromised on any server) that it uses to communicate with an Hashicorp Vault instance.

Usage

Variables

This module takes the following variables as input:

• name: Name to give to the vm.
• vcpus: Number of vcpus to assign to the vm. Defaults to 2.
• memory: Amount of memory in MiB to assign to the vm. Defaults to 8192.
• volume_id: Id of the os image volume to attach to the vm. A recent version of ubuntu is recommended as this is what this module has been validated against.
• data_disks: List of disks to mount to the vm. Each disk entry has the following properties...
  • volume_id: Id of a pre-existing libvirt volume that should be used for the disk. If defined, block_device should be an empty string.
  • block_device: Path to a block device on the vm's host that should be used for the disk. If defined, volume_id should be an empty string.
  • device_name: Device name the disk will have in the vm
  • mount_label: Label for the disk
  • mount_path: Path the disk will be mounted at in the vm
• libvirt_networks: Parameters to connect to libvirt networks. It is an array of objects, each having the following keys:
  • network_name: Name of the libvirt network to connect to (in which case network_id should be an empty string).
  • network_id: Id (ie, uuid) of the libvirt network to connect to (in which case network_name should be an empty string).
  • prefix_length: Length of the network prefix for the network the interface will be connected to. For a 192.168.1.0/24 for example, this would be 24.
  • ip: Ip of interface connecting to the libvirt network.
  • mac: Mac address of interface connecting to the libvirt network.
  • gateway: Ip of the network's gateway. Usually the gateway the first assignable address of a libvirt's network.
  • dns_servers: Dns servers to use. Usually the dns server is first assignable address of a libvirt's network.
• macvtap_interfaces: List of macvtap interfaces to connect the vm to. Note that etcd will only bind on and listen on the first interface (be it macvtap or libvirt network) on its list. Each entry in the list is a map with the following keys:
  • interface: Host network interface that you plan to connect your macvtap interface with.
  • prefix_length: Length of the network prefix for the network the interface will be connected to. For a 192.168.1.0/24 for example, this would be 24.
  • ip: Ip associated with the macvtap interface.
  • mac: Mac address associated with the macvtap interface
  • gateway: Ip of the network's gateway for the network the interface will be connected to.
  • dns_servers: Dns servers for the network the interface will be connected to. If there aren't dns servers setup for the network your vm will connect to, the ip of external dns servers accessible from the network will work as well.
• cloud_init_volume_pool: Name of the volume pool that will contain the cloud-init volume of the vm.
• cloud_init_volume_name: Name of the cloud-init volume that will be generated by the module for your vm. If left empty, it will default to -cloud-init.iso.
• ssh_admin_user: Username of the default sudo user in the image. Defaults to ubuntu.
• admin_user_password: Optional password for the default sudo user of the image. Note that this will not enable ssh password connections, but it will allow you to log into the vm from the host using the virsh console command.
• ssh_admin_public_key: Public part of the ssh key the admin will be able to login as
• minio_server: Paramaters of the minio server for handling external traffic. With the potential exception of the tls key/certificate, these parameters should be the same for all servers in a cluster. The parameters are...
  • tls: Tls configuration for minio. It takes the following parameters...
    • ca_cert: CA certificate that minio will use to authentify other minio servers in the cluster.
    • server_cert: Certificate minio will use to authentify itself to clients.
    • server_key: Private key minio will use to authentify itself to clients.
  • auth: Configuration for minio's root account. It takes the following parameters...
    • root_username: Username of the root user
    • root_password: Password of the root user
  • api_url: Fully qualified (with http protocol and port) url of an external load balancer or domain pointing to all minio instances, which the minio browser console will use to reference the minio api.
  • console_url: Fully qualified (with http protocol and port) url of the external load balancer or domain pointing to all minio instances, which the minio api will use to redirect a browser request to the browser console.
  • console_url: Url of the external load bala
• sse: Parameters for server side encryption so that buckets can encrypted at rest. It takes the following parameters...
  • enabled: Whether encryption at rest is enabled.
  • server: Parameters for the kes proxy. It takes the following arguments...
    • tls: Tls parameters for traffic between the kes server and minio over 127.0.0.1. It takes the following parameters...
      • ca_cert: CA cert used by kes and minio to authentify each other
      • server_key: Private key of the kes server to authentify itself to minio.
      • server_cert: Certificate of the kes server to authentify itself to minio. It should include the 127.0.0.1 ip.
      • client_key: Client key that minio uses to authentify itself to kes
      • client_cert: Client certificate that minio uses to authentify itself to kes
    • cache_expiry: How long the kes proxy should cache a key fetched from Vault. See: https://min.io/docs/kes/tutorials/configuration/#cache-configuration
    • audit_logs: Whether to enable logs for some of the api calls kes receives or not.
  • vault: Configuration for kes to talk to Vault. It takes the following parameters...
    • endpoint: Url of the vault server.
    • mount: Mount where vault stores kes secrets.
    • kv_version: KV version to use for the above mount (can be v1 or v2)
    • prefix: Path prefix to use to access kes secrets if it doesn't have a dedicated mount, otherwise it can be the empty string to use the entire mount.
    • approle: Configuration for the approle kes will use to authentify itself to vault. It takes the following parameters...
      • mount: Mount of the approle
      • id: Identifier of the approle
      • secret: Secret used to authentify that kes can use the approle
      • retry_interval: Retry interval if authentication to vault fails
    • ca_cert: CA certificate kes uses to authentify the vault server
    • ping_interval: Interval at which kes should ping vault to ensure it is alive
• prometheus_auth_type: Authentication mode for prometheus scraping endpoints. Defaults to jwt.
• godebug_settings: Comma-separated list of settings for environment variable GODEBUG. Defaults to empty.
• minio_os_uid: Uid that will be assigned to the minio user for the os. Defaults to 999. This is needed to reprovision vms with pre-existing data volumes reliably without incurring delays chowing a lot of files.
• server_pools: List of server pools. This is used for non-ferio deployments. Each entry takes the following fields:
  • domain_template: Template for the domains of the servers in the pool, which will be expanded with the servers_count_begin and servers_count_end values. Here is an example. For more details, see distributed deployments in minio's documentation.
  • servers_count_begin: Numerical value that will be put in the domain template for the first valid domain in the pool.
  • servers_count_end: Numerical value that will be put in the domain template for the last valid domain in the pool.
  • mount_path_template: Template for the volume paths of each server in the pool (they should be identical), which is assumed to start with 1 and end with mounts_count. Here is an example. For more details, see server pools in minio's documentation.
  • mounts_count: Numerical value that will be put in the mount path template for the last valid path of the mounted minio disks.
• minio_download_url: Url where the minio binary can be downloaded. This is only used for non-ferio deployments. Defaults to a recent minio release.
• ferio: Parameters to setup ferio on the vm. If ferio is not used, it can be omitted. It takes the following parameters...
  • config_prefix: Etcd key prefix where ferio will look for specified minio configurations.
  • workspace_prefix: Etcd key prefix where the ferio instances in the minio cluster will write to synchronize with each other and to keep track of their progress during updates.
  • endpoints: List of endpoints of the etcd cluster ferio will talk to. Each endpoint should take an <ip>:<port> format.
  • auth: Authentification parameters to talk to the etcd cluster in a secure way. It takes the following parameters...
    • ca_cert: CA certificate ferio will use to authentify the etcd servers it talks to.
    • client_cert: Client certificate that will authentify ferio to the etcd servers if certificate authentication is used.
    • client_key: Client private key that will authentify ferio to the etcd servers if certificate authentication is used.
    • username: Client username that will authentify ferio to the etcd servers if password authentication is used.
    • password: Client password that will authentify ferio to the etcd servers if password authentication is used.
• chrony: Optional chrony configuration for when you need a more fine-grained ntp setup on your vm. It is an object with the following fields:
  • enabled: If set the false (the default), chrony will not be installed and the vm ntp settings will be left to default.
  • servers: List of ntp servers to sync from with each entry containing two properties, url and options (see: https://chrony.tuxfamily.org/doc/4.2/chrony.conf.html#server)
  • pools: A list of ntp server pools to sync from with each entry containing two properties, url and options (see: https://chrony.tuxfamily.org/doc/4.2/chrony.conf.html#pool)
  • makestep: An object containing remedial instructions if the clock of the vm is significantly out of sync at startup. It is an object containing two properties, threshold and limit (see: https://chrony.tuxfamily.org/doc/4.2/chrony.conf.html#makestep)
• fluentbit: Optional fluent-bit configuration to securely route logs to a fluend/fluent-bit node using the forward plugin. Alternatively, configuration can be 100% dynamic by specifying the parameters of an etcd store or git repo to fetch the configuration from. It has the following keys:
  • enabled: If set the false (the default), fluent-bit will not be installed.
  • minio_tag: Tag to assign to logs coming from the minio process.
  • kes_tag: Tag to assign to logs coming from the kes process.
  • ferio_tag: Tag to assign to logs coming from the ferio process.
  • node_exporter_tag Tag to assign to logs coming from the prometheus node exporter
  • forward: Configuration for the forward plugin that will talk to the external fluend/fluent-bit node. It has the following keys:
    • domain: Ip or domain name of the remote fluend node.
    • port: Port the remote fluend node listens on
    • hostname: Unique hostname identifier for the vm
    • shared_key: Secret shared key with the remote fluentd node to authentify the client
    • ca_cert: CA certificate that signed the remote fluentd node's server certificate (used to authentify it)
• fluentbit_dynamic_config: Optional configuration to update fluent-bit configuration dynamically either from an etcd key prefix or a path in a git repo.
  • enabled: Boolean flag to indicate whether dynamic configuration is enabled at all. If set to true, configurations will be set dynamically. The default configurations can still be referenced as needed by the dynamic configuration. They are at the following paths:
    • Global Service Configs: /etc/fluent-bit-customization/default-config/service.conf
    • Default Variables: /etc/fluent-bit-customization/default-config/default-variables.conf
    • Systemd Inputs: /etc/fluent-bit-customization/default-config/inputs.conf
    • Forward Output For All Inputs: /etc/fluent-bit-customization/default-config/output-all.conf
    • Forward Output For Default Inputs Only: /etc/fluent-bit-customization/default-config/output-default-sources.conf
  • source: Indicates the source of the dynamic config. Can be either etcd or git.
  • etcd: Parameters to fetch fluent-bit configurations dynamically from an etcd cluster. It has the following keys:
    • key_prefix: Etcd key prefix to search for fluent-bit configuration
    • endpoints: Endpoints of the etcd cluster. Endpoints should have the format <ip>:<port>
    • ca_certificate: CA certificate against which the server certificates of the etcd cluster will be verified for authenticity
    • client: Client authentication. It takes the following keys:
      • certificate: Client tls certificate to authentify with. To be used for certificate authentication.
      • key: Client private tls key to authentify with. To be used for certificate authentication.
      • username: Client's username. To be used for username/password authentication.
      • password: Client's password. To be used for username/password authentication.
  • git: Parameters to fetch fluent-bit configurations dynamically from an git repo. It has the following keys:
    • repo: Url of the git repository. It should have the ssh format.
    • ref: Git reference (usually branch) to checkout in the repository
    • path: Path to sync from in the git repository. If the empty string is passed, syncing will happen from the root of the repository.
    • trusted_gpg_keys: List of trusted gpp keys to verify the signature of the top commit. If an empty list is passed, the commit signature will not be verified.
    • auth: Authentication to the git server. It should have the following keys:
      • client_ssh_key Private client ssh key to authentication to the server.
      • server_ssh_fingerprint: Public ssh fingerprint of the server that will be used to authentify it.
• install_dependencies: Whether cloud-init should install external dependencies (should be set to false if you already provide an image with the external dependencies built-in).

Example

See: https://github.com/Ferlab-Ste-Justine/kvm-dev-orchestrations/tree/main/minio