Artifact Storage

[p5]

Summary

Add new functionality to Terragrunt so that module outputs, local variables and arbitrary string data (I'll collectively refer to these as artifacts until a better name comes around) can be stored in an IaC-agnostic location. This will allow other tools and services to consume these values directly without requiring the user to bake this into their module.

This location would start off simple, storing these artifacts inside AWS Systems Manager Parameter Store (SSM), and could evolve into DNS, S3 JSON objects etc.

Motivation

If Terraform/OpenTofu/Terragrunt is your only IaC tool, passing data between different pieces of infrastructure is fairly straight forward - either use the Terraform remote_state datasource or have it all abstracted away from you through Terragrunt dependencies.

When you inevitably want to start integrating Terraform resources with other IaC frameworks, such as Serverless Framework or AWS CDK, sharing resource IDs or URLs become tricky.

In the past, I've opted for one of two options - either wrap the third-party Terraform module inside my own to store everything I need inside aws_ssm_parameter resources, or manually create these SSM parameters and remember to update them each time a resource is updated.

The goal of this proposal is to allow a seamless integration and data sharing between Terragrunt-managed resources and other Infrastructure as Code frameworks, initially through the use of SSM parameters.

Proposal

Introduce a new top-level configuration named "artifacts". This "artifacts" config contains options to choose the storage backend and define the data to be stored in those artifacts. These artifact values can be fetched from a local Terragrunt variable, module outputs or arbitrary strings defined inline.

This proposal addresses the motivation outlined above by acting as a native way to store data in a location where most other IaC frameworks are able to read from. CloudFormation, Serverless Framework, Pulumni and CDK all directly support consuming data from SSM parameters.

The following is an example of the interface that may be exposed to a user:

artifacts {
	backend = "ssm"
	data = {
		"/SHARED/ACCOUNT/ID"	= local.account_id		// Terragrunt local variable
		"/SHARED/VPC/ID"		= output.vpc_id			// Module output
		"/SHARED/VPC/NAME"		= "arbitrary string"	// Inline strings
	}
}

Although I don't believe this should be a requirement, other Terragrunt units would be able to reference this data through the typical dependency blocks, under an object named "artifacts". I.E. dependency.vpc.artifacts.data["/SHARED/ACCOUNT/ID"].

Technical Details

Introduce a new top-level block - artifacts.
This artifacts block will start off containing the following configuration items:

• backend - for selecting between SSM, DNS, S3 etc
• data - to define each artifact's location and data contained within
• ssm_XYZ - allows the user to customize the SSM location, such as account, region etc.

Over time, more backends can be supported, however SSM seems to be the most popular currently.

This configuration block would be evaluated after Terragrunt has finished a plan or apply step, and will make the necessary SDK calls or generate the Terraform resources required to create the artifacts.

Press Release

Terragrunt now supports sharing data with other frameworks!

Using the new artifacts block, users can define framework-agnostic locations to store data from your Terragrunt units. A VPC module deployed through Terragrunt could store the VPC and subnet IDs as artifacts to allow alternative IaC frameworks to consume this information.

Drawbacks

No response

Alternatives

No response

Migration Strategy

No response

Unresolved Questions

No response

References

No response

Proof of Concept Pull Request

No response

Support Level

• I have Terragrunt Enterprise Support
• I am a paying Gruntwork customer
• Previously a paying Gruntwork customer until belts tightened

Customer Name

No response

[yhakbar]

We largely stick to American spelling in this project, AFAIK. Calling it "Artifacts" would be more appropriate.