[Enhancement]: aws_ecr_repository data source support for external registries such as public.ecr.aws

[acdha]

Description

The AWS ECS service team introduced a regression where they started forcing container image ID pinning. This means that jobs using public containers like cloudwatch-agent/cloudwatch-agent or xray/aws-xray-daemon will fail to launch after the upstream image tags are updated, which means that it would be really useful to be able to use something like data.aws_ecr_repository.xray.most_recent_image_tags tags in Terraform rather than setting up for future deployment failures when the latest tag changes.

This is also complicated because the aws_ecr_repository resource does not implement all of the attributes which the data source provides, so if you want to manage this currently in Terraform you have to do the following:

1. Enable pull-through caching (e.g. “ecr-public”)
2. Create an aws_ecr_repository resource with the target container (e.g. "ecr-public/cloudwatch-agent/cloudwatch-agent") to ensure that Terraform won't fail due to errors before the container image has been pulled at least once.
3. Create an aws_ecr_repository data source to get the most recent image tags
4. Filter the desired tags

Affected Resource(s) and/or Data Source(s)

• aws_ecr_repository

Potential Terraform Configuration

data "aws_ecr_repository" "xray-daemon" {
    name = "cloudwatch-agent/cloudwatch-agent"
    repository_url = "public.ecr.aws"
}

References

#22509 was opened earlier but closed without progress.

There's a separate bug in aws_ecr_repository which causes the most_recent_image_tags attribute not to be populated (#36835) which is also a blocker.

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acdha]

I ended up implementing this with use of the Docker provider. It's not a great amount of code but it does work and allows arbitrarily-complex filtering. Unfortunately, it does require you to have a configured Docker host to work around kreuzwerker/terraform-provider-docker#634

data "http" "public_ecr_gallery_versions" {
  for_each = {
    cloudwatch = {
      registryAliasName = "cloudwatch-agent",
      repositoryName    = "cloudwatch-agent"
    },
    xray = {
      registryAliasName = "xray",
      repositoryName    = "aws-xray-daemon"
    },
  }

  url    = "https://api.us-east-1.gallery.ecr.aws/describeImageTags"
  method = "POST"

  request_headers = {
    Accept       = "application/json"
    Content-Type = "application/json"
  }

  request_body = jsonencode({
    registryAliasName = each.value["registryAliasName"]
    repositoryName    = each.value["repositoryName"]
  })

  lifecycle {
    postcondition {
      condition     = self.status_code == 200
      error_message = "Unexpected HTTP ${self.status_code} for ${self.url}"
    }
  }
}

data "docker_registry_image" "cloudwatch_latest" {
  name = "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest"
}

data "docker_registry_image" "xray_latest" {
  name = "public.ecr.aws/xray/aws-xray-daemon:latest"
}

locals {
  # We need to get immutable versions so we have to filter each repo following its policy.

  # The CloudWatch agent team publishes tags like "latest-amd64",
  # "1.247355.0b252062-arm64", "1.300034.1b536", and "stable-amd64"
  cloudwatch_versions = {
    for i in jsondecode(data.http.public_ecr_gallery_versions["cloudwatch"].response_body)["imageTagDetails"] :
    i["imageDetail"]["imageDigest"] => i["imageTag"] if length(regexall("^1[.].*", i["imageTag"])) > 0
  }
  cloudwatch_latest_version = local.cloudwatch_versions[data.docker_registry_image.cloudwatch_latest.sha256_digest]

  # The X-Ray team publishes versions like "3.3.13", "3.x", "alpha", and "latest":
  xray_versions = {
    for i in jsondecode(data.http.public_ecr_gallery_versions["xray"].response_body)["imageTagDetails"] :
    i["imageDetail"]["imageDigest"] => i["imageTag"] if length(regexall("^3[.][0-9]+", i["imageTag"])) > 0
  }
  xray_latest_version = local.xray_versions[data.docker_registry_image.xray_latest.sha256_digest]
}