Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: extra_templates allows users to control templates #1046

Merged
merged 19 commits into from
Apr 29, 2024

Conversation

jakeyheath
Copy link
Contributor

@jakeyheath jakeyheath commented Apr 26, 2024

Summary

This PR adds a feature to fogg.yml that allows users to add custom golang templates to components. Users will have full access to the plan's struct in the template, allowing them to write configuration files of their own without changing the fogg code base. This is handy when we have repeatable stacks for a specific application or environment, but don't necessarily need it to be hardcoded into the fogg codebase. We can think of these a mini fogg extensions.

You can provide a set of files to any component (env, account, global, default,...) and you can also specify if fogg should overwrite a file that already exists with the overwrite option (defaults to false). Setting overwrite to true would allow someone to scaffold and template an infra stack, but still be allowed to manage the component's infrastructure normally without worrying about fogg overwritting the changes.

Here's an example of how to call it:

envs:
  dev:
    eks:
      extra_templates:
      - overwrite: true
        files:
            test.txt: |
              blah1
            variables.tf: | 
              locals {
                owner_roles = [
                  "poweruser",
                  "okta-czi-admin",
                  "tfe-si",
                ]

                env = {{.Env | quote}}
                tags            = var.tags
                vpc_id          = data.terraform_remote_state.cloud-env.outputs.vpc_id
                subnet_ids      = data.terraform_remote_state.cloud-env.outputs.private_subnets

                node_groups = {
                  "arm" = {
                    size = 1
                    architecture = {
                      ami_type       = "AL2_ARM_64"
                      instance_types = ["t4g.xlarge"]
                    }
                  },
                }
                authorized_github_repos = {
                  chanzuckerberg = [var.github_repo]
                }
                addons = {
                  enable_aws_for_fluentbit = false
                  enable_guardduty         = false
                }
              }
      depends_on:
        components:
          - cloud-env
      module_source: [email protected]:chanzuckerberg/shared-infra//terraform/modules/eks-cluster-v2?ref=eks-cluster-v2-v6.9.1
      variables:
        - cluster_name=var.cluster_name
        - cluster_version=var.cluster_version
        - owner_roles
        - node_groups
        - authorized_github_repos
        - addons
      provider_aliases:
        aws.us-east-1: aws.us-east-1
      providers:
        aws:
          additional_regions:
            - us-east-1
        datadog:
          version: 3.20.0
        kubernetes:
          enabled: true
          version: 2.23.0
          cluster_component_name: eks-cluster-v2
          custom_provider: false
        kubectl:
          enabled: true
          version: 1.14.0
        helm:
          enabled: true
          version: 2.11.0
        okta:
          custom_provider: true
          version: 3.41.0
          registry_namespace: okta
          org_name: czi-prod

In addition to the normal stuff fogg creates, it would also produce these files:

terraform/envs/dev/eks/variables.tf

locals {
  owner_roles = [
    "poweruser",
    "okta-czi-admin",
    "tfe-si",
  ]
  env        = "virtual-cell-dev"
  tags       = var.tags
  vpc_id     = data.terraform_remote_state.cloud-env.outputs.vpc_id
  subnet_ids = data.terraform_remote_state.cloud-env.outputs.private_subnets
  node_groups = {
    "arm" = {
      size = 1
      architecture = {
        ami_type       = "AL2_ARM_64"
        instance_types = ["t4g.xlarge"]
      }
    },
  }
  authorized_github_repos = {
    chanzuckerberg = [var.github_repo]
  }
  addons = {
    enable_aws_for_fluentbit = false
    enable_guardduty         = false
  }
}

terraform/envs/dev/eks/test.txt

blah1

Notice how instead of relying on env = var.tags.env, we could code-generate the local variable env with env = {{.Env | quote}}, since the template has access to the full component plan.

@jakeyheath jakeyheath requested a review from a team as a code owner April 26, 2024 23:32
@jakeyheath jakeyheath requested review from beroy, alldoami, jjb007, alexlokshin-czi and hspitzley-czi and removed request for a team and beroy April 26, 2024 23:32
config/v2/config.go Outdated Show resolved Hide resolved
Comment on lines 295 to 302
templates := []ExtraTemplate{}
for _, v := range *comm.ExtraTemplates {
templates = append(templates, ExtraTemplate{
Overwrite: v.Overwrite,
Files: v.Files,
})
}
return &templates
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential opportunity to make a helper to consolidate same mapping functionality (1)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right, but unfortunately, the way this codebase's packages are laid out makes this difficult without doing a bit of refactoring. The config stage structs are all pointers because the configuration can or cannot have values and then they duplicate these same structs in the plan phase once they are "resolved" (going down the tree of overwrites from global -> default -> account -> env -> component) through the tree overwrites. Resolved structs don't have pointers. I would love to simplify this code since its so copy/pasted, but maybe I will try that in a different PR

plan/plan.go Outdated
Comment on lines 1263 to 1269
extraTemplates := []ExtraTemplate{}
for _, v := range v2.ResolveExtraTemplates(commons...) {
extraTemplates = append(extraTemplates, ExtraTemplate{
Files: v.Files,
Overwrite: v.Overwrite,
})
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential opportunity to make a helper to consolidate same mapping functionality (2)

@jakeyheath jakeyheath merged commit f13687a into main Apr 29, 2024
5 checks passed
@jakeyheath jakeyheath deleted the heathj/extra_files branch April 29, 2024 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants