Dependency optimization broken since version 0.55.4 #3441

mickeder · 2024-09-26T08:35:59Z

Describe the bug

Recently I've noticed that newer versions of Terragrunt run very slow when there are many recursive dependencies. This was not the case in the past, and I pinpointed the version where it changed to 0.55.4 - the introduction of Reference to dependency inputs.

According to the documentation about dependency block, there is a way to fetch only the outputs of the lowest level dependencies by using remote_state block. This way the outputs are not fetched from any higher level dependencies.

Unfortunately, this behavior was broken with the introduction of referencing dependency inputs in version 0.55.4. Currently all the outputs are fetched recursively for all dependency levels, which takes a long time when the dependency graph is complex but in reality only the lowest level dependency output is required for the root module.

Steps To Reproduce

Given below minimum configuration structure, I was able to reproduce the improper behavior.

# .
# ├── terragrunt.hcl
# ├── module-a
# │  ├── main.tf
# │  └── terragrunt.hcl
# ├── module-b
# │  ├── main.tf
# │  └── terragrunt.hcl
# └── module-c
#     ├── main.tf
#     └── terragrunt.hcl


# terragrunt.hcl
remote_state {
  backend = "local"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    path = "terraform.tfstate"
  }
}


# module-a/main.tf
output "test_output" {
  value = "hello from module-b"
}


# module-a/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}


# module-b/main.tf
variable "test_var" {
  type = string
}

output "test_output" {
  value = "hello from module-b"
}


# module-b/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}

dependency "module_a" {
  config_path = "../module-a"
}

inputs = {
  test_var = dependency.module_a.outputs.test_output
}



# module-c/main.tf
variable "test_var" {
  type = string
}

output "result" {
  value = var.test_var
}


# module-c/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}

dependency "module_b" {
  config_path = "../module-b"
}

inputs = {
  test_var = dependency.module_b.outputs.test_output
}

After applying module-a and module-b, I ran terragrunt plan for module-c with version 0.55.3, which gave the following debug logs:

time=2024-09-26T10:34:51+02:00 level=debug msg=Terragrunt Version: 0.55.3
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform --version prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=terraform version: 1.9.5
time=2024-09-26T10:34:51+02:00 level=debug msg=Reading Terragrunt config file at /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/terragrunt.hcl
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Getting output of dependency /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl for config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/terragrunt.hcl
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Detected module /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform output -json prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Retrieved output from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-b"
  }
}
 prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=info msg=Downloading Terraform configurations from file:///home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c into /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=Copying files from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c into /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=Setting working directory to /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=The file path /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Generated file /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform init -no-color prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
Initializing the backend...
Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
time=2024-09-26T10:34:51+02:00 level=debug msg=Copying lock file from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/.terraform.lock.hcl to /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform plan -no-color prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 

Changes to Outputs:
  + result = "hello from module-b"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Then I ran the same command with the newest Terragrunt 0.67.13, and the debug logs clearly show that outputs of module-a were fetched as well, even though they are not required by module-c.

10:33:42.580 DEBUG  Terragrunt Version: 0.67.13
10:33:42.580 INFO   Terragrunt Cache server is listening on 127.0.0.1:36205
10:33:42.580 INFO   Start Terragrunt Cache server
10:33:42.580 DEBUG  Provider cache dir "../../../../../../../.cache/terragrunt/providers"
10:33:42.581 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.581 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.581 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.581 DEBUG  Running command: terraform --version
10:33:42.609 DEBUG  terraform version: 1.9.5
10:33:42.609 DEBUG  Reading Terragrunt config file at ./terragrunt.hcl
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.609 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.610 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.611 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.614 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-b] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.614 DEBUG  [../module-b] Getting output of dependency ../module-a/terragrunt.hcl for config ../module-b/terragrunt.hcl
10:33:42.614 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Detected module ../module-a/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory.
10:33:42.616 DEBUG  [../module-a] Running command: terraform output -json
10:33:42.645 DEBUG  [../module-a] Retrieved output from ../module-a/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-a"
  }
}
10:33:42.645 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.646 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.646 DEBUG  Getting output of dependency ../module-b/terragrunt.hcl for config ./terragrunt.hcl
10:33:42.646 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.647 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.648 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.648 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.648 DEBUG  [../module-b] Detected module ../module-b/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory.
10:33:42.648 DEBUG  [../module-b] Running command: terraform output -json
10:33:42.678 DEBUG  [../module-b] Retrieved output from ../module-b/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-b"
  }
}
10:33:42.679 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.679 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.679 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.679 INFO   Downloading Terraform configurations from . into ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  Copying files from . into ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  Setting working directory to ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  The file path ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.
10:33:42.680 DEBUG  Generated file ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf.
10:33:42.779 DEBUG  Discovered "registry.terraform.io" registry URLs: {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}
10:33:42.940 DEBUG  Discovered "registry.opentofu.org" registry URLs: {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}
10:33:42.941 INFO   Caching terraform providers for ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.941 DEBUG  Running command: terraform init -no-color -no-color
10:33:43.016 STDOUT terraform: Initializing the backend...
10:33:43.016 STDOUT terraform: Initializing provider plugins...
10:33:43.016 STDOUT terraform: Terraform has been successfully initialized!
10:33:43.016 STDOUT terraform: You may now begin working with Terraform. Try running "terraform plan" to see
10:33:43.016 STDOUT terraform: any changes that are required for your infrastructure. All Terraform commands
10:33:43.017 STDOUT terraform: should now work.
10:33:43.017 STDOUT terraform: If you ever set or change modules or backend configuration for Terraform,
10:33:43.017 STDOUT terraform: rerun this command to reinitialize your working directory. If you forget, other
10:33:43.017 STDOUT terraform: commands will detect it and remind you to do so if necessary.
10:33:43.018 DEBUG  Running command: terraform init -no-color
10:33:43.061 STDOUT terraform: Initializing the backend...
10:33:43.061 STDOUT terraform: Initializing provider plugins...
10:33:43.061 STDOUT terraform: Terraform has been successfully initialized!
10:33:43.061 STDOUT terraform: You may now begin working with Terraform. Try running "terraform plan" to see
10:33:43.061 STDOUT terraform: any changes that are required for your infrastructure. All Terraform commands
10:33:43.061 STDOUT terraform: should now work.
10:33:43.062 STDOUT terraform: If you ever set or change modules or backend configuration for Terraform,
10:33:43.062 STDOUT terraform: rerun this command to reinitialize your working directory. If you forget, other
10:33:43.062 STDOUT terraform: commands will detect it and remind you to do so if necessary.
10:33:43.063 DEBUG  Copying lock file from ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/.terraform.lock.hcl to .
10:33:43.063 DEBUG  Running command: terraform plan -no-color
10:33:43.108 STDOUT terraform: Changes to Outputs:
10:33:43.108 STDOUT terraform:   + result = "hello from module-b"
10:33:43.108 STDOUT terraform: You can apply this plan to save these new output values to the Terraform
10:33:43.108 STDOUT terraform: state, without changing any real infrastructure.
10:33:43.108 STDOUT terraform: ─────────────────────────────────────────────────────────────────────────────
10:33:43.108 STDOUT terraform: Note: You didn't use the -out option to save this plan, so Terraform can't
10:33:43.108 STDOUT terraform: guarantee to take exactly these actions if you run "terraform apply" now.
10:33:43.109 INFO   Shutting down Terragrunt Cache server...
10:33:43.109 INFO   Terragrunt Cache server stopped

Expected behavior

Terragrunt should not fetch dependency outputs recursively, but just for the immediate dependency that is required. Basically the behavior from version 0.55.3 should be restored, but taking into consideration the newer feature of referencing dependency inputs as well.

Nice to haves

Terminal output
Screenshots

Versions

Terragrunt version: 0.67.13
OpenTofu/Terraform version: 1.9.5
Environment details (Ubuntu 20.04, Windows 10, etc.): Ubuntu 22.04

Additional context

We use Azure remote_state and dependency graphs as below are not uncommon. This causes the dependency fetching part take several minutes instead of several seconds.

The text was updated successfully, but these errors were encountered:

yhakbar · 2024-09-26T10:24:02Z

Hey @mickeder

Terragrunt does have to fetch dependencies recursively, as ancestor dependencies can influence outputs of immediate parents.

It should be caching the outputs of ancestor dependencies to minimize the performance penalty, however. Have you done a performance analysis to see what the impact has been in the latest version of Terragrunt?

You can take advantage of the OpenTelemetry integration to get more insight on that:
https://terragrunt.gruntwork.io/docs/features/debugging/#opentelemetry-integration

mickeder · 2024-09-26T11:39:26Z

Hi @yhakbar, as I stated in my report, I tested it with the newest version of Terragrunt. It doesn't matter which version I use, after version 0.55.3 outputs of all ancestor dependencies are fetched, even though it was not the case before. The current behavior is also not reflected in the documentation that I mentioned (Can I speed up dependency fetching?), which states:
If these conditions are met, terragrunt will only parse out the remote_state blocks and use that to pull down the state for the target module without parsing the dependency blocks, avoiding the recursive dependency retrieval.

In my opinion, the new feature implemented in version 0.55.4 giving the ability to reference inputs of dependencies broke the previous behavior and was not designed to change this in the first place.

Rumbles · 2024-10-07T12:28:35Z

I've noticed the same performance degradation, a job that used to take 1-2min and build 2 dependencies, now takes 5mins+ and builds 14 dependencies.

Most of my colleagues stopped updating last year and have been using a version around v0.51.7

joshbsemperis · 2024-10-07T12:34:48Z

I have experienced the same issue, some runs that used to take seconds now take many minutes, and then breaks. This is quite a critical bug

stdmje · 2024-10-07T12:43:58Z

+1

deBFM · 2024-10-09T16:37:08Z

@denis256
The performance began to degrade with v0.55.4, but the actual issue, where the dependency fetching optimization quits working, was introduced in v0.55.6.

Proof

When I plan the green module within the following dependency graph and --terragrunt-log-level debug

with v0.55.3
terraform init -get=false was called 1 time (just the direct dependency)
Runtime: 0m20,324s

with v0.55.4
terraform init -get=false was called 1 time (just the direct dependency)
Runtime: 0m34,343s

with v0.55.5
terraform init -get=false was called 1 time (just the direct dependency)
Runtime: 0m33,811s

with v0.55.6
terraform init -get=false was called 7 times (on every module before the actual target module)
Runtime: 1m32,233s

Tested on

OS: Windows 10 Enterprise
Architecture: amd64

A fix would be greatly appreciated, as our production dependency graphs are significantly larger than the ones used in these tests.
Thank you in advance for your help!

gchiesa · 2024-10-14T07:18:10Z

Hello, I would like to ask you if the #3461 can be prioritized since we are also impacted by this performance degradation due to broken optimization for dependencies.

csabakollar · 2024-10-15T11:37:04Z

Hi all, this is also an issue for us. I'm not an expert, so more of an anecdotal experience:
As our terragrunt stack constantly grew in the past months, we noticed longer and longer plan times and as of today, we are running terragrunt run-all plan (with initialisation) in 16m45s.
A pre-initialised plan takes 11m26s.

I tried to download TG v0.55.3, but couldn't find it any longer.

yhakbar · 2024-10-15T15:45:46Z

@csabakollar

https://github.com/gruntwork-io/terragrunt/releases/tag/v0.55.3

FYI, you can find any release of Terragrunt by punching in the value into the URL, even if it doesn't show up in GitHub releases search for whatever reason.

We are actively investigating this issue. Performance improvements require a lot of testing and validation, so please have patience as we make sure this issue is resolved.

csabakollar · 2024-10-17T12:27:02Z

@csabakollar

https://github.com/gruntwork-io/terragrunt/releases/tag/v0.55.3

FYI, you can find any release of Terragrunt by punching in the value into the URL, even if it doesn't show up in GitHub releases search for whatever reason.

We are actively investigating this issue. Performance improvements require a lot of testing and validation, so please have patience as we make sure this issue is resolved.

Thank you, and I promise I searched for the specific version and github search didn't drop this out :\

edit: quick update, for some reason 0.55.3 cannot even produce a stable plan (cannot find included files, which are working with 0.68.0, I believe we have some other issues as well, so maybe this is a mini red herring for us.

denis256 · 2024-10-21T21:40:44Z

Hello,
I prepared a beta release with a fix for this issue, but it will require setting special control flag:

TERRAGRUNT_STRICT_CONTROL="skip-dependencies-inputs"

https://github.com/gruntwork-io/terragrunt/releases/tag/v0.68.5-beta2024102101

Let me know if this approach improve performance on real projects

mickeder · 2024-10-22T09:10:23Z

@denis256 thank you for this fix. I tested it out with our real environments with below command:
TERRAGRUNT_STRICT_CONTROL=skip-dependencies-inputs terragrunt plan
and it works correctly showing a warning

10:58:36.225 INFO   Terragrunt Cache server is listening on 127.0.0.1:40899
10:58:36.225 INFO   Start Terragrunt Cache server
10:58:36.339 WARN   [../xxx] Skipping inputs reading from ../xxx/terragrunt.hcl inputs for better performance
10:58:36.340 WARN   [../xxx] Skipping inputs reading from ../../../terragrunt.hcl inputs for better performance
...
10:59:15.763 STDOUT terraform: No changes. Your infrastructure matches the configuration.
10:59:15.763 STDOUT terraform: Terraform has compared your real infrastructure against your configuration
10:59:15.763 STDOUT terraform: and found no differences, so no changes are needed.
10:59:15.957 INFO   Shutting down Terragrunt Cache server...
10:59:15.957 INFO   Terragrunt Cache server stopped

The time of dependency outputs reading improved a lot, in this example from over 3 minutes to under a minute.

However, I have a reservation to the implementation of the skip-dependencies-inputs flag. According to the current documentation, the strict mode is to be used when Terragrunt should fail if you use a deprecated command (e.g. plan-all).

➜ TERRAGRUNT_STRICT_CONTROL=plan-all terragrunt plan-all
11:01:38.399 INFO   Terragrunt Cache server is listening on 127.0.0.1:33513
11:01:38.399 INFO   Start Terragrunt Cache server
11:01:38.399 INFO   Shutting down Terragrunt Cache server...
11:01:38.399 INFO   Terragrunt Cache server stopped
11:01:38.399 ERROR  The `plan-all` command is no longer supported. Use `terragrunt run-all plan` instead.
11:01:38.399 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Therefore I suggest adding a separate flag --terragrunt-skip-dependencies-inputs outside of the strict mode.

csabakollar · 2024-10-22T09:37:32Z

@denis256 thanks for the fix. I also tested for our stack and now it's sub 7 mins for plan with initialised TG modules. 🥳

gchiesa · 2024-10-24T14:31:14Z

I can confirm the fix improved enormously the performance. Looking forward to see in stable release.

denis256 · 2024-10-28T13:15:53Z

Initial optimization included in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.68.5

Optimization can be enabled with export TERRAGRUNT_STRICT_CONTROL="skip-dependencies-inputs", with plans for future improvements.

mickeder added the bug Something isn't working label Sep 26, 2024

mickeder changed the title ~~Dependencies optimization broken since version 0.55.4~~ Dependency optimization broken since version 0.55.4 Sep 26, 2024

yhakbar added the performance Issues related to performance label Sep 26, 2024

yhakbar assigned denis256 Sep 26, 2024

gchiesa mentioned this issue Oct 4, 2024

Terragrunt with/without cache very slow #3447

Open

denis256 mentioned this issue Oct 9, 2024

Dependency optimisation fix #3461

Merged

4 tasks

denis256 closed this as completed in #3461 Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependency optimization broken since version 0.55.4 #3441

Dependency optimization broken since version 0.55.4 #3441

mickeder commented Sep 26, 2024

yhakbar commented Sep 26, 2024

mickeder commented Sep 26, 2024

Rumbles commented Oct 7, 2024

joshbsemperis commented Oct 7, 2024

stdmje commented Oct 7, 2024

deBFM commented Oct 9, 2024

gchiesa commented Oct 14, 2024

csabakollar commented Oct 15, 2024

yhakbar commented Oct 15, 2024

csabakollar commented Oct 17, 2024 •

edited

Loading

denis256 commented Oct 21, 2024 •

edited

Loading

mickeder commented Oct 22, 2024

csabakollar commented Oct 22, 2024

gchiesa commented Oct 24, 2024

denis256 commented Oct 28, 2024

Dependency optimization broken since version 0.55.4 #3441

Dependency optimization broken since version 0.55.4 #3441

Comments

mickeder commented Sep 26, 2024

Describe the bug

Steps To Reproduce

Expected behavior

Nice to haves

Versions

Additional context

yhakbar commented Sep 26, 2024

mickeder commented Sep 26, 2024

Rumbles commented Oct 7, 2024

joshbsemperis commented Oct 7, 2024

stdmje commented Oct 7, 2024

deBFM commented Oct 9, 2024

Proof

Tested on

gchiesa commented Oct 14, 2024

csabakollar commented Oct 15, 2024

yhakbar commented Oct 15, 2024

csabakollar commented Oct 17, 2024 • edited Loading

denis256 commented Oct 21, 2024 • edited Loading

mickeder commented Oct 22, 2024

csabakollar commented Oct 22, 2024

gchiesa commented Oct 24, 2024

denis256 commented Oct 28, 2024

csabakollar commented Oct 17, 2024 •

edited

Loading

denis256 commented Oct 21, 2024 •

edited

Loading