run-all show -json mangles output for very large plans

[hojerst]

Describe the bug

When running terragrunt run-all show -json for very large plans, the output is mangled and no longer a valid json document. i assume its an issue with a line buffer, as the json from terraform show -json is all printed to a single line. using the normal, colored output the json document is cut after 65536 characters and when using --terragrunt-forward-tf-stdout the output seems to be cut at the beginning.

Steps To Reproduce

1. create a plan which creates a json output of more than 65536 characters when running terraform show -json planfile
2. run terragrunt run-all show -json for the folder containing the plan

Expected behavior

the json output should be complete without any content cut

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: v0.67.3
• OpenTofu/Terraform version: OpenTofu v1.8.2
• Environment details (Ubuntu 20.04, Windows 10, etc.): darwin_arm64

Additional context

Add any other context about the problem here.

[denis256]

Hi,
I tried to set an example that render long output as json in https://github.com/denis256/terragrunt-tests/tree/master/show-json but output looks valid

Can you share an example code/repo where this issue happens?

[hojerst]

i was able to reproduce the error in your test project with a slight change:

cp -a app app2
cp -a app app3
terragrunt run-all plan -out planfile
terragrunt run-all show -json planfile >out.json

looking at out.json, you will get a bunch of garbled json starting from line 2

[levkohimins]

Hi @hojerst, I repeated this

cp -a app app2
cp -a app app3
terragrunt run-all plan -out planfile
terragrunt run-all show -json planfile >out.json

and what I see in out.json

Could you show me what exactly is wrong here?

[hojerst]

unfortunately i don't see how you created this colored output, so i'm not sure which of the 3 planfiles it actually is. in my setup the first plan-json (first line) is completely fine. The second and 3rd plan are cut off however. I can reproduce this by

# print only second line of stdout
terragrunt run-all show -json planfile | sed -n 2p

this will look something like this:

10:06:48.497 INFO   [app] Retrieved output from app/terragrunt.hcl
10:06:48.497 INFO   [app2] Retrieved output from app2/terragrunt.hcl
10:06:48.498 INFO   [app3] Retrieved output from app3/terragrunt.hcl
reateLargeDataBlock_1024ThisIsAVeryLongStringRepeatedManyTimesToCreateLargeDataBlock_1024ThisIsAVeryLongStringRepeatedManyTimesToCreateLargeDataBlock_1024ThisIsAVeryLongStringRepeatedManyTimesToCreateLargeDataBlock_1024ThisIsAVeryLongStringRepeatedManyTimesToCreateLargeDataBlock_10

note: its not an issue of the processor/sed - i just used sed here for sake of simplicity. i get weird errors piping the output to jq or yq as well (thats how i actually stumbled upon this).

I noticed now, that this actually seems to be a race condition of a kind: Sometime there are actually 3 valid json documents (one per line) created, especially when piping in a file instead of a program. however repeating the steps some times, it actually cuts off the documents again.

Could it be some kind of "buffer is full and data is dropped, if the buffer isn't flushed fast enough" issue?

[hojerst]

out.json.gz

[levkohimins]

@hojerst

Try with --terragrunt-log-disable, I guess it's what you need

terragrunt run-all show --terragrunt-log-disable -json planfile | sed -n 2p

[hojerst]

hm my version is 3 weeks old (0.67.3) and doesn't have this flag yet. i'm currently using nixpkgs for my toolchain and the latest version right now is 0.67.3... i'll try with the latest binary from here...

[hojerst]

note: terragrunt version v0.67.14 on M1 Max MacBook Pro

i just ran the snippet multiple times, piping directly to jq, so i don't have to scroll and look for a broken json manually. the problem still persists with randomly broken json documents:

this is the output of two consecutive runs - the first printing the first json document fine, then failing on the second. the next call failed even on the first document / plan json output

[... snip - everything fine ...]
      ]
    }
  ],
  "timestamp": "2024-09-24T20:22:17Z",
  "errored": false
}
jq: parse error: Invalid literal at line 2, column 32785
❯ ~/.local/bin/terragrunt run-all show --terragrunt-log-disable -json planfile | jq
jq: parse error: Invalid literal at line 1, column 98321

[levkohimins]

I figured out why this happens. When we run run-all show, modules are executed in parallel, and the output from different modules starts to overlap, thus corrupting each other's results. So we need to buffer the output of each module, and then output it atomically. I'll fix this soon. @hojerst Thanks!

[levkohimins]

While we are working on a fix, you can use the --terragrunt-parallelism 1 flag as a workaround.