ECS execute-command failed due to an internal error (aws/aws-cli/issues/6562)

[ssyberg]

This is just a wrapper for this issue that may have been prematurely closed: /issues/6562

I don't really have anything new to add except that it's definitely happening to us too, we appear to have an identical structure to that described in the related ticket with the only exception being a different region.

[nateprewitt]

Hi @ssyberg, was there new information regarding #6562? The issue is due to a defect in ECS that's causing a server side exception which it doesn't appear the CLI can control. I believe the ECS ticket (aws-containers/amazon-ecs-exec-checker#47) was proposed as the tracking ticket for the issue going forward. It appears you've chimed in there as well.

Would you mind providing more info on what we can track here?

[ssyberg]

The issue is due to a defect in ECS that's causing a server side exception which it doesn't appear the CLI can control

@nateprewitt I may have missed this somewhere, can you point me to where this was determined to be the cause? If so is there an ECS specific bug documented somewhere that we can track or any known workarounds (I recognize that's not CLI related but any help is appreciated!).

Both tickets have been closed and it seems like my team, @nic-russo, and @justfathi are still experiencing this issue. I'm not in a position to determine if it's a CLI issue and agree that the ecs-exec-checker should be able to expose this as well (though honestly doesn't seem unreasonable that the CLI might be about to expose more details on failure...)

[nateprewitt]

The error itself is directly from the ECS API which is claiming an internal error in the service. This message is simply proxied back to the CLI but provides us no further insight into why ECS failed. This lead to the introduction of amazon-ecs-exec-checker to diagnose configuration issues. ECS also provides a knowledge center article on debugging this problem with some other things to check.

If the tool is unable to diagnose the current issue, it may need additional functionality, which I believe is what's being determined in the above ticket. We could leave this open for visibility, but I don't believe there are any changes that can be made in the CLI to remedy this issue.

[ssyberg]

@nateprewitt totally get where you're coming from, and I think you're technically correct that this is not an issue with the CLI, unfortunately for us the CLI is our only interface to the execute-command so anything beyond that is pretty opaque to your users. I don't really have a suggestion as to if this ticket should stay open or where the responsibility is to address this, but if you have any special access to the ECS team I think we'd all be very grateful. FWIW more comments are flowing in from users in the closed bug, this appears to be a more widespread issue than we originally thought.

[tim-finnigan]

Looking more into this now, this has already been escalated internally due to the uptick in these issues being reported. Will post an update when I have more information.

[tim-finnigan]

Closing this, updates posted here: aws/amazon-ssm-agent#435. Will add more information there going forward. It appears that the issue relates to aws-containers/amazon-ecs-exec-checker#49.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.