ECS execute-command failed due to an internal error (aws/aws-cli/issues/6834)

[istvanfedak-nbcu]

Describe the bug

Related issues:

• ECS execute-command failed due to an internal error. #6562
• ECS execute-command failed due to an internal error (aws/aws-cli/issues/6562) #6834
• Recent updates possibly broke CLI execute-command amazon-ssm-agent#435
• ECS execute-command failed due to an internal error. aws-containers/amazon-ecs-exec-checker#47

Unable to run aws ecs execute-command.

Expected Behavior

Open an interactive shell with the container.

Current Behavior

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

Reproduction Steps

1. Make sure that your ECS task execution role has all the required configuration defined in: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html
2. Run:
   aws ecs execute-command
   --cluster MyCluster
   --task arn:aws:ecs:us-east-1:123456789012:task/MyCluster/d789e94343414c25b9f6bd59eEXAMPLE
   --container MyContainer
   --interactive
   --command "/bin/sh"

Possible Solution

No response

Additional Information/Context

I was able to run aws ecs execute-command last week and it stopped working today.

• Amazon ECS-optimized Amazon Linux 2 AMI used in ECS clusters: ami-0507dff4275d8dd6d

CLI version used

aws-cli/2.11.20 Python/3.11.3 Darwin/20.6.0 source/x86_64 prompt/off

Environment details (OS name and version, etc.)

MacOS v11.7.8

[istvanfedak-nbcu]

I tested the ECS containers that are running on AWS Fargate and they're not affected by this issue (I'm able to open a shell with them).

This is the SSM version of the Amazon ECS-optimized Amazon Linux 2 AMI used in ECS clusters (ami-0507dff4275d8dd6d):

Loaded plugins: priorities, update-motd, upgrade-helper
Installed Packages
Name        : amazon-ssm-agent
Arch        : x86_64
Version     : 3.1.1732.0
Release     : 1.amzn2
Size        : 90 M
Repo        : installed
From repo   : amzn2-core
Summary     : Manage EC2 Instances using SSM APIs
URL         : http://docs.aws.amazon.com/ssm/latest/APIReference/Welcome.html
License     : ASL 2.0
Description : This package provides Amazon SSM Agent for managing EC2 Instances using SSM APIs

[istvanfedak-nbcu]

Workaround:

1. Connect to EC2 instance via Session Manager.
2. List out all the containers running in the EC2 instance: sudo docker ps.
3. Run the following command to open up a shell with the desired container: sudo docker exec --interactive --tty <container-id-value> sh

[tim-finnigan]

Hi @istvanfedak-nbcu thanks for reaching out. Can you confirm that the permissions you are using have not changed recently? In this troubleshooting post for TargetNotConnectedException it notes:

You might get this error due to the following reasons:

• The Amazon ECS task role doesn't have the required permissions to run the execute-command command.
• The AWS Identity and Access Management (IAM) role or user that's running the command doesn't have the required permissions.

This may be a better issue to track in aws-containers/amazon-ecs-exec-checker#47 or the SSM agent repository (https://github.com/aws/amazon-ssm-agent/issues).

Have you tried updating to a newer version of the SSM Agent? Here are the release notes for reference: https://github.com/aws/amazon-ssm-agent/blob/mainline/RELEASENOTES.md

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.