ECS execute-command failed due to an internal error. #6562

nic-russo · 2021-11-23T01:09:26Z

Confirm by changing [ ] to [x] below to ensure that it's a bug:

I've gone though the User Guide and the API reference
I've searched for previous similar issues and didn't find any solution

Describe the bug
Attempting to run execute-command onto a task using aws ecs execute-command

SDK version number

aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off

Platform/OS/Hardware/Device
What are you running the cli on?

Linux 5.11.0-40-generic x86_64 GNU/Linux

To Reproduce (observed behavior)
Steps to reproduce the behavior
Run aws ecs execute-command --cluster clusterName --task taskId --command "/bin/bash" --interactive

Expected behavior
A clear and concise description of what you expected to happen.
It should open an interactive session.
Logs/output
Get full traceback and error logs by adding --debug to the command.

2021-11-23 01:54:12,137 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20
2021-11-23 01:54:12,137 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ecs', 'execute-command', '--region', 'eu-south-1', '--cluster', 'procedure', '--task', 'cf41c924968e426c9be535f3f47545be', '--command', '/bin/bash', '--interactive', '--container', 'procedure', '--debug']
2021-11-23 01:54:12,159 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fc147e8ed30>
2021-11-23 01:54:12,159 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fc147fe7a60>
2021-11-23 01:54:12,159 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc148090280>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc1480983a0>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fc147e1e790>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fc147fad8b0>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-11-23 01:54:12,160 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fc147e959d0>
2021-11-23 01:54:12,160 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.4.0/dist/awscli/data/cli.json
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fc147ee78b0>
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fc147ee8430>
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fc147ee83a0>
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fc147ee8550>
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fc147ee84c0>
2021-11-23 01:54:12,163 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fc147db6480>
2021-11-23 01:54:12,163 - MainThread - botocore.session - DEBUG - Setting config variable for region to 'eu-south-1'
2021-11-23 01:54:12,164 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off
2021-11-23 01:54:12,164 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ecs', 'execute-command', '--region', 'eu-south-1', '--cluster', 'procedure', '--task', 'cf41c924968e426c9be535f3f47545be', '--command', '/bin/bash', '--interactive', '--container', 'procedure', '--debug']
2021-11-23 01:54:12,164 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fc147e8f3a0>
2021-11-23 01:54:12,164 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fc1488d8820>
2021-11-23 01:54:12,164 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fc147dfe160>
2021-11-23 01:54:12,164 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fc1488d2c10>
2021-11-23 01:54:12,164 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fc14883a4c0>
2021-11-23 01:54:12,165 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-11-23 01:54:12,166 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fc147fad790>
2021-11-23 01:54:12,166 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fc147fe2940>
2021-11-23 01:54:12,188 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.4.0/dist/awscli/botocore/data/ecs/2014-11-13/service-2.json
2021-11-23 01:54:12,198 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function inject_commands at 0x7fc147ed9160>
2021-11-23 01:54:12,198 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function add_waiters at 0x7fc147e959d0>
2021-11-23 01:54:12,221 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.4.0/dist/awscli/botocore/data/ecs/2014-11-13/waiters-2.json
2021-11-23 01:54:12,222 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('cluster', <awscli.arguments.CLIArgument object at 0x7fc1473b8ee0>), ('container', <awscli.arguments.CLIArgument object at 0x7fc14735a040>), ('command', <awscli.arguments.CLIArgument object at 0x7fc14735a070>), ('interactive', <awscli.arguments.BooleanArgument object at 0x7fc14735a0a0>), ('no-interactive', <awscli.arguments.BooleanArgument object at 0x7fc14735a0d0>), ('task', <awscli.arguments.CLIArgument object at 0x7fc14735a100>)])
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_streaming_output_arg at 0x7fc147e8f940>
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7fc147da8040>
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7fc147da80d0>
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_cli_input_json at 0x7fc14883ad30>
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_cli_input_yaml at 0x7fc148843040>
2021-11-23 01:54:12,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function unify_paging_params at 0x7fc147ff10d0>
2021-11-23 01:54:12,244 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.4.0/dist/awscli/botocore/data/ecs/2014-11-13/paginators-1.json
2021-11-23 01:54:12,244 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_generate_skeleton at 0x7fc147ed9e50>
2021-11-23 01:54:12,245 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fc14735a250>>
2021-11-23 01:54:12,245 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fc14735a280>>
2021-11-23 01:54:12,245 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc14735a3d0>>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cluster: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc1488807f0>
2021-11-23 01:54:12,246 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'procedure' for parameter "cluster": 'procedure'
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.container: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc1488807f0>
2021-11-23 01:54:12,246 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'procedure' for parameter "container": 'procedure'
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.command: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc1488807f0>
2021-11-23 01:54:12,246 - MainThread - awscli.arguments - DEBUG - Unpacked value of '/bin/bash' for parameter "command": '/bin/bash'
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.interactive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.task: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,246 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc1488807f0>
2021-11-23 01:54:12,246 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'cf41c924968e426c9be535f3f47545be' for parameter "task": 'cf41c924968e426c9be535f3f47545be'
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc14757f250>
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fc14735a250>>
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fc14735a280>>
2021-11-23 01:54:12,247 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc14735a3d0>>
2021-11-23 01:54:12,264 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-11-23 01:54:12,264 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-11-23 01:54:12,265 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-11-23 01:54:12,265 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-11-23 01:54:12,265 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-11-23 01:54:12,265 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2021-11-23 01:54:12,266 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.4.0/dist/awscli/botocore/data/endpoints.json
2021-11-23 01:54:12,276 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fc149c63820>
2021-11-23 01:54:12,277 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ecs: calling handler <function add_generate_presigned_url at 0x7fc14a278d30>
2021-11-23 01:54:12,280 - MainThread - botocore.endpoint - DEBUG - Setting ecs timeout as (60, 60)
2021-11-23 01:54:12,281 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.ecs.ExecuteCommand: calling handler <function base64_decode_input_blobs at 0x7fc147dfe8b0>
2021-11-23 01:54:12,281 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ecs.ExecuteCommand: calling handler <function generate_idempotent_uuid at 0x7fc149c098b0>
2021-11-23 01:54:12,281 - MainThread - botocore.hooks - DEBUG - Event before-call.ecs.ExecuteCommand: calling handler <function inject_api_version_header_if_needed at 0x7fc149c0f160>
2021-11-23 01:54:12,281 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ExecuteCommand) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off command/ecs.execute-command'}, 'body': b'{"cluster": "procedure", "container": "procedure", "command": "/bin/bash", "interactive": true, "task": "cf41c924968e426c9be535f3f47545be"}', 'url': 'https://ecs.eu-south-1.amazonaws.com/', 'context': {'client_region': 'eu-south-1', 'client_config': <botocore.config.Config object at 0x7fc1470025e0>, 'has_streaming_input': False, 'auth_type': None}}
2021-11-23 01:54:12,281 - MainThread - botocore.hooks - DEBUG - Event request-created.ecs.ExecuteCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc1470026d0>>
2021-11-23 01:54:12,281 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ecs.ExecuteCommand: calling handler <function set_operation_specific_signer at 0x7fc149c09790>
2021-11-23 01:54:12,282 - MainThread - botocore.credentials - DEBUG - Credentials for role retrieved from cache.
2021-11-23 01:54:12,282 - MainThread - botocore.credentials - DEBUG - Retrieved credentials will expire at: 2021-11-23 01:52:39+00:00
2021-11-23 01:54:12,283 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-11-23 01:54:12,283 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ecs.eu-south-1.amazonaws.com
x-amz-date:20211123T005412Z
OMITTED
x-amz-target:AmazonEC2ContainerServiceV20141113.ExecuteCommand

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
9f16e63ffe66d803140798d0577032f5796bb90e7ab4f320eee9d89f6bfef783
2021-11-23 01:54:12,283 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20211123T005412Z
20211123/eu-south-1/ecs/aws4_request
0f33903c5e3d5bacf04dc34a8aececdf17bfadd0d91c6cd49424e0d6374c89ac
2021-11-23 01:54:12,283 - MainThread - botocore.auth - DEBUG - Signature:
1771622c741eb3ca204a8c8ee9e26357ecca8b068ea57bceba0ef41072f02837
2021-11-23 01:54:12,283 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ecs.eu-south-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off command/ecs.execute-command', 'X-Amz-Date': b'20211123T005412Z', 'X-Amz-Security-Token': OMITTED, 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAURFG4DI6CP53NANT/20211123/eu-south-1/ecs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=1771622c741eb3ca204a8c8ee9e26357ecca8b068ea57bceba0ef41072f02837', 'Content-Length': '139'}>
2021-11-23 01:54:12,283 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.4.0/dist/awscli/botocore/cacert.pem
2021-11-23 01:54:12,284 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ecs.eu-south-1.amazonaws.com:443
2021-11-23 01:54:12,545 - MainThread - urllib3.connectionpool - DEBUG - https://ecs.eu-south-1.amazonaws.com:443 "POST / HTTP/1.1" 400 122
2021-11-23 01:54:12,547 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '770b6bc6-9cf0-4a47-bb6d-bebf50dacdcc', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Tue, 23 Nov 2021 00:54:12 GMT', 'Connection': 'close'}
2021-11-23 01:54:12,548 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-11-23 01:54:12,554 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '770b6bc6-9cf0-4a47-bb6d-bebf50dacdcc', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Tue, 23 Nov 2021 00:54:12 GMT', 'Connection': 'close'}
2021-11-23 01:54:12,554 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-11-23 01:54:12,554 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ecs.ExecuteCommand: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fc146fca130>>
2021-11-23 01:54:12,555 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-11-23 01:54:12,555 - MainThread - botocore.hooks - DEBUG - Event after-call.ecs.ExecuteCommand: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fc147002c70>>
2021-11-23 01:54:12,557 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 459, in main
  File "awscli/clidriver.py", line 594, in __call__
  File "awscli/clidriver.py", line 770, in __call__
  File "awscli/customizations/ecs/executecommand.py", line 91, in invoke
  File "awscli/botocore/client.py", line 281, in _api_call
  File "awscli/botocore/client.py", line 609, in _make_api_call
botocore.errorfactory.TargetNotConnectedException: An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

Additional context
Add any other context about the problem here.

check-ecs-exec.sh output:

$ bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) clusterName cf41c924968e426c9be535f3f47545be
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off)
  Session Manager Plugin | OK (1.2.205.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : eu-south-1
Cluster: clusterName
Task   : cf41c924968e426c9be535f3f47545be
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::ACCOUNT_ID:role/ADMIN_ROLE_NAME
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks | 
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "taskName"
    ----------
      Init Process Enabled (taskName:1)
    ----------
         1. Disabled - "taskName"
    ----------
      Read-Only Root Filesystem (taskName:1)
    ----------
         1. Disabled - "taskName"
  Task Role Permissions  | arn:aws:iam::ACCOUNT_ID:role/taskName-ecs-task
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | 
    Found existing endpoints for vpc-ID:
      - com.amazonaws.eu-south-1.ssm
      - com.amazonaws.eu-south-1.ec2messages
      - com.amazonaws.eu-south-1.ssmmessages

The text was updated successfully, but these errors were encountered:

tim-finnigan · 2021-11-23T17:29:02Z

Hi @nic-russo, thanks for reaching out. That error was also referenced in issues #6456 and #6070. Have you tried looking through the comments in those issues?

In both issues there are other issues mentioned from the https://github.com/aws-containers/amazon-ecs-exec-checker repository that might help give more insight into why you are seeing this error.

nic-russo · 2021-11-24T10:00:01Z

Hi @tim-finnigan, thank you for your support. I went through those issues and this guide. I can confirm that:

I'm running execute-command assuming an Admin IAM role on a child account of my Organization.
My container just start the grafana:8.2.4-ubuntu image without any further customization. Using Fargate with network_mode = awsvpc and running as non-root user (also tried using root user with no luck).
My Task Role (and Task Execution Role) has these permissions:

"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel",
"ecs:ExecuteCommand"

My container definition has readonlyRootFilesystem = false and initProcessEnabled = true
Describing my task I see:

"managedAgents": [
                        {
                            "lastStartedAt": "2021-11-23T19:40:07.757000+01:00",
                            "name": "ExecuteCommandAgent",
                            "lastStatus": "RUNNING"
                        }
                    ],
...
"enableExecuteCommand": true

Not sure if I'm missing something else.. I'll continue investigating

tim-finnigan · 2021-11-26T22:52:12Z

Hi @nic-russo, thanks for providing that additional information. Were you able to figure out the problem upon further investigation? I found another issue also mentioning that error which might help: aws-containers/amazon-ecs-exec-checker#43

nic-russo · 2021-11-28T15:12:43Z

Hi @tim-finnigan, thanks again for your help. Nope, I can't really see why I'm unable to connect.. I've double-checked the Managed Agent status (as per your last issue link) suddenly after running execute-command and it's actually RUNNING but I'm still getting TargetNotConnectedException. Do you see anything obvious I need to double-check?

nic-russo · 2021-11-28T16:19:30Z

I can also confirm my container is running into a private subnet (with NAT) with security group that allows all traffic (outbound).

justfathi · 2021-11-30T13:45:19Z

Any update on this??

JoeyHoutenbos · 2021-11-30T20:45:13Z

I was working on kind of the same problem today. In the end my problem is that I was injecting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables in the task definition.

These variables are working perfectly fine, but prevent the execute-command from working.

justfathi · 2021-11-30T20:52:43Z

The AMI `ami-0fe19057e9cb4efd` seems to cause the issue for us

…

On Tue, Nov 30, 2021 at 4:45 PM Joey Houtenbos ***@***.***> wrote: I was working on kind of the same problem today. In the end my problem is that I was injecting the *AWS_ACCESS_KEY_ID* and *AWS_SECRET_ACCESS_KEY* as environment variables in the task definition. These variables are working perfectly fine, but prevent the *execute-command* from working. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6562 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA27R4FEJF6I5FX5WOBHFLTUOUZWJANCNFSM5ISK42HQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

tim-finnigan · 2021-11-30T21:20:14Z

Thanks all for providing your suggestions. @nic-russo unfortunately I'm not sure what the issue could be here and I’m having trouble reproducing it. You may want to open an issue in that other repository too (https://github.com/aws-containers/amazon-ecs-exec-checker) as maybe they can help with this. I’m also going to remove my assignment here and give my teammates the opportunity to investigate, but please let us know if you do find a solution.

kdaily · 2021-12-06T22:22:52Z

@nic-russo, I see that you've engaged the team over on aws-containers/amazon-ecs-exec-checker#47. Closing this out for now. If they narrow down something to the AWS CLI, please feel free to open up again. Thanks!

github-actions · 2021-12-06T22:23:16Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

justfathi · 2022-02-01T02:51:41Z

AmazonSSMFullAccess

This did not fix the issue

nic-russo · 2022-02-01T12:18:57Z

@justfathi are you experiencing the my very same issue?

justfathi · 2022-02-01T12:32:44Z

Yes, still experiencing the same issue unfortunately

…

On Tue., Feb. 1, 2022, 8:19 a.m. nic-russo, ***@***.***> wrote: @justfathi <https://github.com/justfathi> are you experiencing the my very same issue? — Reply to this email directly, view it on GitHub <#6562 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA27R4GOO6ZAX42KJAGDSKLUY7FT3ANCNFSM5ISK42HQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

ssyberg · 2022-03-31T17:41:10Z

Following this as well!

Sohett · 2022-04-06T17:24:19Z

facing exactly the same issue here. Everything was working for a few months and now I can't execute the command.

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I have of course checked the amazon-ecs-exec-checker and everything is green ✅ .

Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/local/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.5.0 Python/3.9.12 Darwin/20.6.0 source/x86_64 prompt/off)
  Session Manager Plugin | OK (1.2.312.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : eu-central-1
Cluster: XXXXXXXX
Task   : XXXXXXXX
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::XXXXXXXX:user/thomas_sohet
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: implicitDeny
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "XXXXXXXX"
         2. RUNNING for "datadog-agent-XXXXXXXX"
    ----------
      Init Process Enabled (XXXXXXXX:64)
    ----------
         1. Disabled - "XXXXXXXX"
         2. Disabled - "XXXXXXXX"
    ----------
      Read-Only Root Filesystem (XXXXXXXX)
    ----------
         1. Disabled - "XXXXXXX"
         2. Disabled - "datadog-agent-XXXXXXXX"
  Task Role Permissions  | arn:aws:iam::xxxxxxxxxxxxxx:role/ecsTaskExecutionRole
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | SKIPPED (vpc-xxxxxxxxxxxxxx - No additional VPC endpoints required)
  Environment Variables  | (XXXXXXXX:64)
       1. container "XXXXXXXXX"
       - AWS_ACCESS_KEY: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       2. container "datadog-agent-XXXXXXXX"
       - AWS_ACCESS_KEY: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

Not sure what to do. If someone finds a solution, please ping me 🙏

rfroetscher · 2022-04-06T20:12:12Z

I'm having the same as above right now. Everything was working for months but now I am getting this error.

-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/local/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.5.2 Python/3.9.11 Darwin/21.3.0 exe/x86_64 prompt/off)
  Session Manager Plugin | OK (1.2.245.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-west-2
Cluster: xxxxx
Task   : xxxxx
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | xxxxx
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks | 
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "xxxxxxxx"
    ----------
      Init Process Enabled (xxxxxxxxxxxx:175)
    ----------
         1. Disabled - "xxxxxxxx"
    ----------
      Read-Only Root Filesystem (xxxxxxxxxxxx:175)
    ----------
         1. Disabled - "xxxxxxxx"
  Task Role Permissions  | xxxxxxxx
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | SKIPPED (vpc-84f12ae3 - No additional VPC endpoints required)
  Environment Variables  | (xxxxxxxxxxxx:175)
       1. container "xxxxxxxx"
       - AWS_ACCESS_KEY: not defined
       - AWS_SECRET_ACCESS_KEY: defined`

tim-finnigan · 2022-04-06T20:23:08Z

Please refer to aws/amazon-ssm-agent#435 for updates on this and to leave related comments.

istvanfedak-nbcu · 2023-08-04T19:43:21Z

I'm experiencing this exact same issue. aws ecs execute-command was working for me last week and it stopped working.

fernandomullerjr · 2024-03-27T21:29:45Z

facing exactly the same issue here.
An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I have of course checked the amazon-ecs-exec-checker and everything is green ✅ .

antonio-gabriele · 2024-07-13T15:49:45Z

Leave AWS. Go on Azure. :-)

ixti · 2024-10-24T04:31:45Z

We have the same issue. ECS connect is highly unstable. Sometimes it allows to connect, then it does not...

leejayhsu · 2024-11-11T22:47:11Z

having the same issue also

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/opt/homebrew/bin/jq)
  AWS CLI | OK (/opt/homebrew/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.19.4 Python/3.12.7 Darwin/24.0.0 source/arm64)
  Session Manager Plugin | OK (1.2.497.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-west-2
Cluster: core-services
Task   : xxxxxxx
-------------------------------------------------------------
  Cluster Configuration  |
     KMS Key       : Not Configured
     Audit Logging : OVERRIDE
     S3 Bucket Name: Not Configured
     CW Log Group  : /ecs/dev/core-services, Encryption Enabled: true
  Can I ExecuteCommand?  | arn:aws:iam::xxxxxx:user/xxxxxx
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "datadog-agent"
         2. RUNNING for "log-router"
         3. RUNNING for "app"
    ----------
      Init Process Enabled (dev-app-task-def:554)
    ----------
         1. Enabled - "app"
         2. Disabled - "datadog-agent"
         3. Disabled - "log-router"
    ----------
      Read-Only Root Filesystem (dev-app-task-def:554)
    ----------
         1. Disabled - "app"
         2. Disabled - "datadog-agent"
         3. Disabled - "log-router"
  Task Role Permissions  | arn:aws:iam::xxxxxx:role/ecsTaskExecutionRole
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
     -----
     logs:DescribeLogGroups: allowed
     logs:CreateLogStream: allowed
     logs:DescribeLogStreams: allowed
     logs:PutLogEvents: allowed
  VPC Endpoints          |
    Found existing endpoints for vpc-xxxxxx:
      - com.amazonaws.us-west-2.s3
      - com.amazonaws.us-west-2.secretsmanager
      - com.amazonaws.us-west-2.ecr.api
      - com.amazonaws.us-west-2.ecr.dkr
      - com.amazonaws.us-west-2.ssmmessages
       1. container "app"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       2. container "datadog-agent"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       3. container "log-router"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

nic-russo added the needs-triage This issue or PR still needs to be triaged. label Nov 23, 2021

tim-finnigan added ecs guidance Question that needs advice or information. labels Nov 23, 2021

tim-finnigan self-assigned this Nov 23, 2021

tim-finnigan added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Nov 23, 2021

tim-finnigan added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Nov 23, 2021

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 24, 2021

tim-finnigan added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 26, 2021

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Nov 28, 2021

tim-finnigan removed their assignment Nov 30, 2021

nic-russo mentioned this issue Dec 1, 2021

ECS execute-command failed due to an internal error. aws-containers/amazon-ecs-exec-checker#47

Open

kdaily closed this as completed Dec 6, 2021

ssyberg mentioned this issue Mar 31, 2022

ECS execute-command failed due to an internal error (aws/aws-cli/issues/6562) #6834

Closed

ssyberg mentioned this issue Apr 1, 2022

Recent updates possibly broke CLI execute-command aws/amazon-ssm-agent#435

Closed

istvanfedak-nbcu mentioned this issue Aug 4, 2023

ECS execute-command failed due to an internal error (aws/aws-cli/issues/6834) #8080

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECS execute-command failed due to an internal error. #6562

ECS execute-command failed due to an internal error. #6562

nic-russo commented Nov 23, 2021

tim-finnigan commented Nov 23, 2021 •

edited

Loading

nic-russo commented Nov 24, 2021 •

edited

Loading

tim-finnigan commented Nov 26, 2021

nic-russo commented Nov 28, 2021

nic-russo commented Nov 28, 2021

justfathi commented Nov 30, 2021

JoeyHoutenbos commented Nov 30, 2021

justfathi commented Nov 30, 2021 via email

tim-finnigan commented Nov 30, 2021

kdaily commented Dec 6, 2021

github-actions bot commented Dec 6, 2021

justfathi commented Feb 1, 2022

nic-russo commented Feb 1, 2022

justfathi commented Feb 1, 2022 via email

ssyberg commented Mar 31, 2022

Sohett commented Apr 6, 2022

rfroetscher commented Apr 6, 2022 •

edited

Loading

tim-finnigan commented Apr 6, 2022

istvanfedak-nbcu commented Aug 4, 2023

fernandomullerjr commented Mar 27, 2024

antonio-gabriele commented Jul 13, 2024

ixti commented Oct 24, 2024

leejayhsu commented Nov 11, 2024 •

edited

Loading

ECS execute-command failed due to an internal error. #6562

ECS execute-command failed due to an internal error. #6562

Comments

nic-russo commented Nov 23, 2021

tim-finnigan commented Nov 23, 2021 • edited Loading

nic-russo commented Nov 24, 2021 • edited Loading

tim-finnigan commented Nov 26, 2021

nic-russo commented Nov 28, 2021

nic-russo commented Nov 28, 2021

justfathi commented Nov 30, 2021

JoeyHoutenbos commented Nov 30, 2021

justfathi commented Nov 30, 2021 via email

tim-finnigan commented Nov 30, 2021

kdaily commented Dec 6, 2021

github-actions bot commented Dec 6, 2021

⚠️COMMENT VISIBILITY WARNING⚠️

justfathi commented Feb 1, 2022

nic-russo commented Feb 1, 2022

justfathi commented Feb 1, 2022 via email

ssyberg commented Mar 31, 2022

Sohett commented Apr 6, 2022

rfroetscher commented Apr 6, 2022 • edited Loading

tim-finnigan commented Apr 6, 2022

istvanfedak-nbcu commented Aug 4, 2023

fernandomullerjr commented Mar 27, 2024

antonio-gabriele commented Jul 13, 2024

ixti commented Oct 24, 2024

leejayhsu commented Nov 11, 2024 • edited Loading

tim-finnigan commented Nov 23, 2021 •

edited

Loading

nic-russo commented Nov 24, 2021 •

edited

Loading

rfroetscher commented Apr 6, 2022 •

edited

Loading

leejayhsu commented Nov 11, 2024 •

edited

Loading