aws-ecs: appends invalid ecs user data

[cheruvian]

Describe the bug

ECS appends the following to the asg userdata

autoScalingGroup.addUserData('sudo service iptables save');

This command is invalid on the most recent versions of the ECS optimized AMI

+ sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP
+ sudo service iptables save
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
+ echo ECS_AWSVPC_BLOCK_IMDS=true

Expanding to look at the whole code block the comments seem to be out of place

            // Deny containers access to instance metadata service
            // Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
            autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
            autoScalingGroup.addUserData('sudo service iptables save');
            // The following is only for AwsVpc networking mode, but doesn't hurt for the other modes.
            autoScalingGroup.addUserData('echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config');

aws-cdk/packages/aws-cdk-lib/aws-ecs/lib/cluster.ts

Lines 567 to 569 in 59e96a3

	// Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
	autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
	autoScalingGroup.addUserData('sudo service iptables save');

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Should have valid commands in userdata. Is a service restart not needed?

Current Behavior

Command fails and suggests using systemctl

Reproduction Steps

Deploy an ECS cluster backed by ASG

Possible Solution

At a minimum reorder the comments, not sure what command it is intending to run, perhaps sudo netfilter-persistent save which isn't installed on the latest EBS optimized GPU AMI

            // Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
			// ??????

            // The following is only for AwsVpc networking mode, but doesn't hurt for the other modes.
            autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
            autoScalingGroup.addUserData('sudo service iptables restart');

            // Deny containers access to instance metadata service
            autoScalingGroup.addUserData('echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config');

Additional Information/Context

No response

CDK CLI Version

2.134.0 (build 265d769)

Framework Version

No response

Node.js Version

v20.10.0

OS

OSX

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

• Per code comment here, the logic implemented is for Amazon ECS-optimized AMI for Amazon Linux 2.
• Per AL2 Core packages not in Amazon Linux 2023, looks like iptables package is removed in AL 2023.
• Create an Amazon ECS cluster with the Amazon Linux 2023 ECS-Optimized-AMI has an example CloudFormation template.

The MachineImageType enum supports 2 values as of now - AMAZON_LINUX_2 and BOTTLEROCKET. May be we also should update this enum to add AMAZON_LINUX_2023 and then move current default case here under MachineImageType.AMAZON_LINUX_2 switch case and add default case for AL 2023!

Using AWS CLI command aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id --region us-east-1 (reference Retrieving Amazon ECS-optimized Linux AMI metadata), we could get the latest AMI for AL 2023 returns the following output:

{
    "Parameters": [
        {
            "Name": "/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id",
            "Type": "String",
            "Value": "ami-0a357ea20d7b79c2c",
            "Version": 61,
            "LastModifiedDate": "2024-11-18T08:07:53.585000-08:00",
            "ARN": "arn:aws:ssm:us-east-1::parameter/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id",
            "DataType": "text"
        }
    ],
    "InvalidParameters": []
}

Below CDK code could be used to reproduce the issue (assuming we could SSH to EC2 instance launched by ECS task):

import * as cdk from 'aws-cdk-lib';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as autoscaling from 'aws-cdk-lib/aws-autoscaling';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'EcsVpc');
    /*
    const ecsAmi = ec2.MachineImage.lookup({
      name: '/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id'
    });
    */

    const asg = new autoscaling.AutoScalingGroup(this, 'EcsAsg', { 
      vpc: vpc,
      instanceType: new ec2.InstanceType('t2.xlarge'),
      machineImage: ecs.EcsOptimizedImage.amazonLinux2023(),
      desiredCapacity: 3
    }); 

    const capacityProvider = new ecs.AsgCapacityProvider(this, 'EcsCapacityProvider', {
      autoScalingGroup: asg
    });

    const cluster = new ecs.Cluster(this, 'MyEcsCluster', {
      vpc: vpc
    });
    cluster.addAsgCapacityProvider(capacityProvider);
  }
}

@cheruvian Please validate the finding above. Also share if you were able to gather logs, perhaps from /var/log/cloud-init.log and /var/log/cloud-init-output.log, and share in this issue.

[phuhung273]

Hi team, upon trying to fix this issue, another one come up:

1. Changing from sudo service iptables save > sudo iptables-save -> instance can join cluster with desired iptable
2. Reran changed snapshot, found that aws-ecs-patterns/test/ec2/integ.multiple-application-load-balanced-ecs-service doesn't work: 1 out of 2 target groups always unhealthy
3. Try running sample image locally, port 90 doesn't work
   docker run -d -p 80:80 -p 90:90 amazon/amazon-ecs-sample:latest > curl http://localhost:90 FAIL
   docker run -d -p 80:80 -p 90:80 amazon/amazon-ecs-sample:latest > curl http://localhost:90 WORK

Which means aws-ecs-patterns/test/ec2/integ.multiple-application-load-balanced-ecs-service doesn't make sense ? Please help correct me on this point.