Failing to launch any spot instances due to EBS throughput insufficient

[aleem-99]

Github issue

Issue type

• Bug Report

Build number

Custom build

9be6b5c5fef54f04e48b5b1db5a28e082b160d4e

Configuration

autospotting_allowed_instance_types = "c4.xlarge"

Environment

• AWS region
  ap-south-1
• Type of environment: (VPC, EC2Classic or DefaultVPC)
  VPC
• Anonymized launch configuration:

    "LaunchConfigurations": [
        {
            "LaunchConfigurationName": "redacted",
            "LaunchConfigurationARN": "redacted",
            "ImageId": "redacted",
            "KeyName": "",
            "SecurityGroups": [
                "redacted"
            ],
            "ClassicLinkVPCSecurityGroups": [],
            "UserData": "",
            "InstanceType": "c5.xlarge",
            "KernelId": "",
            "RamdiskId": "",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "VolumeSize": 100,
                        "VolumeType": "gp2",
                        "DeleteOnTermination": true
                    }
                }
            ],
            "InstanceMonitoring": {
                "Enabled": true
            },
            "IamInstanceProfile": "redacted",
            "CreatedTime": "2019-07-02T07:40:12.028Z",
            "EbsOptimized": false,
            "PlacementTenancy": "default"
        }
    ]
}

Summary

We have autoscaling group with c5.xlarge instance type. We have enabled autospotting on the ASG with 'allowedinstancetype' as "c5.x*, c4.x*". We noticed that everyday c5.xlarge spot instance were being taken away by AWS, and autospotting was replacing the on-demand with only c5.xlarge even though we have also specified "c4.x*" in lambda env. So we changed the 'allowedinstancetype' as "c4.xlarge" and autospotting failed to launch any instance.
In the logs we see the message "EBS throughput insufficient" when it scans instance-types against c4.xlarge.

Steps to reproduce

Create a ASG in ap-south-1 with c5.xlarge in Launch Config, and create autospotting with allowed_instance_type as c4.xlarge.

Expected results

launch c4.xlarge spot instances or a flag to disable EBS performance check.

Actual results

Failed to launch c4.xlarge instance.

2019/07/10 12:47:16 instance.go:373: Comparing current type c5.xlarge with price 0.17 with candidate c4.xlarge with price 0.0566
2019/07/10 12:47:16 instance.go:269: EBS throughput insufficient: 93.75 < 437.5'.
.
.
.
2019/07/10 12:47:16 instance.go:411: Couldn't determine the cheapest compatible spot instance type
2019/07/10 12:47:16 autoscaling.go:129: Could not launch cheapest spot instance: No cheaper spot instance types could be found
2019/07/10 12:47:16 Execution completed, nothing left to do

[cristim]

Thanks for reporting this issue but I am not offering support for custom builds, unless you are subscribed for a dedicated support plan.

Are you able to reproduce this with the latest nightly build?

[aleem-99]

Same result with the latest build

[cristim]

Thanks for checking this in the latest build.

As per the below screenshot of ec2instances.info it turns out that C4 has less throughput than C5, as well as slightly less memory:

At the moment AutoSpotting considers both these parameters and a few others, and only allows you to launch instances that are at least as large and performant as the currently running instances, so this behavior is actually expected.

If you want to get a C4 instance, for now all you can do it so configure the group to run C4 and the current algorithm would likely launch C4 spot instances. You may also try some older builds which were more relaxed in these checks, as this was a relatively recent development.

I'll create a feature request on implementing configuration flags that allow relaxing some of these constraints if desired, but I can't promise any timelines when this would be implemented.

If you're interested in working on this yourself, I'd love to see a PR about it and I'll do my best to help you at it.

[cristim]

Looks like a duplicate of #206, let's continue the conversation there.