Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to launch any spot instances due to EBS throughput insufficient #353

Closed
aleem-99 opened this issue Jul 10, 2019 · 4 comments
Closed

Comments

@aleem-99
Copy link

aleem-99 commented Jul 10, 2019

Github issue

Issue type

  • Bug Report

Build number

Custom build

9be6b5c5fef54f04e48b5b1db5a28e082b160d4e

Configuration

autospotting_allowed_instance_types = "c4.xlarge"

Environment

  • AWS region
    ap-south-1
  • Type of environment: (VPC, EC2Classic or DefaultVPC)
    VPC
  • Anonymized launch configuration:
    "LaunchConfigurations": [
        {
            "LaunchConfigurationName": "redacted",
            "LaunchConfigurationARN": "redacted",
            "ImageId": "redacted",
            "KeyName": "",
            "SecurityGroups": [
                "redacted"
            ],
            "ClassicLinkVPCSecurityGroups": [],
            "UserData": "",
            "InstanceType": "c5.xlarge",
            "KernelId": "",
            "RamdiskId": "",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "VolumeSize": 100,
                        "VolumeType": "gp2",
                        "DeleteOnTermination": true
                    }
                }
            ],
            "InstanceMonitoring": {
                "Enabled": true
            },
            "IamInstanceProfile": "redacted",
            "CreatedTime": "2019-07-02T07:40:12.028Z",
            "EbsOptimized": false,
            "PlacementTenancy": "default"
        }
    ]
}

Summary

We have autoscaling group with c5.xlarge instance type. We have enabled autospotting on the ASG with 'allowedinstancetype' as "c5.x*, c4.x*". We noticed that everyday c5.xlarge spot instance were being taken away by AWS, and autospotting was replacing the on-demand with only c5.xlarge even though we have also specified "c4.x*" in lambda env. So we changed the 'allowedinstancetype' as "c4.xlarge" and autospotting failed to launch any instance.
In the logs we see the message "EBS throughput insufficient" when it scans instance-types against c4.xlarge.

Steps to reproduce

Create a ASG in ap-south-1 with c5.xlarge in Launch Config, and create autospotting with allowed_instance_type as c4.xlarge.

Expected results

launch c4.xlarge spot instances or a flag to disable EBS performance check.

Actual results

Failed to launch c4.xlarge instance.

2019/07/10 12:47:16 instance.go:373: Comparing current type c5.xlarge with price 0.17 with candidate c4.xlarge with price 0.0566
2019/07/10 12:47:16 instance.go:269: EBS throughput insufficient: 93.75 < 437.5'.
.
.
.
2019/07/10 12:47:16 instance.go:411: Couldn't determine the cheapest compatible spot instance type
2019/07/10 12:47:16 autoscaling.go:129: Could not launch cheapest spot instance: No cheaper spot instance types could be found
2019/07/10 12:47:16 Execution completed, nothing left to do
@cristim
Copy link
Member

cristim commented Jul 10, 2019

Thanks for reporting this issue but I am not offering support for custom builds, unless you are subscribed for a dedicated support plan.

Are you able to reproduce this with the latest nightly build?

@aleem-99
Copy link
Author

Same result with the latest build

@cristim
Copy link
Member

cristim commented Jul 15, 2019

Thanks for checking this in the latest build.

As per the below screenshot of ec2instances.info it turns out that C4 has less throughput than C5, as well as slightly less memory:

image

At the moment AutoSpotting considers both these parameters and a few others, and only allows you to launch instances that are at least as large and performant as the currently running instances, so this behavior is actually expected.

If you want to get a C4 instance, for now all you can do it so configure the group to run C4 and the current algorithm would likely launch C4 spot instances. You may also try some older builds which were more relaxed in these checks, as this was a relatively recent development.

I'll create a feature request on implementing configuration flags that allow relaxing some of these constraints if desired, but I can't promise any timelines when this would be implemented.

If you're interested in working on this yourself, I'd love to see a PR about it and I'll do my best to help you at it.

@cristim
Copy link
Member

cristim commented Jul 15, 2019

Looks like a duplicate of #206, let's continue the conversation there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants