Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option to not wait or stop when draining has started #25

Closed
eric-kinsa opened this issue Mar 21, 2021 · 7 comments · Fixed by #27
Closed

Option to not wait or stop when draining has started #25

eric-kinsa opened this issue Mar 21, 2021 · 7 comments · Fixed by #27
Labels
enhancement New feature or request help wanted Extra attention is needed merge

Comments

@eric-kinsa
Copy link

Is there an option or could there be an option to not wait for containers to be fully drained and stopped? Right now I'm testing your tool as an alternative to ecs-deploy, and it works great by the way since all my services are multi-container, however, the time it takes for containers to drain and close would probably annoy the devs I support. And I would prefer not to change the drain time on my target groups.

I currently use the "--skip-deployment-check" on ecs-deploy and run a wait loop using the aws cli utility to check for a drain status on my target groups. Once the draining starts, the job moves on from there with post deployment steps. With my current setup, the connection draining doesn't start until the health check passes on the target group, so I'm confident in not holding up a deployment job until the containers are fully deregistered or closed.
@andresvia andresvia added enhancement New feature or request help wanted Extra attention is needed labels Mar 24, 2021
@andresvia
Copy link
Contributor

@eric-kinsa to make sure I understand, we are looking for a new new option that will still wait after making the UpdateService call to ECS, but the idea is to wait only until draining has started, not until the draining has finished.

@eric-kinsa
Copy link
Author

Yeah, either wait until the connections start draining or not wait at all. Whichever is easier. I can make a AWS CLI call to check for connections to start draining like I was doing for the ecs-deploy script.

@andresvia
Copy link
Contributor

I think I started to see the appeal of this option. I have created a WIP branch (https://github.com/Autodesk/go-awsecs/compare/add-wait-until-option?expand=1), if you want to try it out. Logs are changed like this:

No service update in progress:

./update-aws-ecs-service -region us-east-1 -profile default -wait-until draining-started -cluster mycluster -service myservice
2021/03/31 01:12:06 Initial target states: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:06 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:07 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:08 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:09 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:11 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:13 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:18 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:21 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:27 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:35 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:12:51 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.244":"healthy"}'
2021/03/31 01:13:21 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.237":"initial","x.y.z.244":"healthy"}'
2021/03/31 01:14:18 Waiting for targets transitioning to draining state: '{"x.y.z.209":"healthy","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:14:18 The target 'x.y.z.244' transitioned to draining state

With previous service update still in progress, trigger update again:

./update-aws-ecs-service -region us-east-1 -profile default -wait-until draining-started -cluster mycluster -service myservice
2021/03/31 01:17:15 Initial target states: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:15 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:16 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:17 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:18 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:20 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:23 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:27 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:31 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:36 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:17:44 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:18:00 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:18:30 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy","x.y.z.244":"draining"}'
2021/03/31 01:19:27 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.237":"healthy"}'
2021/03/31 01:20:10 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.209":"draining","x.y.z.233":"healthy","x.y.z.237":"healthy"}'
2021/03/31 01:21:04 Waiting for targets transitioning to draining state: '{"x.y.z.199":"healthy","x.y.z.233":"healthy","x.y.z.237":"draining"}'
2021/03/31 01:21:04 The target 'x.y.z.237' transitioned to draining state

@eric-kinsa please test it out if you can.

@eric-kinsa
Copy link
Author

eric-kinsa commented Apr 4, 2021 •
edited
Loading

Works great! I ran a couple of test runs and it finished as soon as one target starting connection draining. There's another plus to this: I usually see a cpu spike when there's a deployment of a set of new containers, which will kick off the autoscaling once. This would cause the desired count check to kick off (example below) and add a couple of minutes. Now with this drain flag, it ignores what the previous count of containers were, which works out well for me. Thanks for implementing this!

2021/04/04 03:14:23 other than PRIMARY deployment found
2021/04/04 03:18:19 not running the desired count

@andresvia
Copy link
Contributor

@eric-kinsa , these are older logs right? My branch in progress is not yet merged, and you will need to pass the -wait-until draining-started to test out the feature. (you will need to compile the branch https://github.com/Autodesk/go-awsecs/compare/add-wait-until-option?expand=1 and the logs will look more like what I posted before.

Thanks again.

@eric-kinsa
Copy link
Author

That was an example of the logs of what I was occasionally getting without this new "wait-until-draining-started" flag. I compiled this branch and it works great.

@eric-kinsa
Copy link
Author

Do you need any more testing on this feature that I can help with before merging in?

@andresvia andresvia added the merge label Aug 4, 2021
@andresvia andresvia mentioned this issue Aug 4, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

v1.5-rc merge Autodesk/go-awsecs
2 participants
@andresvia @eric-kinsa