Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updownstatus API startstop endpoint has no effect #34

Open
rjaduthie opened this issue Jan 20, 2023 · 1 comment
Open

updownstatus API startstop endpoint has no effect #34

rjaduthie opened this issue Jan 20, 2023 · 1 comment

Comments

@rjaduthie
Copy link
Contributor

rjaduthie commented Jan 20, 2023

I've added as much info as possible here, not sure of the cause of the problem, but I've put my next thing to check in the "Hypotheses" section. I might have time to look at this later, but it might be the weekend before I can sit down to work this through. Any input appreciated!

Issue

After applying the fix in PR #33, the API endpoints appear to work, but in fact the startstop endpoint is having no effect on the desire count of the ECS service.

Recreate

  1. Deploy using code from PR Fix updownstatus API lambda handler export #33
  2. Check server status - by looking at the AWS console or using the serverstatus endpoint - at this point if the status is "not running" the response is empty from the serverstatus endpoint:
curl --request GET 'https://[YOUR_API_GATEWAY_DOMAIN]/prod/serverstatus
  1. If the response is blank try starting the task:
curl --request GET 'https://[YOUR_API_GATEWAY_DOMAIN]/prod/startstop?key=[YOUR_API_PASSWORD_FROM_DOTENV_FILE]&desiredCount=1'

... else try to stop the task:

curl --request GET 'https://[YOUR_API_GATEWAY_DOMAIN]/prod/startstop?key=[YOUR_API_PASSWORD_FROM_DOTENV_FILE]&desiredCount=0'
  1. Check the status of the task again, either in the console or by using the API serverstatus endpoint.

Checks

Lambda Logs

The Lambda backing the startstop endpoint logs that it has made the update command - e.g.:

2023-01-20T00:05:18.106Z	732cdc88-f3e5-4a63-a2ac-a4b41237a9d5	INFO	starting service 
{
    "desiredCount": 1,
    "service": [SERVICE_ARN],
    "cluster": [CLUSTER_ARN]
}

and

2023-01-20T08:37:06.306Z	ca083a87-1c9f-4d5d-91d3-d0ddc1c9fd8f	INFO	changing desiredCount to 0
2023-01-20T08:37:06.389Z	ca083a87-1c9f-4d5d-91d3-d0ddc1c9fd8f	INFO	starting service 
{
    "desiredCount": 0,
    "service": [SERVICE_ARN],
    "cluster": [CLUSTER_ARN]
}

... but this is having no effect.

The code lines are:

const params = {
desiredCount: 1,
service: SERVICE_NAME,
cluster: CLUSTER_ARN
}

const updateCommand = new UpdateServiceCommand(params);
client.send(updateCommand).then(
(data) => {console.log(data);},
(err) => { console.log(err);}
);

These match up with use of the ECS client - docs here: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-ecs/classes/updateservicecommand.html

Updating cluster another way

As a sense check, you can use the aws cli to perform the same task:

aws ecs update-service --cluster [CLUSTER_ARN] --service [SERVICE_ARN] --desired-count [0|1]

After performing this command, the cluster responds. The serverstatus endpoint reflects the change and also the AWS console can be used to check, too.

This command is equivalent to what the Lambda is using - docs are here: https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html

Hypotheses

  • Is this a permissions issue? Does the startstop lambda have the correct Role, security group, ..., whatever it needs, ..., to perform this task?
  • Something else ...?
@rjaduthie
Copy link
Contributor Author

Worked it out. It isn't permissions: thankfully! The "startstop" lambda code doesn't await the resolution of the ECS client command - by adding the await keyword the endpoint works as expected.

See PR #35 for fix (and a couple of other improvements).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant