Create basic CDK support and minimalist demo stack #6

pcholakov · 2023-11-20T13:21:38Z

This change introduces a CDK project with several reusable constructs, and a demonstration stack that allows customers to easily deploy a self-hosted CDK stack and Lambda-based service handlers to AWS account.

As this is still a private repository (see linked GH issue for planned work), at the moment comment is primarily sought along these dimensions:

Meta: this is not a tiny PR; happy to rebase and split it into more logical chunks if it's too much to digest
README - what basic pointers would we like to give a customer who is reasonably proficient at AWS (and specifically CDK and Lambda/the AWS Serverless stack), but new to Restate?
Naming, granularity, and scope of responsibility of various components? ()
Is RestateSelfHostedStack (restate-self-hosted-stack.ts) simple and intuitive to you? (Imagine that in the future, a customer would write this code + their handler code only, while the Restate constructs will be provided as a library)

Testing

"Works on my machine"

❯ npx cdk deploy --require-approval never --no-rollback
...

pavel-RestateStack: deploying... [1/1]
pavel-RestateStack: creating CloudFormation changeset...

 ✅  pavel-RestateStack

✨  Deployment time: 365.63s

Outputs:
pavel-RestateStack.RestateIngressEndpoint = http://pavel--Resta-ZioBu8Ip87RB-1719749793.eu-central-1.elb.amazonaws.com:80
pavel-RestateStack.RestateServiceApiEndpoint61395974 = https://zabs4h8qa3.execute-api.eu-central-1.amazonaws.com/prod/
Stack ARN:
arn:aws:cloudformation:eu-central-1:663487780041:stack/pavel-RestateStack/a5769500-878b-11ee-bb73-064c3e60e9bf

✨  Total time: 378.22s

❯ export RESTATE_INGRESS_ENDPOINT=$(aws cloudformation describe-stacks --stack-name ${USER}-RestateStack \
    --query "Stacks[0].Outputs[?OutputKey=='RestateIngressEndpoint'].OutputValue" --output text)

❯ curl -X POST -w \\n ${RESTATE_INGRESS_ENDPOINT}/Greeter/greet -H 'content-type: application/json' -d '{"key": "Restate"}'

{"response":"Hello, Restate! :-)"}

Issue: #1
Closes: #3

…covery after deployment

…mbda packages)

jackkleeman

love it. lets get this in and iterate from there!

jackkleeman · 2023-11-20T17:24:11Z

lib/restate-constructs/lambda/register-service-handler.ts

+
+      const controller = new AbortController();
+      const healthCheckTimeout = setTimeout(() => controller.abort(), 3000);
+      const healthCheckUrl = `${props.ingressEndpoint}/grpc.health.v1.Health/Check`;


I think we should be using the health endpoint on the meta here, right?

You're quite right, for pre-registration the meta health check is probably the one that matters! I'm going to revisit this entire handler to make it more robust but switching over to meta health check for now. (Separately, the ALB is continuously monitoring the ingress health check.)

jackkleeman · 2023-11-20T17:24:57Z

lib/restate-constructs/lambda/register-service-handler.ts

+      console.log(`Got registration response back: ${await discoveryResponse.text()} (${discoveryResponse.status})`);
+
+      if (!(healthResponse.status >= 200 && healthResponse.status < 300)) {
+        // TODO: retry until successful, or some overall timeout is reached


if only we had some kind of orchestrator that did this for us ;)

jackkleeman · 2023-11-20T17:28:40Z

lib/restate-constructs/single-node-restate-instance.ts

+   * The name of the Secrets Store secret that contains the GitHub token to use for Docker login.
+   * This is `/restate/docker/github-token` by default.
+   */
+  githubTokenPath?: string;


can be a followup, but i think we just made dist publicaly accessible :)

Yeah, bit of a waste! But it was useful to remember how to do this in CDK – it may yet come up again. Something else I want to investigate: if we can push the Restate image into an ECR registry, perhaps we can eliminate the internet connectivity for the Restate VPC altogether. That would be a big security and cost win.

jackkleeman · 2023-11-20T17:29:03Z

lib/restate-constructs/single-node-restate-instance.ts

+  githubTokenPath?: string;
+
+  /**
+   * Temporary! This will disappear once we move to direct Lambda calls.


can be a followup - but latest dist now does support direct calls

Ack! I'll iterate on that as a follow up; I still have something weird going on with the discovery resource that I want to get to the bottom of, I'll update to direct Lambda integration asap after I figure it out.

Ended up doing this now, the latest branch changes switch to direct integration.

jackkleeman · 2023-11-20T17:30:14Z

lib/restate-constructs/single-node-restate-instance.ts

+    // These rules allow the service registration component to trigger service discovery as needed; the requests
+    // originate from a VPC-bound Lambda function that backs the custom resource.
+    this.vpc.privateSubnets.forEach((subnet) => {
+      restateInstanceSecurityGroup.addIngressRule(ec2.Peer.ipv4(subnet.ipv4CidrBlock), ec2.Port.tcp(RESTATE_META_PORT), "Allow traffic from the VPC to Restate meta");


probably will want also 9071 and 9072 (worker http and postgres endpoints)

open question how introspection should work - vpn vs punch through load balancer. if punch through, then need some sort of auth story for the postgres endopint, cant just use aws stuff there

One quick and secure way might be to allow customers to start an interactive psql session using Session Manager? Not ideal for all users but better than opening an unauthenticated endpoint to the internet.

…apping

jackkleeman

lgtm!

jackkleeman · 2023-11-21T15:54:26Z

lib/restate-constructs/lambda/register-service-handler.ts

@@ -43,7 +42,8 @@ export const handler: Handler<CloudFormationCustomResourceEvent, Partial<CloudFo

      const registerCallTimeout = setTimeout(() => controller.abort(), 3000);
      const discoveryEndpointUrl = `${props.metaEndpoint}/endpoints`;
-      const registrationRequest = JSON.stringify({ uri: props.serviceEndpoint });
+      // const registrationRequest = JSON.stringify({ uri: props.serviceEndpoint });


nit; can clean this up

Doh, manage to miss this! Will clean up in the next PR.

pcholakov requested review from gvdongen and jackkleeman November 20, 2023 13:21

pcholakov added 15 commits November 20, 2023 16:12

Add minimal EC2 instance to host Restate daemon

aae5227

Add API Gateway for Lambda backend

7ec36bd

Use variable prefix from CDK context or env for stack name

9b3d463

Add insecure internet-facing ALB exposing the Restate ingress endpoint

b589066

Add custom resource that registers the Lambda handler via Restate dis…

c7ee2bd

…covery after deployment

Fix: log discovery endpoint in custom resource handler

113c6fb

Cleanup: remove some unnecessary outputs

44e7a25

Discover the concrete AMI to use

9d3bc7a

Switch to Secrets Manager secret for passing the GitHub token into EC2

69262bf

Send Restate JSON logs to CloudWatch logs

1ada6ed

Refactoring: Extract custom Restate constructs for all major components

b97a0bd

Update README with deployment & test instructions

2983cf7

Add esbuild dev dependency (avoids Docker requirement for bundling La…

642761d

…mbda packages)

Minor cleanups and comment+TODO updates

1da5f47

Allow Restate to invoke handlers via explicit version qualifier

8831b6d

pcholakov force-pushed the pavel/initial branch from ed4005b to 8831b6d Compare November 20, 2023 14:15

jackkleeman approved these changes Nov 20, 2023

View reviewed changes

pcholakov added 3 commits November 21, 2023 10:37

Ack warning - and actually silence it

4d450f5

Switch to native Lambda invoke integration, remove API Gateway HTTP m…

cf1b009

…apping

Remove GitHub token secret as no longer needed

4496b93

pcholakov removed the request for review from gvdongen November 21, 2023 13:20

jackkleeman approved these changes Nov 21, 2023

View reviewed changes

pcholakov merged commit 4750047 into main Nov 22, 2023

jackkleeman deleted the pavel/initial branch November 22, 2023 22:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create basic CDK support and minimalist demo stack #6

Create basic CDK support and minimalist demo stack #6

pcholakov commented Nov 20, 2023 •

edited

Loading

jackkleeman left a comment

jackkleeman Nov 20, 2023

pcholakov Nov 21, 2023

jackkleeman Nov 20, 2023

jackkleeman Nov 20, 2023

pcholakov Nov 21, 2023

jackkleeman Nov 20, 2023

pcholakov Nov 21, 2023

pcholakov Nov 21, 2023

jackkleeman Nov 20, 2023

jackkleeman Nov 20, 2023

pcholakov Nov 21, 2023

jackkleeman left a comment

jackkleeman Nov 21, 2023

pcholakov Nov 22, 2023

Create basic CDK support and minimalist demo stack #6

Create basic CDK support and minimalist demo stack #6

Conversation

pcholakov commented Nov 20, 2023 • edited Loading

jackkleeman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackkleeman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcholakov commented Nov 20, 2023 •

edited

Loading