Create Staging and Production environments; Refactor and create necessary modules and resources #67

smokestacklightnin · 2024-10-16T10:13:09Z

This PR addresses Issues #60, #61, #62, and #63.

The following changes are made:

Create modules for a bootstrap stage that creates state resources for shared, stage, and prod environments
Create a shared network configuration
Create a shared ECR repository configuration
Create a shared IAM policy configuration for continuous deployment to ECR
Split EC2 configurations into staging and production environments

This PR should not break the current deployment because the deploy workflow is only run on tagged pushes.

Remove `README.md` because it outlines a manual process that is not automated by terraform/opentofu

The IAM roles might need to be modified before this PR is merged

smokestacklightnin · 2024-10-16T10:15:10Z

web/deploy/terraform/modules/ec2/main.tf

+  ami           = data.aws_ami.ubuntu.id
+  instance_type = var.instance_type
+  subnet_id     = data.terraform_remote_state.shared.outputs.subnet_id
+  # key_name                    = var.ec2_key_name


We might want separate keys for stage and prod

We discussed this on a call and we're going to be using a pre-existing key for both instances

smokestacklightnin · 2024-10-16T10:38:27Z

web/deploy/terraform/modules/iam/policies/gha-policy.json.tftpl

+            "Effect": "Allow",
+            "Action": [
+                "ec2:*",
+                "ecr:*",
+                "s3:*",
+                "dynamodb:*"
+            ],
+            "Resource": ${resources}
+        },
+        {
+            "Effect": "Allow",
+            "Action": "ecr:GetAuthorizationToken",
+            "Resource": "*"


These are too permissive, and we will refine them in the very near future

smokestacklightnin · 2024-10-16T11:23:12Z

web/deploy/terraform/staging/main.tf

-output "public_ip" {
-  value = aws_eip.staging.public_ip
+module "ec2" {
+  source        = "../modules/ec2/"


This will need to be changed to a github URL with a reference to a tag after this PR is merged

smokestacklightnin · 2024-10-16T11:23:22Z

web/deploy/terraform/production/main.tf

+}
+
+module "ec2" {
+  source        = "../modules/ec2/"


This will need to be changed to a github URL with a reference to a tag after this PR is merged

@smokestacklightnin why not leave it like this? What's the advantage of referencing the source as a GitHub URL?

If the ec2 config changes from stage to prod, this is how we can keep track of those changes

You mean the module changing? I'd say let's not overcomplicate things for now and keep the reference to the local module

leej3

Thanks. LGTM, but let's discuss with @agt24 and @marcelovilla before merging.

marcelovilla

@smokestacklightnin this looks good but we need to make some minor adjustements before merging it:

Uncomment elastic IP resource and output blocks
Use pre-existing key pair for both instances

I suggest we merge the PR after having deployed the infrastructure on the NIH AWS account

marcelovilla

@leej3 I'm approving this PR but I'd like for you to take a look at the non-Terraform changes that we're suggesting

leej3

Thanks. Looks good to me. I can debug any remaining issues in the python code post merge. I'll most likely do that as part of my WIP PR.

Can you add an explanation for the stored state files (perhaps compared to just storing them ourselves somewhere.

osm/schemas/schema_helpers.py

osm/__init__.py

web/deploy/terraform/state/terraform.tfstate

smokestacklightnin added 30 commits September 27, 2024 18:39

Move state storage modules to bootstrap step in anticipation of refactor

d35a9ee

Refactor state bootstrap to use variables and output modules

2beb465

Use variables in names

03630f1

Refactor versioning and encryption resources

ddd1b13

Add dynamodb policy

06b17f5

Remove `README.md` because it outlines a manual process that is not automated by terraform/opentofu

Add empty bootstrap modules for staging and production

216d0a9

Add bootstrap modules for state storage for staging and production

b726799

Temporarily comment out IAM policy resources

37c370b

Add required version

2d9ed40

Run validate before lint and format

59da31e

Refactor outputs to their own files

247a870

Remove logic for iam policy from bootstrap stage

9cff3b0

Refactor to move variables to separate file

a60882d

Rename development_environment variable to environmentx

1cbdbb6

Move all state bootstrap files to a single directory

f38eea2

Add deprecated files for reference

de9dce8

Reorganize modules

565b193

Add shared state storage

a518d60

Prevent destruction of state infrastructure

13dd738

Add .terraform.lock.hcl to version control

80b9258

Move state modules to appropriate directory

8011072

Move state modules to modules directory

8f372d1

Move network module files to appropriate location

66c2d27

Store network config values in variables

38a6ce6

Set up networking infrastructure config

dfd83e5

Fix typo

898d04b

Add ECR configuration

ce6440d

Add arn output

d941f0a

Use separate ECRs for API and dashboard

253b92d

Add basic IAM roles and infrastructure to be used by github actions

bc82e12

The IAM roles might need to be modified before this PR is merged

smokestacklightnin commented Oct 16, 2024

View reviewed changes

Remove debugging condition

b1644a2

smokestacklightnin force-pushed the infra/meta/deploy-staging-prod branch 2 times, most recently from 082a2eb to 16ca672 Compare October 16, 2024 10:35

Fix indentation

591dafa

smokestacklightnin force-pushed the infra/meta/deploy-staging-prod branch from 16ca672 to 591dafa Compare October 16, 2024 10:37

smokestacklightnin commented Oct 16, 2024

View reviewed changes

leej3 previously approved these changes Oct 16, 2024

View reviewed changes

marcelovilla suggested changes Oct 16, 2024

View reviewed changes

marcelovilla mentioned this pull request Oct 16, 2024

Document Infrastructure as Code approach #68

Open

smokestacklightnin added 2 commits October 16, 2024 13:07

Uncomment eip config

1eb2dbe

Uncomment key config

c3a06a1

smokestacklightnin dismissed leej3’s stale review via c3a06a1 October 16, 2024 20:08

marcelovilla added 4 commits October 23, 2024 20:52

Use larger instance

e227963

Build from local base image rather than from published image

dbf803d

Add instance profile with ECR read permission to EC2 instance

0c47f7d

Use larger instance for production

711e871

leej3 mentioned this pull request Oct 25, 2024

[WIP] Implement terraform bootstrap stage #65

Closed

marcelovilla added 4 commits October 28, 2024 10:00

Add new allowed subdomain

75705ec

Change search filter expression

8fe2c46

Fix os.getenv call

3707aa8

Add Terraform state files for state bootstrapping

34bfa7c

marcelovilla approved these changes Oct 28, 2024

View reviewed changes

leej3 self-requested a review October 28, 2024 10:24

leej3 previously approved these changes Oct 28, 2024

View reviewed changes

osm/schemas/schema_helpers.py Show resolved Hide resolved

osm/__init__.py Show resolved Hide resolved

web/deploy/terraform/state/terraform.tfstate Outdated Show resolved Hide resolved

Remove state and lock files

874576d

marcelovilla dismissed leej3’s stale review via 874576d October 28, 2024 11:40

leej3 merged commit 3a65859 into nimh-dsst:main Oct 29, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Staging and Production environments; Refactor and create necessary modules and resources #67

Create Staging and Production environments; Refactor and create necessary modules and resources #67

smokestacklightnin commented Oct 16, 2024

smokestacklightnin Oct 16, 2024

marcelovilla Oct 16, 2024

smokestacklightnin Oct 16, 2024

smokestacklightnin Oct 16, 2024

smokestacklightnin Oct 16, 2024

marcelovilla Oct 16, 2024

smokestacklightnin Oct 16, 2024

marcelovilla Oct 16, 2024

leej3 left a comment

marcelovilla left a comment

marcelovilla left a comment

leej3 left a comment

Create Staging and Production environments; Refactor and create necessary modules and resources #67

Create Staging and Production environments; Refactor and create necessary modules and resources #67

Conversation

smokestacklightnin commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leej3 left a comment

Choose a reason for hiding this comment

marcelovilla left a comment

Choose a reason for hiding this comment

marcelovilla left a comment

Choose a reason for hiding this comment

leej3 left a comment

Choose a reason for hiding this comment