From f92219b3c0bd2319211077b1ad92a5f34dcf0bc8 Mon Sep 17 00:00:00 2001 From: "Steven J. Miklovic" Date: Tue, 2 Jul 2024 16:11:45 -0400 Subject: [PATCH 01/14] Create README.md --- infrastructure-documenation/deployment/README.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 infrastructure-documenation/deployment/README.md diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/infrastructure-documenation/deployment/README.md @@ -0,0 +1 @@ + From 96b727354efbd040e861eb0c167d3b1e3e5e0302 Mon Sep 17 00:00:00 2001 From: "Steven J. Miklovic" Date: Tue, 2 Jul 2024 16:54:37 -0400 Subject: [PATCH 02/14] Update README.md --- .../deployment/README.md | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 8b13789..fb2cf51 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -1 +1,81 @@ +# Deployment.md + +--- + +## Overview + +This guide covers the deployment architecture and release process for the PASS (Portable Application Security Solution) application. PASS is designed to be flexible and can adapt to various architectures, including cloud infrastructure, hybrid, or on-premises environments. + + +## Technical Deep Dive + +excerpt from [../welcome-guide/deployment-architecture.md] +### Deployment Architecture + +The deployment workflow starts when developers contribute code to the Eclipse PASS Git repository. Changes in this repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline facilitating the deployment of the updated code. In deployment SQS is utilized for initiating the deployment from a GitHub workflow that publishes a SNS topic to the queue. The PASS deployment files contain configurations and environment variables that assist in the deployment of the PASS application and supporting services. Liquibase is utilized to manage database schema changes, which interacts with an AWS RDS instance running PostgreSQL. + +For other organizations looking to adopt a similar AWS application and deployment model, it's important to recognize that while the core architecture offers a template, it should be adapted to meet an organization's own requirements and needs. Each organization will need to evaluate its own application demands, data sensitivity, and user base to tailor the cloud resources, network configurations, and security policies accordingly. Moreover, integrating other types of cloud infrastructure or even on-premise solutions might be necessary to address specific technological preferences or regulatory requirements. Hybrid cloud environments or multi-cloud strategies could be employed to leverage the strengths of various cloud providers, enhance resilience, and avoid vendor lock-in. PASS is designed to be flexible and can adapt to a variety of architectures; whether a cloud infrastructure, hybrid or on-premise. + +--- +excerpt from [../docs/dev/release.md] +## Release Process + +A PASS release produces a set of Java artifacts, and Docker images. +Java artifacts are pushed to Sonatype Nexus and Maven Central repositories. Docker images are pushed to GitHub Container Registry. + +The PASS Docker environment is updated with references to the released images. +Source code is tagged and release notes made available. + +Each release of PASS has its own version which is used by every component. PASS uses `MAJOR.MINOR.PATCH` [semantic versioning](https://semver.org/) approach. + +--- +excerpt from [../docs/dev/release-steps-with-automation.md] +#### GitHub Actions Workflow +. + +##### Release All Projects + +The [Publish: Release All](/.github/workflows/pass-complete-release.yml) is a GitHub workflow that will release all the PASS projects in the correct order, build and push docker images, and create GitHub Releases. + +--- +excerpt from [../docs/infra/digitalocean.md] +## Infrastructure + +The current PASS infrastructure includes the following components: + +* EC2 + +* ECS + +* RDS + +* S3 + +* ALB + +* WAF + +#### Installing PASS Dependencies + +On the server, you will need [docker / docker-compose](https://docs.docker.com/compose/) and [pass-docker](https://github.com/eclipse-pass/pass-docker) + + +```bash +apt-get -y update +apt-get install -y gnupg2 pass docker-compose +mkdir -p /src +cd /src +git clone git@github.com:eclipse-pass/pass-docker.git +cd pass-docker +git checkout minimal-assets +``` + +#### Running PASS + +```bash +cd /src/pass-docker && \ + docker-compose pull && \ + docker-compose up +``` + From 9439de66e8385cad7f023d08d41488a0dca71f38 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 12:34:43 -0400 Subject: [PATCH 03/14] Update deployment/README.md --- .../deployment/README.md | 134 ++++++++++++++---- 1 file changed, 105 insertions(+), 29 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index fb2cf51..841e802 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -4,12 +4,13 @@ ## Overview -This guide covers the deployment architecture and release process for the PASS (Portable Application Security Solution) application. PASS is designed to be flexible and can adapt to various architectures, including cloud infrastructure, hybrid, or on-premises environments. +This guide covers the deployment architecture and release process for the PASS (Public Access Submission System) application. PASS is designed to be flexible and can adapt to various architectures, including cloud infrastructure, hybrid, or on-premises environments. ## Technical Deep Dive excerpt from [../welcome-guide/deployment-architecture.md] + ### Deployment Architecture The deployment workflow starts when developers contribute code to the Eclipse PASS Git repository. Changes in this repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline facilitating the deployment of the updated code. In deployment SQS is utilized for initiating the deployment from a GitHub workflow that publishes a SNS topic to the queue. The PASS deployment files contain configurations and environment variables that assist in the deployment of the PASS application and supporting services. Liquibase is utilized to manage database schema changes, which interacts with an AWS RDS instance running PostgreSQL. @@ -17,45 +18,30 @@ The deployment workflow starts when developers contribute code to the Eclipse PA For other organizations looking to adopt a similar AWS application and deployment model, it's important to recognize that while the core architecture offers a template, it should be adapted to meet an organization's own requirements and needs. Each organization will need to evaluate its own application demands, data sensitivity, and user base to tailor the cloud resources, network configurations, and security policies accordingly. Moreover, integrating other types of cloud infrastructure or even on-premise solutions might be necessary to address specific technological preferences or regulatory requirements. Hybrid cloud environments or multi-cloud strategies could be employed to leverage the strengths of various cloud providers, enhance resilience, and avoid vendor lock-in. PASS is designed to be flexible and can adapt to a variety of architectures; whether a cloud infrastructure, hybrid or on-premise. --- -excerpt from [../docs/dev/release.md] -## Release Process - -A PASS release produces a set of Java artifacts, and Docker images. -Java artifacts are pushed to Sonatype Nexus and Maven Central repositories. Docker images are pushed to GitHub Container Registry. - -The PASS Docker environment is updated with references to the released images. -Source code is tagged and release notes made available. -Each release of PASS has its own version which is used by every component. PASS uses `MAJOR.MINOR.PATCH` [semantic versioning](https://semver.org/) approach. +excerpt from [../docs/infra/digitalocean.md] ---- -excerpt from [../docs/dev/release-steps-with-automation.md] -#### GitHub Actions Workflow -. +## Deployment Infrastructure -##### Release All Projects +The current PASS infrastructure is running in Amazon Web Service (AWS). It includes the following core components in its stack: -The [Publish: Release All](/.github/workflows/pass-complete-release.yml) is a GitHub workflow that will release all the PASS projects in the correct order, build and push docker images, and create GitHub Releases. +* EC2 - Amazon Elastic Compute Cloud. ---- -excerpt from [../docs/infra/digitalocean.md] -## Infrastructure +* ECS - Amazon Elastic Container Service. -The current PASS infrastructure includes the following components: +* RDS - Amazon Relational Database Service. -* EC2 +* S3 - Amazon S3. -* ECS +* ALB - Amazon Application Load Balancer. -* RDS +* WAF - AWS Web Application Firewall. -* S3 +## Provisioning PASS -* ALB +### Installing PASS Dependencies -* WAF - -#### Installing PASS Dependencies +#### VM/EC2 Instructions Using Docker Compose On the server, you will need [docker / docker-compose](https://docs.docker.com/compose/) and [pass-docker](https://github.com/eclipse-pass/pass-docker) @@ -70,12 +56,102 @@ cd pass-docker git checkout minimal-assets ``` -#### Running PASS +### Runing PASS +#### VM/EC2 Instructions Using Docker Compose ```bash cd /src/pass-docker && \ docker-compose pull && \ docker-compose up ``` +--- +excerpt from [../docs/dev/release.md] + +## Release Process + +A PASS release produces a set of Java artifacts, and Docker images. + +Each release of PASS has its own version which is used by every component. PASS uses `MAJOR.MINOR.PATCH` [semantic versioning](https://semver.org/) approach. The PASS Docker environment is updated with references to the released images. +Source code is tagged and release notes made available. + +The PASS release process involves the following steps: + +Code Contribution : Developers contribute code changes to the PASS Git repository. + +Continuous Integration (CI) : Changes in the repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline. + +Build and Test : The CI pipeline builds and tests the PASS application, ensuring code quality and functionality. + +Release Artifacts : Upon successful testing, the pipeline generates release artifacts, including Java artifacts and Docker images. + +Artifact Publishing : Java artifacts are published to Sonatype Nexus and Maven Central repositories, while Docker images are pushed to the GitHub Container Registry. + +Deployment Trigger : The CI pipeline publishes a message to an Amazon Simple Queue Service (SQS) queue, initiating the deployment process. + +Infrastructure Updates : The deployment process updates the PASS infrastructure with the latest release artifacts, including updating the ECS cluster with the new Docker images. + +Release Notes : Release notes are generated and made available to stakeholders. + + +--- +excerpt from [../docs/dev/release-steps-with-automation.md] + +#### GitHub Actions +. + +##### Workflow: Release All Projects + +The [Publish: Release All](/.github/workflows/pass-complete-release.yml) is a GitHub workflow that will release all the PASS projects in the correct order, build and push docker images, and create GitHub Releases. + +This workflow performs the following tasks: + +1. Checks out the PASS repository code. + +2. Sets up the build environment (e.g., Java, Docker). + +3. Builds and tests the PASS application components. + +4. Publishes Java artifacts to Sonatype Nexus and Maven Central. + +5. Builds and pushes Docker images to the GitHub Container Registry. + +6. Creates GitHub Releases with release notes and artifacts. + +# Example snippet from the "Publish: Release All" workflow +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Set up JDK + uses: actions/setup-java@v3 + with: + java-version: '11' + distribution: 'adopt' + - name: Build and Release + run: | + ./mvnw clean install + ./mvnw deploy -Prelease + - name: Build and Push Docker Images + run: | + docker build -t pass/app:${{ github.sha }} . + docker push pass/app:${{ github.sha }} + - name: Create GitHub Release + uses: actions/create-release@v1 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + with: + tag_name: v${{ github.ref }} + release_name: Release ${{ github.ref }} + body: ${{ steps.release-notes.outputs.content }} + +For more detailed information and configuration, refer to the pass-complete-release.yml file in the AWS-PASS-Deploymnet repository. + +## Additional Resources + +PASS Documentation + +PASS Docker Repository +For further assistance or questions, please _______. \ No newline at end of file From fb494c74f4ad07451ad34f6bbf5e1a19168a6165 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 12:53:59 -0400 Subject: [PATCH 04/14] Add github-cicd.md --- .../deployment/github-cicd.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 infrastructure-documenation/deployment/github-cicd.md diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md new file mode 100644 index 0000000..a60648d --- /dev/null +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -0,0 +1,77 @@ +## GitHub Secrets + +GitHub secrets are intended for use by GitHub Actions workflows. If you want a workflow to interact with something that requires authentication, you will likely need to create a secret which will allow authentication to happen securely without making any private information available to anybody having access to the repositories. + +There are three levels that GitHub secrets can exist at: organization, repo, and environment. + +An organization secret belongs to the `eclipse-pass` organization and is available to workflows that are part of any repository that exists under the `eclipse-pass` organization. Secrets that will be used across many repositories should be placed here to prevent duplication. + +A repository secret exists within that repository and can only be used by workflows belonging to that repository. + +An environment secret exists within a specific environment within a repository. It can only be used by workflows belonging to that repository and the workflow must specify the environment that it's using. + +### Creating Secrets + +GitHub offers two mechanisms for creating secrets: UI and API. + +The GitHub UI requires that a person creating a secret have admin privileges for that entity (either org or repo). Since members of the PASS project don't have admin privileges at any level, we need to open a ticket with Eclipse to add secrets through the UI. + +Oddly, the GitHub API allows the creation of repository secrets with `repo` privileges. All members of the PASS project should have sufficient privileges to do so. + +To create a repository or environment secret, use the [github_secrets.py script](/tools/github_secrets.py) in this repo like so: +``` +python github_secrets.py -u -t -r -n -v [-e ] +``` +* username: Your GitHub username +* token: A GitHub personal access token with 'repo' access +* repo: The name of the GitHub repository containing the secret +* name: Name of the secret being created or updated +* value: The value to be set for the secret. The value will be encrypted before being sent to GitHub. + +To create an organization secret, a ticket must be opened with the [Eclipse Help Desk](https://gitlab.eclipse.org/eclipsefdn/helpdesk). + +### Reading Secrets + +The value of a secret can only be read by GitHub Actions workflows. + +The metadata associated with a secret can be retrieved with the `github_secrets.py` script by omitting the `value`. The returned data includes the secret's `name`, `created_at`, and `updated_at`. This is primarily useful for verifying that a secret exists where you expect it to be. + +### Secrets in GitHub Actions workflows + +To access a GitHub secret from a top-level workflow, you can reference it from within the secrets context like this: +``` +${{ secrets.KUBE_CONFIG }} +``` + +If you want to use a secret in a reusable or "called" workflow, you must pass the secrets that will be used. To do this, when you reference the workflow you additionally provide a `secrets` property containing a list of the secrets that must be passed. + +In this example two secrets, KUBE_CONFIG and DIGITALOCEAN_ACCESS_TOKEN are being passed to the `html-app-publish.yml` workflow: + +``` +call-publish-docker: + name: Build and Publish Docker image + uses: eclipse-pass/playground/.github/workflows/html-app-publish.yml@main + secrets: + KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }} + DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }} +``` + +Within the called workflow, you must declare the secrets that can be expected like this: + +``` +on: + workflow_call: + secrets: + KUBE_CONFIG: + required: true + description: Configuration info for test k8s cluster + DIGITALOCEAN_ACCESS_TOKEN: + required: true + description: Access token for Digital Ocean +``` + +The secret can now be referenced within the workflow as it would be from a top-level workflow: + +``` +${{ secrets.KUBE_CONFIG }} +``` From 34ad122e88b6e6e7b05865fe0b9e5fde3131eb2b Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 12:54:11 -0400 Subject: [PATCH 05/14] Update README --- .../deployment/README.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 841e802..e44eddd 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -6,6 +6,9 @@ This guide covers the deployment architecture and release process for the PASS (Public Access Submission System) application. PASS is designed to be flexible and can adapt to various architectures, including cloud infrastructure, hybrid, or on-premises environments. +## Going Forward + +As we move towards a cloud-native version of PASS there will be ongoing changes to the architecture, changes to the infrastructure, and changes to the deployment process, e.g. transitioning from Docker Compose to Kubernetes, or using Terraform for IAC. ## Technical Deep Dive @@ -25,17 +28,17 @@ excerpt from [../docs/infra/digitalocean.md] The current PASS infrastructure is running in Amazon Web Service (AWS). It includes the following core components in its stack: -* EC2 - Amazon Elastic Compute Cloud. +* EC2 - Amazon Elastic Compute Cloud. Hosts Docker Compose. -* ECS - Amazon Elastic Container Service. +* ECS - Amazon Elastic Container Service. Hosts auxiliary microservices. -* RDS - Amazon Relational Database Service. +* RDS - Amazon Relational Database Service. Stores metadata. -* S3 - Amazon S3. +* S3 - Amazon S3. Used to store binary data that is managed by OCFL. -* ALB - Amazon Application Load Balancer. +* ALB - Amazon Application Load Balancer provides SSL to the frontend. -* WAF - AWS Web Application Firewall. +* WAF - AWS Web Application Firewall. Protects the frontend. ## Provisioning PASS @@ -98,7 +101,8 @@ Release Notes : Release notes are generated and made available to stakeholders. excerpt from [../docs/dev/release-steps-with-automation.md] #### GitHub Actions -. + +* [Working with GitHub CICD](github-cicd.md) ##### Workflow: Release All Projects @@ -150,8 +154,8 @@ For more detailed information and configuration, refer to the pass-complete-rele ## Additional Resources -PASS Documentation +[PASS main](https://github.com/eclipse-pass/main) -PASS Docker Repository +[PASS Docker Repository](https://github.com/eclipse-pass/pass-docker) For further assistance or questions, please _______. \ No newline at end of file From dd084fcf129b40908c01b934cafd1fe0e8a69244 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 22:18:13 -0400 Subject: [PATCH 06/14] Add github-cicd.md --- .../deployment/github-cicd.md | 163 +++++++++++++----- 1 file changed, 119 insertions(+), 44 deletions(-) diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md index a60648d..8001dc6 100644 --- a/infrastructure-documenation/deployment/github-cicd.md +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -1,77 +1,152 @@ -## GitHub Secrets +# GitHub CI/CD Guide -GitHub secrets are intended for use by GitHub Actions workflows. If you want a workflow to interact with something that requires authentication, you will likely need to create a secret which will allow authentication to happen securely without making any private information available to anybody having access to the repositories. +## Overview of GitHub Actions and Workflows -There are three levels that GitHub secrets can exist at: organization, repo, and environment. +GitHub Actions is a powerful CI/CD platform integrated directly into GitHub repositories. It allows you to automate various software development workflows, including building, testing, and deploying your code. -An organization secret belongs to the `eclipse-pass` organization and is available to workflows that are part of any repository that exists under the `eclipse-pass` organization. Secrets that will be used across many repositories should be placed here to prevent duplication. +### Key Concepts -A repository secret exists within that repository and can only be used by workflows belonging to that repository. +1. **Workflows**: YAML files that define a set of jobs to be executed when triggered by an event. +2. **Jobs**: A set of steps that execute on the same runner. +3. **Steps**: Individual tasks that can run commands or actions. +4. **Actions**: Reusable units of code that can be shared across workflows. +5. **Events**: Specific activities that trigger a workflow run. -An environment secret exists within a specific environment within a repository. It can only be used by workflows belonging to that repository and the workflow must specify the environment that it's using. +### Basic Workflow Structure -### Creating Secrets +```yaml +name: CI -GitHub offers two mechanisms for creating secrets: UI and API. +on: [push, pull_request] -The GitHub UI requires that a person creating a secret have admin privileges for that entity (either org or repo). Since members of the PASS project don't have admin privileges at any level, we need to open a ticket with Eclipse to add secrets through the UI. +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Run a script + run: echo Hello, world! +``` -Oddly, the GitHub API allows the creation of repository secrets with `repo` privileges. All members of the PASS project should have sufficient privileges to do so. +This simple workflow runs on push and pull request events, checks out the repository, and runs a simple command. -To create a repository or environment secret, use the [github_secrets.py script](/tools/github_secrets.py) in this repo like so: -``` -python github_secrets.py -u -t -r -n -v [-e ] -``` -* username: Your GitHub username -* token: A GitHub personal access token with 'repo' access -* repo: The name of the GitHub repository containing the secret -* name: Name of the secret being created or updated -* value: The value to be set for the secret. The value will be encrypted before being sent to GitHub. +## GitHub Secrets -To create an organization secret, a ticket must be opened with the [Eclipse Help Desk](https://gitlab.eclipse.org/eclipsefdn/helpdesk). +GitHub secrets are encrypted environment variables used to store sensitive information securely. They are crucial for handling authentication and other confidential data in your workflows. -### Reading Secrets +### Types of Secrets -The value of a secret can only be read by GitHub Actions workflows. +1. **Organization Secrets**: Available to all repositories in the `eclipse-pass` organization. +2. **Repository Secrets**: Specific to a single repository. +3. **Environment Secrets**: Tied to a specific environment within a repository. -The metadata associated with a secret can be retrieved with the `github_secrets.py` script by omitting the `value`. The returned data includes the secret's `name`, `created_at`, and `updated_at`. This is primarily useful for verifying that a secret exists where you expect it to be. +### Creating Secrets -### Secrets in GitHub Actions workflows +Due to permission restrictions, PASS project members should use the provided Python script to create repository or environment secrets: -To access a GitHub secret from a top-level workflow, you can reference it from within the secrets context like this: -``` -${{ secrets.KUBE_CONFIG }} +```bash +python github_secrets.py -u -t -r -n -v [-e ] ``` -If you want to use a secret in a reusable or "called" workflow, you must pass the secrets that will be used. To do this, when you reference the workflow you additionally provide a `secrets` property containing a list of the secrets that must be passed. +For organization secrets, open a ticket with the [Eclipse Help Desk](https://gitlab.eclipse.org/eclipsefdn/helpdesk). + +### Using Secrets in Workflows -In this example two secrets, KUBE_CONFIG and DIGITALOCEAN_ACCESS_TOKEN are being passed to the `html-app-publish.yml` workflow: +Reference secrets in your workflows like this: +```yaml +${{ secrets.SECRET_NAME }} ``` -call-publish-docker: - name: Build and Publish Docker image - uses: eclipse-pass/playground/.github/workflows/html-app-publish.yml@main + +For reusable workflows, pass secrets explicitly: + +```yaml +jobs: + call-publish-docker: + uses: ./.github/workflows/docker-publish.yml secrets: - KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }} - DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} ``` -Within the called workflow, you must declare the secrets that can be expected like this: +In the called workflow, declare expected secrets: -``` +```yaml on: - workflow_call: + workflow_call: secrets: - KUBE_CONFIG: + AWS_ACCESS_KEY_ID: required: true - description: Configuration info for test k8s cluster - DIGITALOCEAN_ACCESS_TOKEN: + AWS_SECRET_ACCESS_KEY: required: true - description: Access token for Digital Ocean ``` -The secret can now be referenced within the workflow as it would be from a top-level workflow: +## AWS / EKS Integration +To interact with AWS services, including EKS (Elastic Kubernetes Service), you'll need to set up appropriate secrets and use AWS-specific actions in your workflows. + +### Setting up AWS Credentials + +Store your AWS credentials as secrets: + +1. `AWS_ACCESS_KEY_ID` +2. `AWS_SECRET_ACCESS_KEY` + +### Example Workflow for EKS Deployment + +```yaml +name: Deploy to EKS + +on: + push: + branches: [main] + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + + - name: Login to Amazon ECR + id: login-ecr + uses: aws-actions/amazon-ecr-login@v1 + + - name: Build, tag, and push image to Amazon ECR + env: + ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} + ECR_REPOSITORY: my-ecr-repo + IMAGE_TAG: ${{ github.sha }} + run: | + docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG . + docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG + + - name: Update kube config + run: aws eks get-token --cluster-name my-cluster | kubectl apply -f - + + - name: Deploy to EKS + run: | + kubectl set image deployment/my-app my-app=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG + kubectl rollout status deployment/my-app ``` -${{ secrets.KUBE_CONFIG }} -``` + +This workflow builds a Docker image, pushes it to Amazon ECR, and deploys it to an EKS cluster. + +Remember to replace placeholder values like `my-cluster`, `my-ecr-repo`, and `my-app` with your actual resource names. + +## Best Practices + +1. Use reusable workflows for common tasks to maintain DRY principles. +2. Leverage GitHub-hosted runners when possible to reduce maintenance overhead. +3. Use environment protection rules for sensitive deployments. +4. Regularly audit and rotate your secrets. +5. Use GitHub Actions marketplace for pre-built actions to speed up development. + +For more detailed information, refer to the [GitHub Actions documentation](https://docs.github.com/en/actions). +``` \ No newline at end of file From e310e02107c02fef0d2c4dda726be1ff755a366f Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 22:18:21 -0400 Subject: [PATCH 07/14] Add roadmap.md --- .../deployment/roadmap.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 infrastructure-documenation/deployment/roadmap.md diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md new file mode 100644 index 0000000..07a7454 --- /dev/null +++ b/infrastructure-documenation/deployment/roadmap.md @@ -0,0 +1,113 @@ +# Eclipse PASS Cloud-Native Kubernetes Deployment Roadmap + +## Table of Contents +1. [Introduction](#introduction) +2. [Current State](#current-state) +3. [Target Architecture](#target-architecture) +4. [Roadmap Phases](#roadmap-phases) +5. [Key Milestones](#key-milestones) +6. [Challenges and Considerations](#challenges-and-considerations) +7. [Timeline](#timeline) + +## Introduction + +This document outlines the roadmap for transitioning the Eclipse Public Access Submission System (PASS) from its current Docker Compose-based deployment to a cloud-native Kubernetes deployment. This transition aims to improve scalability, resilience, and manageability of the PASS platform. + +## Current State + +- Deployment primarily uses Docker Compose +- Infrastructure hosted on AWS (EC2, ECS, RDS, S3, ALB, WAF) +- Manual scaling and management of components + +## Target Architecture + +The target architecture will leverage Kubernetes for orchestration and cloud-native principles: + +- Kubernetes cluster (e.g., Amazon EKS, self-managed K8s on EC2) +- Containerized microservices +- Helm charts for deployment management + +### Future Possibilities + +- Prometheus and Grafana for monitoring +- Istio for service mesh +- CI/CD pipeline for Kubernetes deployment + +## Roadmap Phases + +### Phase 1: Preparation and Planning +- Analyze current architecture and identify components for containerization +- Define Kubernetes resource requirements +- Plan data migration strategy +- Identify necessary changes to application code for Kubernetes compatibility + +### Phase 2: Containerization and Kubernetes Basics +- Containerize all PASS components +- Create initial Kubernetes manifests (Deployments, Services, ConfigMaps) +- Set up a test Kubernetes cluster +- Implement basic health checks and readiness probes + +### Phase 3: Kubernetes Integration and Testing +- Develop Helm charts for PASS components +- Integrate persistent storage solutions (e.g., persistent volumes for databases) +- Implement secrets management +- Set up ingress controllers and configure routing +- Conduct thorough testing of the Kubernetes deployment + +### Phase 4: Advanced Kubernetes Features and Optimization +- Implement horizontal pod autoscaling +- Set up cluster autoscaling +- Optimize resource requests and limits +- Implement pod disruption budgets for high availability +- Configure liveness probes for self-healing + +### Phase 5: Monitoring, Logging, and Security +- Set up Prometheus and Grafana for monitoring +- Implement centralized logging (e.g., ELK stack) +- Configure network policies for enhanced security +- Implement pod security policies + +### Phase 6: CI/CD Pipeline Integration +- Adapt CI/CD pipelines for Kubernetes deployment +- Implement GitOps practices with tools like ArgoCD or Flux +- Set up canary deployments and rollback mechanisms + +### Phase 7: Production Migration and Optimization +- Perform staged migration to production Kubernetes environment +- Optimize performance and resource utilization +- Conduct load testing and fine-tuning +- Document new deployment and management processes + +## Key Milestones + +1. All PASS components successfully containerized +2. Basic Kubernetes deployment functional in test environment +3. Helm charts created and tested for all components +4. Monitoring and logging solutions implemented +5. CI/CD pipeline fully integrated with Kubernetes deployment +6. Production environment migrated to Kubernetes +7. Performance optimization and fine-tuning completed + +## Challenges and Considerations + +- Ensuring data integrity during migration +- Managing stateful applications in Kubernetes (e.g., databases) +- Balancing resource allocation and cost optimization +- Training team members on Kubernetes and cloud-native technologies +- Ensuring security and compliance in the new architecture + +## Timeline + +- Phase 1: 1-2 months +- Phase 2: 2-3 months +- Phase 3: 3-4 months +- Phase 4: 2-3 months +- Phase 5: 2-3 months +- Phase 6: 1-2 months +- Phase 7: 2-3 months + +Total estimated timeline: 13-20 months + +Note: This timeline is approximate and may need adjustment based on team capacity, unforeseen challenges, and organizational priorities. + +--- \ No newline at end of file From 3be24fc7ffdcb09952343235c2b544e6342214dd Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 22:18:33 -0400 Subject: [PATCH 08/14] Revise README.md --- .../deployment/README.md | 201 ++++++------------ 1 file changed, 70 insertions(+), 131 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index e44eddd..3ed2777 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -1,161 +1,100 @@ -# Deployment.md +# Eclipse PASS Deployment Guide ---- +## Table of Contents +1. [Overview](#overview) +2. [Deployment Architecture](#deployment-architecture) +3. [Infrastructure Components](#infrastructure-components) +4. [Deployment Process](#deployment-process) +5. [Release Process](#release-process) +6. [Additional Resources](#additional-resources) ## Overview -This guide covers the deployment architecture and release process for the PASS (Public Access Submission System) application. PASS is designed to be flexible and can adapt to various architectures, including cloud infrastructure, hybrid, or on-premises environments. +The Public Access Submission System (PASS) is an open-source platform designed to streamline compliance with funder and institutional open access policies. This guide outlines the deployment process for PASS, which is adaptable to various architectures including cloud, hybrid, or on-premises environments. -## Going Forward +> **Note**: PASS is transitioning towards a cloud-native version. Expect ongoing changes to the architecture, infrastructure, and deployment process, such as moving from Docker Compose to Kubernetes or implementing Infrastructure as Code with Terraform. -As we move towards a cloud-native version of PASS there will be ongoing changes to the architecture, changes to the infrastructure, and changes to the deployment process, e.g. transitioning from Docker Compose to Kubernetes, or using Terraform for IAC. +## Deployment Architecture -## Technical Deep Dive +The PASS deployment workflow involves: -excerpt from [../welcome-guide/deployment-architecture.md] +1. Code contributions to the Eclipse PASS Git repository +2. GitHub Actions workflows for CI/CD +3. AWS SQS for deployment initiation +4. Liquibase for database schema management +5. AWS RDS (PostgreSQL) for data storage -### Deployment Architecture +This architecture serves as a template and should be adapted to meet specific organizational requirements. -The deployment workflow starts when developers contribute code to the Eclipse PASS Git repository. Changes in this repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline facilitating the deployment of the updated code. In deployment SQS is utilized for initiating the deployment from a GitHub workflow that publishes a SNS topic to the queue. The PASS deployment files contain configurations and environment variables that assist in the deployment of the PASS application and supporting services. Liquibase is utilized to manage database schema changes, which interacts with an AWS RDS instance running PostgreSQL. +## Infrastructure Components -For other organizations looking to adopt a similar AWS application and deployment model, it's important to recognize that while the core architecture offers a template, it should be adapted to meet an organization's own requirements and needs. Each organization will need to evaluate its own application demands, data sensitivity, and user base to tailor the cloud resources, network configurations, and security policies accordingly. Moreover, integrating other types of cloud infrastructure or even on-premise solutions might be necessary to address specific technological preferences or regulatory requirements. Hybrid cloud environments or multi-cloud strategies could be employed to leverage the strengths of various cloud providers, enhance resilience, and avoid vendor lock-in. PASS is designed to be flexible and can adapt to a variety of architectures; whether a cloud infrastructure, hybrid or on-premise. +The current PASS infrastructure in AWS includes: ---- +- **EC2**: Hosts Docker Compose +- **ECS**: Hosts auxiliary microservices +- **RDS**: Stores metadata +- **S3**: Stores binary data (managed by OCFL) +- **ALB**: Provides SSL for the frontend +- **WAF**: Protects the frontend -excerpt from [../docs/infra/digitalocean.md] +## Deployment Process -## Deployment Infrastructure +### Prerequisites +- Docker and Docker Compose +- Git -The current PASS infrastructure is running in Amazon Web Service (AWS). It includes the following core components in its stack: +### Steps for VM/EC2 Deployment Using Docker Compose -* EC2 - Amazon Elastic Compute Cloud. Hosts Docker Compose. +1. Install dependencies: + ```bash + apt-get -y update + apt-get install -y gnupg2 pass docker-compose + ``` -* ECS - Amazon Elastic Container Service. Hosts auxiliary microservices. +2. Clone the repository: + ```bash + mkdir -p /src + cd /src + git clone git@github.com:eclipse-pass/pass-docker.git + cd pass-docker + git checkout minimal-assets + ``` -* RDS - Amazon Relational Database Service. Stores metadata. - -* S3 - Amazon S3. Used to store binary data that is managed by OCFL. - -* ALB - Amazon Application Load Balancer provides SSL to the frontend. - -* WAF - AWS Web Application Firewall. Protects the frontend. - -## Provisioning PASS - -### Installing PASS Dependencies - -#### VM/EC2 Instructions Using Docker Compose - -On the server, you will need [docker / docker-compose](https://docs.docker.com/compose/) and [pass-docker](https://github.com/eclipse-pass/pass-docker) - - -```bash -apt-get -y update -apt-get install -y gnupg2 pass docker-compose -mkdir -p /src -cd /src -git clone git@github.com:eclipse-pass/pass-docker.git -cd pass-docker -git checkout minimal-assets -``` - -### Runing PASS - -#### VM/EC2 Instructions Using Docker Compose -```bash -cd /src/pass-docker && \ - docker-compose pull && \ - docker-compose up -``` - ---- -excerpt from [../docs/dev/release.md] +3. Run PASS: + ```bash + cd /src/pass-docker && \ + docker-compose pull && \ + docker-compose up + ``` ## Release Process -A PASS release produces a set of Java artifacts, and Docker images. - -Each release of PASS has its own version which is used by every component. PASS uses `MAJOR.MINOR.PATCH` [semantic versioning](https://semver.org/) approach. The PASS Docker environment is updated with references to the released images. -Source code is tagged and release notes made available. - -The PASS release process involves the following steps: - -Code Contribution : Developers contribute code changes to the PASS Git repository. - -Continuous Integration (CI) : Changes in the repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline. - -Build and Test : The CI pipeline builds and tests the PASS application, ensuring code quality and functionality. +PASS uses semantic versioning (`MAJOR.MINOR.PATCH`). The release process includes: -Release Artifacts : Upon successful testing, the pipeline generates release artifacts, including Java artifacts and Docker images. +1. Code contribution +2. CI/CD via GitHub Actions +3. Building and testing +4. Generating release artifacts (Java artifacts and Docker images) +5. Publishing artifacts to repositories +6. Triggering deployment via AWS SQS +7. Updating infrastructure with new artifacts +8. Generating release notes -Artifact Publishing : Java artifacts are published to Sonatype Nexus and Maven Central repositories, while Docker images are pushed to the GitHub Container Registry. +### GitHub Actions Workflow -Deployment Trigger : The CI pipeline publishes a message to an Amazon Simple Queue Service (SQS) queue, initiating the deployment process. +The "Publish: Release All" workflow automates the release process: -Infrastructure Updates : The deployment process updates the PASS infrastructure with the latest release artifacts, including updating the ECS cluster with the new Docker images. +1. Builds and tests components +2. Publishes Java artifacts +3. Builds and pushes Docker images +4. Creates GitHub Releases -Release Notes : Release notes are generated and made available to stakeholders. - - ---- -excerpt from [../docs/dev/release-steps-with-automation.md] - -#### GitHub Actions - -* [Working with GitHub CICD](github-cicd.md) - -##### Workflow: Release All Projects - -The [Publish: Release All](/.github/workflows/pass-complete-release.yml) is a GitHub workflow that will release all the PASS projects in the correct order, build and push docker images, and create GitHub Releases. - -This workflow performs the following tasks: - -1. Checks out the PASS repository code. - -2. Sets up the build environment (e.g., Java, Docker). - -3. Builds and tests the PASS application components. - -4. Publishes Java artifacts to Sonatype Nexus and Maven Central. - -5. Builds and pushes Docker images to the GitHub Container Registry. - -6. Creates GitHub Releases with release notes and artifacts. - -# Example snippet from the "Publish: Release All" workflow -jobs: - release: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - name: Set up JDK - uses: actions/setup-java@v3 - with: - java-version: '11' - distribution: 'adopt' - - name: Build and Release - run: | - ./mvnw clean install - ./mvnw deploy -Prelease - - name: Build and Push Docker Images - run: | - docker build -t pass/app:${{ github.sha }} . - docker push pass/app:${{ github.sha }} - - name: Create GitHub Release - uses: actions/create-release@v1 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - with: - tag_name: v${{ github.ref }} - release_name: Release ${{ github.ref }} - body: ${{ steps.release-notes.outputs.content }} - -For more detailed information and configuration, refer to the pass-complete-release.yml file in the AWS-PASS-Deploymnet repository. +For detailed configuration, refer to the `pass-complete-release.yml` file in the AWS-PASS-Deployment repository. ## Additional Resources -[PASS main](https://github.com/eclipse-pass/main) - -[PASS Docker Repository](https://github.com/eclipse-pass/pass-docker) +- [PASS main repository](https://github.com/eclipse-pass/main) +- [PASS Docker repository](https://github.com/eclipse-pass/pass-docker) -For further assistance or questions, please _______. \ No newline at end of file +For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues). +``` \ No newline at end of file From 41ad2327c17a9fea0a6bdce05b095f75c0fb2609 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 9 Jul 2024 22:34:51 -0400 Subject: [PATCH 09/14] Update supporting docs --- .../deployment/github-cicd.md | 19 +++++------------- .../deployment/roadmap.md | 20 +++++++++---------- 2 files changed, 14 insertions(+), 25 deletions(-) diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md index 8001dc6..95ed53b 100644 --- a/infrastructure-documenation/deployment/github-cicd.md +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -81,9 +81,9 @@ on: required: true ``` -## AWS / EKS Integration +## AWS Integration -To interact with AWS services, including EKS (Elastic Kubernetes Service), you'll need to set up appropriate secrets and use AWS-specific actions in your workflows. +To interact with AWS services, including ECR (Elastic Container Registry), you'll need to set up appropriate secrets and use AWS-specific actions in your workflows. ### Setting up AWS Credentials @@ -92,10 +92,10 @@ Store your AWS credentials as secrets: 1. `AWS_ACCESS_KEY_ID` 2. `AWS_SECRET_ACCESS_KEY` -### Example Workflow for EKS Deployment +### Example Workflow for AWS Deployment ```yaml -name: Deploy to EKS +name: Deploy to ECR on: push: @@ -127,18 +127,9 @@ jobs: docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG . docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG - - name: Update kube config - run: aws eks get-token --cluster-name my-cluster | kubectl apply -f - - - - name: Deploy to EKS - run: | - kubectl set image deployment/my-app my-app=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG - kubectl rollout status deployment/my-app ``` -This workflow builds a Docker image, pushes it to Amazon ECR, and deploys it to an EKS cluster. - -Remember to replace placeholder values like `my-cluster`, `my-ecr-repo`, and `my-app` with your actual resource names. +This workflow builds a Docker image and pushes it to Amazon ECR. ## Best Practices diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md index 07a7454..d89e5ca 100644 --- a/infrastructure-documenation/deployment/roadmap.md +++ b/infrastructure-documenation/deployment/roadmap.md @@ -98,16 +98,14 @@ The target architecture will leverage Kubernetes for orchestration and cloud-nat ## Timeline -- Phase 1: 1-2 months -- Phase 2: 2-3 months -- Phase 3: 3-4 months -- Phase 4: 2-3 months -- Phase 5: 2-3 months -- Phase 6: 1-2 months -- Phase 7: 2-3 months - -Total estimated timeline: 13-20 months - -Note: This timeline is approximate and may need adjustment based on team capacity, unforeseen challenges, and organizational priorities. +Some phases may run concurrently, others sequentially. + +- Phase 1: 1-2 sprints +- Phase 2: 2-3 sprints +- Phase 3: 3-4 sprints +- Phase 4: 2-3 sprints +- Phase 5: 2-3 sprints +- Phase 6: 1-2 sprints +- Phase 7: 2-3 sprints --- \ No newline at end of file From b88c7414377d17a9c319cd83e07d4c90a1144e25 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Fri, 12 Jul 2024 12:15:56 -0400 Subject: [PATCH 10/14] Add cross-referencing to docs. --- infrastructure-documenation/deployment/README.md | 6 ++++-- .../deployment/github-cicd.md | 2 +- infrastructure-documenation/deployment/roadmap.md | 12 ------------ 3 files changed, 5 insertions(+), 15 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 3ed2777..7e2f6a8 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -12,7 +12,7 @@ The Public Access Submission System (PASS) is an open-source platform designed to streamline compliance with funder and institutional open access policies. This guide outlines the deployment process for PASS, which is adaptable to various architectures including cloud, hybrid, or on-premises environments. -> **Note**: PASS is transitioning towards a cloud-native version. Expect ongoing changes to the architecture, infrastructure, and deployment process, such as moving from Docker Compose to Kubernetes or implementing Infrastructure as Code with Terraform. +> **Note**: PASS is transitioning towards a cloud-native version. Expect ongoing changes to the architecture, infrastructure, and deployment process, such as moving from Docker Compose to Kubernetes or implementing Infrastructure as Code with Terraform. See (roadmap)[./roadmap.md] for more information. ## Deployment Architecture @@ -91,10 +91,12 @@ The "Publish: Release All" workflow automates the release process: For detailed configuration, refer to the `pass-complete-release.yml` file in the AWS-PASS-Deployment repository. +Please refer to [github-cicd.md](./github-cicd.md) for further information on its useage. + ## Additional Resources - [PASS main repository](https://github.com/eclipse-pass/main) - [PASS Docker repository](https://github.com/eclipse-pass/pass-docker) -For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues). +For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in [PASS Slack](#)! ``` \ No newline at end of file diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md index 95ed53b..7005e5d 100644 --- a/infrastructure-documenation/deployment/github-cicd.md +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -2,7 +2,7 @@ ## Overview of GitHub Actions and Workflows -GitHub Actions is a powerful CI/CD platform integrated directly into GitHub repositories. It allows you to automate various software development workflows, including building, testing, and deploying your code. +GitHub Actions is a powerful [CI/CD](./github-cicd.md) platform integrated directly into GitHub repositories. It allows you to automate various software development workflows, including building, testing, and [deploying](./README.md) your code. ### Key Concepts diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md index d89e5ca..ec13d05 100644 --- a/infrastructure-documenation/deployment/roadmap.md +++ b/infrastructure-documenation/deployment/roadmap.md @@ -96,16 +96,4 @@ The target architecture will leverage Kubernetes for orchestration and cloud-nat - Training team members on Kubernetes and cloud-native technologies - Ensuring security and compliance in the new architecture -## Timeline - -Some phases may run concurrently, others sequentially. - -- Phase 1: 1-2 sprints -- Phase 2: 2-3 sprints -- Phase 3: 3-4 sprints -- Phase 4: 2-3 sprints -- Phase 5: 2-3 sprints -- Phase 6: 1-2 sprints -- Phase 7: 2-3 sprints - --- \ No newline at end of file From 52b441de35681dbdfe64f21c24daf6972264ec9d Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 17 Sep 2024 09:37:44 -0400 Subject: [PATCH 11/14] Update deployment docs structure --- .../deployment/README.md | 40 +++++++------------ .../deployment/github-cicd.md | 19 ++++----- .../deployment/roadmap.md | 2 +- 3 files changed, 25 insertions(+), 36 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 7e2f6a8..8fbfea3 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -1,32 +1,20 @@ # Eclipse PASS Deployment Guide ## Table of Contents -1. [Overview](#overview) +1. [Summary](#summary) 2. [Deployment Architecture](#deployment-architecture) -3. [Infrastructure Components](#infrastructure-components) -4. [Deployment Process](#deployment-process) -5. [Release Process](#release-process) -6. [Additional Resources](#additional-resources) +3. [AWS Infrastructure Components](#aws-infrastructure-components) +4. [PASS Deployment Process](#pass-deployment-process) +5. [PASS Release Process](#pass-release-process) +6. [Related Information](#related-infomormation) -## Overview +## Summary The Public Access Submission System (PASS) is an open-source platform designed to streamline compliance with funder and institutional open access policies. This guide outlines the deployment process for PASS, which is adaptable to various architectures including cloud, hybrid, or on-premises environments. > **Note**: PASS is transitioning towards a cloud-native version. Expect ongoing changes to the architecture, infrastructure, and deployment process, such as moving from Docker Compose to Kubernetes or implementing Infrastructure as Code with Terraform. See (roadmap)[./roadmap.md] for more information. -## Deployment Architecture - -The PASS deployment workflow involves: - -1. Code contributions to the Eclipse PASS Git repository -2. GitHub Actions workflows for CI/CD -3. AWS SQS for deployment initiation -4. Liquibase for database schema management -5. AWS RDS (PostgreSQL) for data storage - -This architecture serves as a template and should be adapted to meet specific organizational requirements. - -## Infrastructure Components +## AWS Infrastructure Components The current PASS infrastructure in AWS includes: @@ -37,13 +25,13 @@ The current PASS infrastructure in AWS includes: - **ALB**: Provides SSL for the frontend - **WAF**: Protects the frontend -## Deployment Process +## Deep Dive: Deployment & Release ### Prerequisites - Docker and Docker Compose - Git -### Steps for VM/EC2 Deployment Using Docker Compose +### PASS Deployment Process 1. Install dependencies: ```bash @@ -67,7 +55,7 @@ The current PASS infrastructure in AWS includes: docker-compose up ``` -## Release Process +### PASS Release Process PASS uses semantic versioning (`MAJOR.MINOR.PATCH`). The release process includes: @@ -80,7 +68,7 @@ PASS uses semantic versioning (`MAJOR.MINOR.PATCH`). The release process include 7. Updating infrastructure with new artifacts 8. Generating release notes -### GitHub Actions Workflow +#### GitHub Actions Release Workflow The "Publish: Release All" workflow automates the release process: @@ -89,14 +77,14 @@ The "Publish: Release All" workflow automates the release process: 3. Builds and pushes Docker images 4. Creates GitHub Releases -For detailed configuration, refer to the `pass-complete-release.yml` file in the AWS-PASS-Deployment repository. +For detailed configuration, refer to the [pass-complete-release.yml](https://github.com/eclipse-pass/main/blob/main/.github/workflows/pass-complete-release.yml) Actions workflow. Please refer to [github-cicd.md](./github-cicd.md) for further information on its useage. -## Additional Resources +## Related Information - [PASS main repository](https://github.com/eclipse-pass/main) - [PASS Docker repository](https://github.com/eclipse-pass/pass-docker) -For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in [PASS Slack](#)! +For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in the [PASS Slack](https://eclipse-pass.slack.com)! ``` \ No newline at end of file diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md index 7005e5d..6c4723c 100644 --- a/infrastructure-documenation/deployment/github-cicd.md +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -1,6 +1,6 @@ # GitHub CI/CD Guide -## Overview of GitHub Actions and Workflows +## Summary GitHub Actions is a powerful [CI/CD](./github-cicd.md) platform integrated directly into GitHub repositories. It allows you to automate various software development workflows, including building, testing, and [deploying](./README.md) your code. @@ -12,7 +12,7 @@ GitHub Actions is a powerful [CI/CD](./github-cicd.md) platform integrated direc 4. **Actions**: Reusable units of code that can be shared across workflows. 5. **Events**: Specific activities that trigger a workflow run. -### Basic Workflow Structure +### Workflow Action Structure ```yaml name: CI @@ -80,6 +80,13 @@ on: AWS_SECRET_ACCESS_KEY: required: true ``` +### Best Practices + +1. Use reusable workflows for common tasks to maintain DRY principles. +2. Leverage GitHub-hosted runners when possible to reduce maintenance overhead. +3. Use environment protection rules for sensitive deployments. +4. Regularly audit and rotate your secrets. +5. Use GitHub Actions marketplace for pre-built actions to speed up development. ## AWS Integration @@ -131,13 +138,7 @@ jobs: This workflow builds a Docker image and pushes it to Amazon ECR. -## Best Practices - -1. Use reusable workflows for common tasks to maintain DRY principles. -2. Leverage GitHub-hosted runners when possible to reduce maintenance overhead. -3. Use environment protection rules for sensitive deployments. -4. Regularly audit and rotate your secrets. -5. Use GitHub Actions marketplace for pre-built actions to speed up development. +## Related Information For more detailed information, refer to the [GitHub Actions documentation](https://docs.github.com/en/actions). ``` \ No newline at end of file diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md index ec13d05..303ee2f 100644 --- a/infrastructure-documenation/deployment/roadmap.md +++ b/infrastructure-documenation/deployment/roadmap.md @@ -9,7 +9,7 @@ 6. [Challenges and Considerations](#challenges-and-considerations) 7. [Timeline](#timeline) -## Introduction +## Summary This document outlines the roadmap for transitioning the Eclipse Public Access Submission System (PASS) from its current Docker Compose-based deployment to a cloud-native Kubernetes deployment. This transition aims to improve scalability, resilience, and manageability of the PASS platform. From f315780f62e5851844d7349b6b4078ac9cdcca62 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 17 Sep 2024 10:43:15 -0400 Subject: [PATCH 12/14] Fix deployment doc markdown --- infrastructure-documenation/deployment/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 8fbfea3..45f4214 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -86,5 +86,4 @@ Please refer to [github-cicd.md](./github-cicd.md) for further information on it - [PASS main repository](https://github.com/eclipse-pass/main) - [PASS Docker repository](https://github.com/eclipse-pass/pass-docker) -For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in the [PASS Slack](https://eclipse-pass.slack.com)! -``` \ No newline at end of file +For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in the [PASS Slack](https://eclipse-pass.slack.com). \ No newline at end of file From db7112ec4f1bb89033e30dfc6a1048068b3a06d7 Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 17 Sep 2024 10:45:07 -0400 Subject: [PATCH 13/14] Fix supplemental deploy docs markdown --- infrastructure-documenation/deployment/github-cicd.md | 4 +--- infrastructure-documenation/deployment/roadmap.md | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/infrastructure-documenation/deployment/github-cicd.md b/infrastructure-documenation/deployment/github-cicd.md index 6c4723c..e99c364 100644 --- a/infrastructure-documenation/deployment/github-cicd.md +++ b/infrastructure-documenation/deployment/github-cicd.md @@ -133,12 +133,10 @@ jobs: run: | docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG . docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG - ``` This workflow builds a Docker image and pushes it to Amazon ECR. ## Related Information -For more detailed information, refer to the [GitHub Actions documentation](https://docs.github.com/en/actions). -``` \ No newline at end of file +For more detailed information, refer to the [GitHub Actions documentation](https://docs.github.com/en/actions). \ No newline at end of file diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md index 303ee2f..d6b1525 100644 --- a/infrastructure-documenation/deployment/roadmap.md +++ b/infrastructure-documenation/deployment/roadmap.md @@ -94,6 +94,4 @@ The target architecture will leverage Kubernetes for orchestration and cloud-nat - Managing stateful applications in Kubernetes (e.g., databases) - Balancing resource allocation and cost optimization - Training team members on Kubernetes and cloud-native technologies -- Ensuring security and compliance in the new architecture - ---- \ No newline at end of file +- Ensuring security and compliance in the new architecture \ No newline at end of file From 7b3553ff3d0bf7b755d4c92d006f17e93e7a3f7e Mon Sep 17 00:00:00 2001 From: Steven Miklovic Date: Tue, 17 Sep 2024 13:10:16 -0400 Subject: [PATCH 14/14] Fix TOC in deployment docs --- infrastructure-documenation/deployment/README.md | 6 +++--- infrastructure-documenation/deployment/roadmap.md | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/infrastructure-documenation/deployment/README.md b/infrastructure-documenation/deployment/README.md index 45f4214..8724f26 100644 --- a/infrastructure-documenation/deployment/README.md +++ b/infrastructure-documenation/deployment/README.md @@ -2,11 +2,11 @@ ## Table of Contents 1. [Summary](#summary) -2. [Deployment Architecture](#deployment-architecture) -3. [AWS Infrastructure Components](#aws-infrastructure-components) +2. [AWS Infrastructure Components](#aws-infrastructure-components) +3. [Deep Dive: Deployment & Release](#deep-dive-deployment--release) 4. [PASS Deployment Process](#pass-deployment-process) 5. [PASS Release Process](#pass-release-process) -6. [Related Information](#related-infomormation) +6. [Related Information](#related-information) ## Summary diff --git a/infrastructure-documenation/deployment/roadmap.md b/infrastructure-documenation/deployment/roadmap.md index d6b1525..434c428 100644 --- a/infrastructure-documenation/deployment/roadmap.md +++ b/infrastructure-documenation/deployment/roadmap.md @@ -1,13 +1,12 @@ # Eclipse PASS Cloud-Native Kubernetes Deployment Roadmap ## Table of Contents -1. [Introduction](#introduction) +1. [Summary](#summary) 2. [Current State](#current-state) 3. [Target Architecture](#target-architecture) 4. [Roadmap Phases](#roadmap-phases) 5. [Key Milestones](#key-milestones) 6. [Challenges and Considerations](#challenges-and-considerations) -7. [Timeline](#timeline) ## Summary