feat(database): RDS Configurations #218
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
jjfrench
commented
Aug 24, 2023
jjfrench
commented
- extracts rds_instance_class, rds_instance_size, rds_engine_version, and rds_encryption to the configurations file
- moves shared database configurations to a database_config object to be passed in
- moves advanced configurations to main readme, links to configuration files
- renames credentials -> snapshot_credentials for clarity
anayeaye
approved these changes
Aug 24, 2023
slesaad
pushed a commit
to US-GHG-Center/ghgc-backend
that referenced
this pull request
Nov 10, 2023
* Fix mangum db connection pool * lint * Point README to veda-docs (NASA-IMPACT#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) NASA-IMPACT#148 (comment) * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * pin to urllib3<2 * pin urllib3 in test * format * revert * pin httpx * revert * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.4 * use titiler custom JSONResponse to handle NaN values * change db search path * hacky fix * try exporting os env vars * export os env vars * lint * comment * move to APISettings check * black changes * hide route * Pin pydantic for now * put raster assume role config in lambda construct * pass lambda env variable bool as string * NAT SG rules * order of operation * remove implicity SGs * black formatted * fix invalid string in rule description * updates to deployment docs in readme * update pgstac version in docker compose * use max/end datetime in periodic collection datetime summary * correct table name typo * use NAT Gateway instead of EC2 Instance as NAT * aws_ec2 mistake * use aws_ec2.SubnetType.PRIVATE_WITH_EGRESS * add back in NAT count * add rds subnet group manual removal step to instructions for destroying a stack * Enable tilematrixsets * feat(database): RDS Configurations (NASA-IMPACT#218) * feat(database): RDS Configurations * chore: string configurations * feat(database): Configurable RDS Configurations * consolidate configuration docs in readme * update table header --------- Co-authored-by: anayeaye <[email protected]> * raster path prefix * stac path prefix * format * config changes * format * Revert "format" This reverts commit 3a26c1c. * Revert "config changes" This reverts commit 1425de6. * feat: add configuration and github actions for python-semantic-release * feat: lint conventional pr * ci: cdk deploy should not require approval in github actions workflow * docs: add contributing.md * chore(ci): fix typo * chore: explain versioning init choice * chore: expalin automated version in setup.py * chore: typo Co-authored-by: Jamison French <[email protected]> * docs: typos and improvements in contributing docs * docs: typos in contributing * ci: handle semantic version releases with tags only * chore: add exclude commit patterns from release changelog * cloudfront in veda backend * format * fix: if provided, apply permissions boundary to all roles created within a stack * mypy, pydocstyle fixes pydocstyle fixes format isort changes optional config format config changes format config changes construct change * include apigateway parameter mapping * remove unused import * update readme * remove unused config * docs: s/veda-data-pipelines/veda-data-airflow/ I can't update the diagram b/c it's draw.io. * include parameter mapping in raster api * ci: branch specific pre-deploy check on pull request action * ci: pre-deploy test against dev stack by default * fix github var name in action * typo * include conditional param mapping * change param mapping kwargs * fix cloudfront subdomain * incude missing argument * formatting * feat(raster)!: update titiler-pgstac from 0.2.3 to 0.8.0 * add dependencies * add tests deps * fix tests * 🤦 * update handler * pin pydantic to 1.* for cdk deployment * fix deps * fix(raster) use root path for prefixing and add to gateway base url in handler * fix(stac) use root path for prefixing urls * root path in fastapi app * updates to for using a proxy * lint * only add additional servers to fast api if root path in config * root path setting is optional * cleanup * cleanup * do not configure alias for shared subdomain * pr review cleanup * fastapi additional servers not needed with apigw base path=app.root_path * add custom colormap for EPA dataset * Update raster_api/runtime/src/cmap_data/README.md * rename cmap * fix(apis)!: overwrite host in api integration if custom host provided * fix(apis)!: overwrite host in api integration if custom host provided (p2) * fix(routes)!: ingest api root path is /api/publish * change ingestor api cf behavior * format * fix(domain)! only create custom api subdomains when configured * improve config descriptions * typo * improve domain config docs * account for custom staged url of veda-stac-ingestor * do not run lint conventional pr * merge duplicate corrected * lint * merge mismatched rds config naming * merge inconsistencies * merge inconsistencies --------- Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: Julia Signell <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]>
slesaad
added a commit
to US-GHG-Center/ghgc-backend
that referenced
this pull request
Nov 14, 2023
* Sync updates from upstream, add permissions boundary to created roles if needed (#8) * Point README to veda-docs (NASA-IMPACT#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) NASA-IMPACT#148 (comment) * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.9 * Update set environment * Add support for permissions boundary * use titiler custom JSONResponse to handle NaN values --------- Co-authored-by: Julia Signell <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: Caden Helbling <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: Alexandra Kirk <[email protected]> * Update bootstrapper lambda timeout (#11) * Fix order of db search path (#13) * Fix order of db search path * Remove environment from predeploy * Make RDS instance type configurable (#18) * Fix missing rds config update (#22) * Update rds type example config value (#25) * Fix missing rds config update * Update rds type config example value * update collection summaries methods to include max end datetime * fix docker compose in pr action * Add DB storage encryption configurability (#34) * Only add storage_encrypted db config if true (#35) * use NAT Gateway * Add TMS route for tile matrices (#40) * Optionally overwrite host param in stac api (#43) * Update mappingvalue to custom * Overwrite host header for both raster and stac apis (#46) * Overwrite host header for both raster and stac apis * Add docstrings * Fix stac config env prefix * Host param override for api gateway (#48) * Fetch upstream changes (#51) * Fix mangum db connection pool * lint * Point README to veda-docs (NASA-IMPACT#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) NASA-IMPACT#148 (comment) * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * pin to urllib3<2 * pin urllib3 in test * format * revert * pin httpx * revert * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.4 * use titiler custom JSONResponse to handle NaN values * change db search path * hacky fix * try exporting os env vars * export os env vars * lint * comment * move to APISettings check * black changes * hide route * Pin pydantic for now * put raster assume role config in lambda construct * pass lambda env variable bool as string * NAT SG rules * order of operation * remove implicity SGs * black formatted * fix invalid string in rule description * updates to deployment docs in readme * update pgstac version in docker compose * use max/end datetime in periodic collection datetime summary * correct table name typo * use NAT Gateway instead of EC2 Instance as NAT * aws_ec2 mistake * use aws_ec2.SubnetType.PRIVATE_WITH_EGRESS * add back in NAT count * add rds subnet group manual removal step to instructions for destroying a stack * Enable tilematrixsets * feat(database): RDS Configurations (NASA-IMPACT#218) * feat(database): RDS Configurations * chore: string configurations * feat(database): Configurable RDS Configurations * consolidate configuration docs in readme * update table header --------- Co-authored-by: anayeaye <[email protected]> * raster path prefix * stac path prefix * format * config changes * format * Revert "format" This reverts commit 3a26c1c. * Revert "config changes" This reverts commit 1425de6. * feat: add configuration and github actions for python-semantic-release * feat: lint conventional pr * ci: cdk deploy should not require approval in github actions workflow * docs: add contributing.md * chore(ci): fix typo * chore: explain versioning init choice * chore: expalin automated version in setup.py * chore: typo Co-authored-by: Jamison French <[email protected]> * docs: typos and improvements in contributing docs * docs: typos in contributing * ci: handle semantic version releases with tags only * chore: add exclude commit patterns from release changelog * cloudfront in veda backend * format * fix: if provided, apply permissions boundary to all roles created within a stack * mypy, pydocstyle fixes pydocstyle fixes format isort changes optional config format config changes format config changes construct change * include apigateway parameter mapping * remove unused import * update readme * remove unused config * docs: s/veda-data-pipelines/veda-data-airflow/ I can't update the diagram b/c it's draw.io. * include parameter mapping in raster api * ci: branch specific pre-deploy check on pull request action * ci: pre-deploy test against dev stack by default * fix github var name in action * typo * include conditional param mapping * change param mapping kwargs * fix cloudfront subdomain * incude missing argument * formatting * feat(raster)!: update titiler-pgstac from 0.2.3 to 0.8.0 * add dependencies * add tests deps * fix tests * 🤦 * update handler * pin pydantic to 1.* for cdk deployment * fix deps * fix(raster) use root path for prefixing and add to gateway base url in handler * fix(stac) use root path for prefixing urls * root path in fastapi app * updates to for using a proxy * lint * only add additional servers to fast api if root path in config * root path setting is optional * cleanup * cleanup * do not configure alias for shared subdomain * pr review cleanup * fastapi additional servers not needed with apigw base path=app.root_path * add custom colormap for EPA dataset * Update raster_api/runtime/src/cmap_data/README.md * rename cmap * fix(apis)!: overwrite host in api integration if custom host provided * fix(apis)!: overwrite host in api integration if custom host provided (p2) * fix(routes)!: ingest api root path is /api/publish * change ingestor api cf behavior * format * fix(domain)! only create custom api subdomains when configured * improve config descriptions * typo * improve domain config docs * account for custom staged url of veda-stac-ingestor * do not run lint conventional pr * merge duplicate corrected * lint * merge mismatched rds config naming * merge inconsistencies * merge inconsistencies --------- Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: Julia Signell <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]> * Remove unnecessary remanants of merge * Remove "host" config var --------- Co-authored-by: Julia Signell <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: Caden Helbling <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: Alexandra Kirk <[email protected]> Co-authored-by: Abdelhak Marouane <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]>
slesaad
added a commit
to US-GHG-Center/ghgc-backend
that referenced
this pull request
Nov 15, 2023
* Fix STAC catalog urls, add support for alternative projections in titiler (#44) * Header host override for stac and raster apis (#47) * Fetch upstream changes (#52) * Sync updates from upstream, add permissions boundary to created roles if needed (#8) * Point README to veda-docs (NASA-IMPACT#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) NASA-IMPACT#148 (comment) * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.9 * Update set environment * Add support for permissions boundary * use titiler custom JSONResponse to handle NaN values --------- Co-authored-by: Julia Signell <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: Caden Helbling <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: Alexandra Kirk <[email protected]> * Update bootstrapper lambda timeout (#11) * Fix order of db search path (#13) * Fix order of db search path * Remove environment from predeploy * Make RDS instance type configurable (#18) * Fix missing rds config update (#22) * Update rds type example config value (#25) * Fix missing rds config update * Update rds type config example value * update collection summaries methods to include max end datetime * fix docker compose in pr action * Add DB storage encryption configurability (#34) * Only add storage_encrypted db config if true (#35) * use NAT Gateway * Add TMS route for tile matrices (#40) * Optionally overwrite host param in stac api (#43) * Update mappingvalue to custom * Overwrite host header for both raster and stac apis (#46) * Overwrite host header for both raster and stac apis * Add docstrings * Fix stac config env prefix * Host param override for api gateway (#48) * Fetch upstream changes (#51) * Fix mangum db connection pool * lint * Point README to veda-docs (NASA-IMPACT#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) NASA-IMPACT#148 (comment) * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * pin to urllib3<2 * pin urllib3 in test * format * revert * pin httpx * revert * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.4 * use titiler custom JSONResponse to handle NaN values * change db search path * hacky fix * try exporting os env vars * export os env vars * lint * comment * move to APISettings check * black changes * hide route * Pin pydantic for now * put raster assume role config in lambda construct * pass lambda env variable bool as string * NAT SG rules * order of operation * remove implicity SGs * black formatted * fix invalid string in rule description * updates to deployment docs in readme * update pgstac version in docker compose * use max/end datetime in periodic collection datetime summary * correct table name typo * use NAT Gateway instead of EC2 Instance as NAT * aws_ec2 mistake * use aws_ec2.SubnetType.PRIVATE_WITH_EGRESS * add back in NAT count * add rds subnet group manual removal step to instructions for destroying a stack * Enable tilematrixsets * feat(database): RDS Configurations (NASA-IMPACT#218) * feat(database): RDS Configurations * chore: string configurations * feat(database): Configurable RDS Configurations * consolidate configuration docs in readme * update table header --------- Co-authored-by: anayeaye <[email protected]> * raster path prefix * stac path prefix * format * config changes * format * Revert "format" This reverts commit 3a26c1c. * Revert "config changes" This reverts commit 1425de6. * feat: add configuration and github actions for python-semantic-release * feat: lint conventional pr * ci: cdk deploy should not require approval in github actions workflow * docs: add contributing.md * chore(ci): fix typo * chore: explain versioning init choice * chore: expalin automated version in setup.py * chore: typo Co-authored-by: Jamison French <[email protected]> * docs: typos and improvements in contributing docs * docs: typos in contributing * ci: handle semantic version releases with tags only * chore: add exclude commit patterns from release changelog * cloudfront in veda backend * format * fix: if provided, apply permissions boundary to all roles created within a stack * mypy, pydocstyle fixes pydocstyle fixes format isort changes optional config format config changes format config changes construct change * include apigateway parameter mapping * remove unused import * update readme * remove unused config * docs: s/veda-data-pipelines/veda-data-airflow/ I can't update the diagram b/c it's draw.io. * include parameter mapping in raster api * ci: branch specific pre-deploy check on pull request action * ci: pre-deploy test against dev stack by default * fix github var name in action * typo * include conditional param mapping * change param mapping kwargs * fix cloudfront subdomain * incude missing argument * formatting * feat(raster)!: update titiler-pgstac from 0.2.3 to 0.8.0 * add dependencies * add tests deps * fix tests * 🤦 * update handler * pin pydantic to 1.* for cdk deployment * fix deps * fix(raster) use root path for prefixing and add to gateway base url in handler * fix(stac) use root path for prefixing urls * root path in fastapi app * updates to for using a proxy * lint * only add additional servers to fast api if root path in config * root path setting is optional * cleanup * cleanup * do not configure alias for shared subdomain * pr review cleanup * fastapi additional servers not needed with apigw base path=app.root_path * add custom colormap for EPA dataset * Update raster_api/runtime/src/cmap_data/README.md * rename cmap * fix(apis)!: overwrite host in api integration if custom host provided * fix(apis)!: overwrite host in api integration if custom host provided (p2) * fix(routes)!: ingest api root path is /api/publish * change ingestor api cf behavior * format * fix(domain)! only create custom api subdomains when configured * improve config descriptions * typo * improve domain config docs * account for custom staged url of veda-stac-ingestor * do not run lint conventional pr * merge duplicate corrected * lint * merge mismatched rds config naming * merge inconsistencies * merge inconsistencies --------- Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: Julia Signell <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]> * Remove unnecessary remanants of merge * Remove "host" config var --------- Co-authored-by: Julia Signell <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: Caden Helbling <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: Alexandra Kirk <[email protected]> Co-authored-by: Abdelhak Marouane <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]> * Fix rds encryption var in example .env file (#55) --------- Co-authored-by: Julia Signell <[email protected]> Co-authored-by: ividito <[email protected]> Co-authored-by: smohiudd <[email protected]> Co-authored-by: Saadiq Mohiuddin <[email protected]> Co-authored-by: Nathan Zimmerman <[email protected]> Co-authored-by: Caden Helbling <[email protected]> Co-authored-by: vincentsarago <[email protected]> Co-authored-by: Alexandra Kirk <[email protected]> Co-authored-by: Abdelhak Marouane <[email protected]> Co-authored-by: ranchodeluxe <[email protected]> Co-authored-by: sudobangbang <[email protected]> Co-authored-by: Jamison French <[email protected]> Co-authored-by: Pete Gadomski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.