From 1133e0e3c364bf11ec73f2768892fd7481d4b1f4 Mon Sep 17 00:00:00 2001 From: Slesa Adhikari Date: Tue, 14 Nov 2023 17:15:10 -0600 Subject: [PATCH] Fetch upstream changes (#52) * Sync updates from upstream, add permissions boundary to created roles if needed (#8) * Point README to veda-docs (#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) https://github.com/NASA-IMPACT/veda-backend/pull/148#issuecomment-1466160512 * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.9 * Update set environment * Add support for permissions boundary * use titiler custom JSONResponse to handle NaN values --------- Co-authored-by: Julia Signell Co-authored-by: ividito Co-authored-by: smohiudd Co-authored-by: Saadiq Mohiuddin <34844565+smohiudd@users.noreply.github.com> Co-authored-by: Nathan Zimmerman Co-authored-by: Caden Helbling Co-authored-by: vincentsarago Co-authored-by: Alexandra Kirk * Update bootstrapper lambda timeout (#11) * Fix order of db search path (#13) * Fix order of db search path * Remove environment from predeploy * Make RDS instance type configurable (#18) * Fix missing rds config update (#22) * Update rds type example config value (#25) * Fix missing rds config update * Update rds type config example value * update collection summaries methods to include max end datetime * fix docker compose in pr action * Add DB storage encryption configurability (#34) * Only add storage_encrypted db config if true (#35) * use NAT Gateway * Add TMS route for tile matrices (#40) * Optionally overwrite host param in stac api (#43) * Update mappingvalue to custom * Overwrite host header for both raster and stac apis (#46) * Overwrite host header for both raster and stac apis * Add docstrings * Fix stac config env prefix * Host param override for api gateway (#48) * Fetch upstream changes (#51) * Fix mangum db connection pool * lint * Point README to veda-docs (#171) * Upgrade titiler and titiler-pgstac * Fix br/gzip header test * Remove factory (it's included in new titiler-pgstac version) https://github.com/NASA-IMPACT/veda-backend/pull/148#issuecomment-1466160512 * Remove pin on cramjam * Load test data in container (actions not updated) * fix actions tests * adjust container strategy * actions connections fix * remove -it flag (it broke actions) * re-enable lint, propagate test changes to other actions * docker-compose -> docker compose * Fix template import * include private subnet config * fix subnet type * format changes * pub accessible default true * remove publicly accessible variable * Add sql connection and execution abstraction * Add sql logic to fix projection extension types * add script to delete null stac_extensions * Lint and move sql command scripts to support_scripts dir * Use pythonic naming conventions * pgstac readme change * pin to urllib3<2 * pin urllib3 in test * format * revert * pin httpx * revert * docker compose pgstac version * pre-deploy action fix * RDS Proxy initial implementation * proxy secret + urlllib fix * use pgstac 0.7.4 * use titiler custom JSONResponse to handle NaN values * change db search path * hacky fix * try exporting os env vars * export os env vars * lint * comment * move to APISettings check * black changes * hide route * Pin pydantic for now * put raster assume role config in lambda construct * pass lambda env variable bool as string * NAT SG rules * order of operation * remove implicity SGs * black formatted * fix invalid string in rule description * updates to deployment docs in readme * update pgstac version in docker compose * use max/end datetime in periodic collection datetime summary * correct table name typo * use NAT Gateway instead of EC2 Instance as NAT * aws_ec2 mistake * use aws_ec2.SubnetType.PRIVATE_WITH_EGRESS * add back in NAT count * add rds subnet group manual removal step to instructions for destroying a stack * Enable tilematrixsets * feat(database): RDS Configurations (#218) * feat(database): RDS Configurations * chore: string configurations * feat(database): Configurable RDS Configurations * consolidate configuration docs in readme * update table header --------- Co-authored-by: anayeaye * raster path prefix * stac path prefix * format * config changes * format * Revert "format" This reverts commit 3a26c1c2e751378dc0b6571b60c09efdca0fdf06. * Revert "config changes" This reverts commit 1425de6bd0e02e0cacf378fda2555a4e2f56e651. * feat: add configuration and github actions for python-semantic-release * feat: lint conventional pr * ci: cdk deploy should not require approval in github actions workflow * docs: add contributing.md * chore(ci): fix typo * chore: explain versioning init choice * chore: expalin automated version in setup.py * chore: typo Co-authored-by: Jamison French <50224594+jjfrench@users.noreply.github.com> * docs: typos and improvements in contributing docs * docs: typos in contributing * ci: handle semantic version releases with tags only * chore: add exclude commit patterns from release changelog * cloudfront in veda backend * format * fix: if provided, apply permissions boundary to all roles created within a stack * mypy, pydocstyle fixes pydocstyle fixes format isort changes optional config format config changes format config changes construct change * include apigateway parameter mapping * remove unused import * update readme * remove unused config * docs: s/veda-data-pipelines/veda-data-airflow/ I can't update the diagram b/c it's draw.io. * include parameter mapping in raster api * ci: branch specific pre-deploy check on pull request action * ci: pre-deploy test against dev stack by default * fix github var name in action * typo * include conditional param mapping * change param mapping kwargs * fix cloudfront subdomain * incude missing argument * formatting * feat(raster)!: update titiler-pgstac from 0.2.3 to 0.8.0 * add dependencies * add tests deps * fix tests * :facepalm: * update handler * pin pydantic to 1.* for cdk deployment * fix deps * fix(raster) use root path for prefixing and add to gateway base url in handler * fix(stac) use root path for prefixing urls * root path in fastapi app * updates to for using a proxy * lint * only add additional servers to fast api if root path in config * root path setting is optional * cleanup * cleanup * do not configure alias for shared subdomain * pr review cleanup * fastapi additional servers not needed with apigw base path=app.root_path * add custom colormap for EPA dataset * Update raster_api/runtime/src/cmap_data/README.md * rename cmap * fix(apis)!: overwrite host in api integration if custom host provided * fix(apis)!: overwrite host in api integration if custom host provided (p2) * fix(routes)!: ingest api root path is /api/publish * change ingestor api cf behavior * format * fix(domain)! only create custom api subdomains when configured * improve config descriptions * typo * improve domain config docs * account for custom staged url of veda-stac-ingestor * do not run lint conventional pr * merge duplicate corrected * lint * merge mismatched rds config naming * merge inconsistencies * merge inconsistencies --------- Co-authored-by: Saadiq Mohiuddin <34844565+smohiudd@users.noreply.github.com> Co-authored-by: ividito Co-authored-by: Julia Signell Co-authored-by: smohiudd Co-authored-by: Nathan Zimmerman Co-authored-by: vincentsarago Co-authored-by: ranchodeluxe Co-authored-by: sudobangbang Co-authored-by: Jamison French <50224594+jjfrench@users.noreply.github.com> Co-authored-by: Pete Gadomski * Remove unnecessary remanants of merge * Remove "host" config var --------- Co-authored-by: Julia Signell Co-authored-by: ividito Co-authored-by: smohiudd Co-authored-by: Saadiq Mohiuddin <34844565+smohiudd@users.noreply.github.com> Co-authored-by: Nathan Zimmerman Co-authored-by: Caden Helbling Co-authored-by: vincentsarago Co-authored-by: Alexandra Kirk Co-authored-by: Abdelhak Marouane <33136280+amarouane-ABDELHAK@users.noreply.github.com> Co-authored-by: ranchodeluxe Co-authored-by: sudobangbang Co-authored-by: Jamison French <50224594+jjfrench@users.noreply.github.com> Co-authored-by: Pete Gadomski --- .example.env | 15 +- .github/workflows/pr.yml | 4 +- .github/workflows/tests/test_raster.py | 4 +- .pre-commit-config.yaml | 2 +- .readme/veda-backend.drawio.svg | 4 +- .readme/veda-backend.drawio.xml | 113 +++++++++++++- README.md | 45 +++++- app.py | 20 ++- database/infrastructure/config.py | 36 ++++- database/infrastructure/construct.py | 27 ++-- database/runtime/Dockerfile | 7 +- docker-compose.yml | 4 +- docs/advanced_configuration.md | 25 ---- domain/infrastructure/config.py | 7 + domain/infrastructure/construct.py | 10 +- raster_api/infrastructure/config.py | 51 +++---- raster_api/infrastructure/construct.py | 55 ++++--- raster_api/runtime/Dockerfile | 4 +- raster_api/runtime/handler.py | 13 +- raster_api/runtime/setup.py | 15 +- raster_api/runtime/src/algorithms.py | 45 ++++++ raster_api/runtime/src/app.py | 141 ++++++++---------- raster_api/runtime/src/cmap_data/README.md | 30 ++++ .../runtime/src/cmap_data/epa-ghgi-ch4.npy | Bin 0 -> 1152 bytes raster_api/runtime/src/config.py | 60 +++++--- raster_api/runtime/src/datasetparams.py | 50 ------- raster_api/runtime/src/dependencies.py | 104 ++++--------- raster_api/runtime/src/extensions.py | 43 ++++++ .../runtime/src/templates/stac-viewer.html | 2 +- routes/infrastructure/config.py | 50 +++++++ routes/infrastructure/construct.py | 114 ++++++++++++++ setup.py | 4 +- stac_api/infrastructure/config.py | 37 ++--- stac_api/infrastructure/construct.py | 38 +++-- stac_api/runtime/handler.py | 5 +- stac_api/runtime/src/api.py | 2 + stac_api/runtime/src/app.py | 10 +- stac_api/runtime/src/config.py | 3 +- stac_api/runtime/src/extension.py | 3 +- standalone_base_infrastructure/README.md | 3 +- 40 files changed, 801 insertions(+), 404 deletions(-) delete mode 100644 docs/advanced_configuration.md create mode 100644 raster_api/runtime/src/algorithms.py create mode 100644 raster_api/runtime/src/cmap_data/README.md create mode 100644 raster_api/runtime/src/cmap_data/epa-ghgi-ch4.npy delete mode 100644 raster_api/runtime/src/datasetparams.py create mode 100644 raster_api/runtime/src/extensions.py create mode 100755 routes/infrastructure/config.py create mode 100755 routes/infrastructure/construct.py diff --git a/.example.env b/.example.env index 1a14613f..84a8576e 100644 --- a/.example.env +++ b/.example.env @@ -19,9 +19,16 @@ VEDA_DOMAIN_ALT_HOSTED_ZONE_ID=[OPTIONAL SECOND HOSTED ZONE] VEDA_DOMAIN_ALT_HOSTED_ZONE_NAME=[OPTIONAL SECOND DOMAIN] VEDA_RASTER_ENABLE_MOSAIC_SEARCH=TRUE -VEDA_RASTER_DATA_ACCESS_ROLE_ARN=[OPTIONAL ARN OF IAM ROLE TO BE ASSUMED BY RASTER API] +VEDA_RASTER_DATA_ACCESS_ROLE_ARN=[OPTIONAL ARN OF IAM ROLE TO BE ASSUMED BY RASTER API] +VEDA_RASTER_EXPORT_ASSUME_ROLE_CREDS_AS_ENVS=False -VEDA_RASTER_PATH_PREFIX=[OPTIONAL PATH PREFIX TO ADD TO TITILER ENDPOINTS] -VEDA_STAC_PATH_PREFIX=[OPTIONAL PATH PREFIX TO ADD TO TITILER ENDPOINTS] +VEDA_DB_PUBLICLY_ACCESSIBLE=TRUE + +VEDA_RASTER_ROOT_PATH= +VEDA_STAC_ROOT_PATH= -VEDA_HOST=[OPTIONAL HOST/DOMAIN_NAME TO PROPAGATE TO STAC AND RASTER APIS] +STAC_BROWSER_BUCKET= +STAC_URL= +CERT_ARN= +VEDA_CLOUDFRONT= +VEDA_CUSTOM_HOST= diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index e5b21aa4..e25fd1a8 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -61,6 +61,7 @@ jobs: with: python-version: '3.9' + - uses: actions/cache@v3 with: path: ${{ env.pythonLocation }} @@ -69,7 +70,7 @@ jobs: - name: Install python dependencies run: | python -m pip install --upgrade pip - python -m pip install -e .[dev,deploy,test] + python -m pip install -e .[dev,deploy,test] - name: Launch services run: AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY=${{secrets.AWS_SECRET_ACCESS_KEY}} docker compose up --build -d @@ -82,6 +83,7 @@ jobs: run: sleep 10s shell: bash + - name: Integrations tests run: python -m pytest .github/workflows/tests/ -vv -s diff --git a/.github/workflows/tests/test_raster.py b/.github/workflows/tests/test_raster.py index fb7483e9..beba896b 100644 --- a/.github/workflows/tests/test_raster.py +++ b/.github/workflows/tests/test_raster.py @@ -33,7 +33,7 @@ def test_mosaic_api(): assert list(resp.json()[0]) == ["id", "bbox", "assets", "collection"] assert resp.json()[0]["id"] == "20200307aC0853900w361030" - resp = httpx.get(f"{raster_endpoint}/mosaic/{searchid}/15/8589/12849/assets") + resp = httpx.get(f"{raster_endpoint}/mosaic/{searchid}/tiles/15/8589/12849/assets") assert resp.status_code == 200 assert len(resp.json()) == 1 assert list(resp.json()[0]) == ["id", "bbox", "assets", "collection"] @@ -41,7 +41,7 @@ def test_mosaic_api(): z, x, y = 15, 8589, 12849 resp = httpx.get( - f"{raster_endpoint}/mosaic/tiles/{searchid}/{z}/{x}/{y}", + f"{raster_endpoint}/mosaic/{searchid}/tiles/{z}/{x}/{y}", params={"assets": "cog"}, headers={"Accept-Encoding": "br, gzip"}, timeout=10.0, diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index c064fbc2..4912662d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -12,7 +12,7 @@ repos: language_version: python - repo: https://github.com/PyCQA/flake8 - rev: 3.8.3 + rev: 6.1.0 hooks: - id: flake8 language_version: python diff --git a/.readme/veda-backend.drawio.svg b/.readme/veda-backend.drawio.svg index 53c7f05f..88acd8a4 100644 --- a/.readme/veda-backend.drawio.svg +++ b/.readme/veda-backend.drawio.svg @@ -1,4 +1,4 @@ - + -
Data Store
Data Store
PgSTAC
PgSTAC
TiTiler
TiTiler
STAC-API
STAC-API
Scientists &
Public
Scientis...
veda-stac-ingestor
veda-stac-ingestor
veda-data-pipelines
veda-data-pipelines
 
Analysis
Platform
...
VPC
VPC
Scientists &
Data Providers
Scientis...
Dashboard (veda-ui & veda-config)
Dashboard (veda-...Text is not SVG - cannot display \ No newline at end of file +
Data Store
Data Store
PgSTAC
PgSTAC
TiTiler
TiTiler
STAC-API
STAC-API
Scientists &
Public
Scientis...
veda-stac-ingestor
veda-stac-ingestor
veda-data-airflow
veda-data-airflow
 
Analysis
Platform
...
VPC
VPC
Scientists &
Data Providers
Scientis...
Dashboard (veda-ui & veda-config)
Dashboard (veda-...
veda-data
veda-dataText is not SVG - cannot display \ No newline at end of file diff --git a/.readme/veda-backend.drawio.xml b/.readme/veda-backend.drawio.xml index 0310b577..0c8d0ea7 100644 --- a/.readme/veda-backend.drawio.xml +++ b/.readme/veda-backend.drawio.xml @@ -1,2 +1,111 @@ - -7Vxtc+MoEv41qbr7YJXeZX+0neRuq/auXJupvdtPU1giNhdZ6BCOnf31BwJsCZBfYtmZ3GQykzENQgiefrppWr4Lpqvt3wgol//AGczvfDfb3gX3d74/ShL2mwvehCAKXSFYEJQJkbcXPKE/oRSqZmuUwarVkGKcU1S2hSkuCpjSlgwQgjftZs84b9+1BAtoCJ5SkJvSf6GMLoV0GLl7+d8hWizVnT1X1sxB+rIgeF3I+935wUP9R1SvgOpLtq+WIMObhih4uAumBGMqPq22U5jzqVXTJq577KjdjZvAgp5ygS8ueAX5GqoRxzm7dPKMWQ9s6kAqKuL/rvmgJlO8JggSVvVPuNmL2aeF/L++fK4LZH8VfcvbHfKKQVVDYMwaeEG5FZdpPd8DClj9E8UEqk7Zg4l+2/diYuP+RlO/NRofr2mOCjjdIcqVY5viHJO6TeCN+Q+XozxvyB+j8STi7StK8AtUNQUuWOeTDFRLmMkeXyGhiAHtVzCH+QxXiCJcsLo5phSvGg3GOVrwCopLJgWylLJ1ZZMfTJZ0lbOyJwcpVcjzVVk+GL8lqErxRM9oy8cxKTHivTy8ss4q2QkDYskvWG0XXKMdsKlCZ75OXyCfrRrRu2cQT6kUw9vNJh863HYi0dvhm9EGxCtIyRtrIi9IvFBcIikjcH0nEpLNXgUTKVo2tC8ZSsWXSr/Y9b0HPvsgsW/Xg6BTDzL0yvuW87+HZQ6faTf8L0D75So3Wzx9G0/PUoaduH7cDhWpcVNPcjRhfxmzTMW/iNVOucTxI4vQJktMoWc2Y/95tjvoQpssMYWe2YyX1KjbQpssicwR61d7lqs97Wr2N5icxThMKzPEVOkeEdZYcEaBCdeLE9mI1Yxc/tMzJdWKcBkhWcmHwIrhPoW/pHw0E1YUn7RWWWXwUw9sFOlspDyaJhe5Fi5yL+ei0FC89uPxVccLXLAVw3wl6jn/D6T0TU46WFPcXhEDCd7Qf+B378RbzsEw2XkzjSbSn+GdAkLH3NvihikHVYVSJX5Eubo3LDLVSBpEJpH1+9WCmeaRicUXoli6gIAsIG05LiesKIE5oOi13fsl6xP1vz7sGcjbv+X1deEPXuDUIsv322bt/VuzNIMEsafgfsG9Ve8//WoHH7fa8TEP+WNsfG9u9Tf0DeX8xh/rUXfbqi7r9sO51RdYsRys5hn4/rwupG3v36IFkd+2aFFgWLQ4MS2akl2iQ8ntGDP5GRlz+EMx5vDzxBT4Pmkwnv3yxX4/GfuFoenPx5bYgpJdog8jAxtcmdVkSp3vkRCdHQX+0azr4MNPTn0qYtzaGww/jvzUeLoiSVYCu5gQn1IeG0AVre54wDcGq1q/699CItqTdzGbPTh09Yearec5g03vYazqBdJ0qTTtK+xrsPa6qu/VA+16btCmXdd0OiNLGCXqIYzieRZNvAERHybUR7BCOZ+NtjZ0Y24HI+806j2FM/2b+Ivy0hmHXSOyFrUja6EeMhOjkldpy70bxmkIuP7p1ivMwICZtXSAigWsKCZn8ZUGx80SUfhUikFtmEK2kWXniCYpSQwfcCffEV5uIVBnmatHXiNvZDkH8iLPZI0g6IE1uk+CesVMBigYlKiE3PpUX6C5EDSBBhpFKjeBjBmwv7KDn/xUHn70g3n40TkeflOdhT9ezKv3uuWn+d+7K8cMXm8Vqm5wqxmb5GdMVuf0qulMA+p2p/wcyjpETe30Bd/iBReQbjB5qZw0x+vs0IbgmEenHPRAp0RemAHK/PGilvhuT3kUseZxjyKb+QyudHbpxca6/uyHlyr+0+Sv+APp6wrB+E++QoElhvSRBsYWQP/kG1e4RVT4LqNwKMvcdxm4jhurBnv/hRfeGgX9vKanffDtNr2jwAkD1428oReEeqpbFGvMKwZpbIGNbhOFXEX23vX20iMDlL/PpgYurSliRnqYnhpmpIW1U8KMdCs91cpIs2onfBk5YXrimJFd1k5AM3LA9EQxI5vsYIpXQ+Vse7djcb2UlQHrlMg+9LCeaJPnoKzQfL/lgumaVAzBv0FxLia2BrbwH9epsvahbIc1de331zI9vGU0nbWTssgqNheoWPxal+6DQ7vPVny1B8fJD7WUL8/cQ4a2YGWga917SN+3nRv835B+GIVt0vfc6GzSf+8JV7/GoukkqADj9Q1I7B40ILrrfqoBiRItGDu6mgHZzemtoiRNiJyQF9KzIvQDOkUCLdB9YGqHGs/5sY9LIqiXnG6ekc5xZsjjwoeSr7PMCH5FGSRVD2PW1OvrnPNm55yhO2rxaByYvsO1zjmt765oUFAuXdeCd06BfMEOzFVP7rlTo8Wj7K/1xJ4l89DXjdG7Zic8PjtHQXv0OMXAr1VTOtVKB+efawKdFK/KtejNhuizl7K5ZAIz5po1Q4Re4oyiaDjkuVOj0LOt0LCH0xNLOv2VT0/8K5+eNGIdiXKDpRMyUrGPXkMd142UmbjRfJTQ9FFE+OwWDkkP6fl+R3Lo5WmnZs/vcBOq5RwDkrEGf6lPjtfI6gXVdYyun9Hir+/zJT7glFkR4yUHzMqRufCAuYsS1RW+E/qJu+dDv23wDZM2GjlDt8GeFvrsxfonx+2bsi9oVb/2fnzdjhrEuqexcrysXpi82f2SUv7S/pg/kf+YZoXroBqmbIEIN3Lci6194UcuZ37wI2T3rz8P/LDcDl5xPuATwonmkVtFUA08f+iUxcLGz620Bd1xPObtSkDWg5803+VvVWCSwUbrXvDlMXxFLUypBLamUY6dINhBKg4toOpochHGTjiQUBhjj0wRyH/jTmOx4LXWIGcDggpUdYHIB6mT3UQQkH+sYdl1CsyWiKJiMa19L254ByJAdH5CC8EUSKiPeiMON3YS/t0R2oLsFnnoeGZSpOcOnQZ5xGFsWemhk/SQnu6bkX1zdd+9lTQWgZEi+zngDZ+0n1yhLMttUeUdRR3aUh4Lu+smpd5Bisg0KwVd+8mS4GydUvmyQ2kNn4MSfV8ACjfgrZ+tp6+lPgUqv7K5vQpN9CjZRd+acELQ+gs614HOVb6EQwfTFV+TYcX919yIePL+q4SCh/8B \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/README.md b/README.md index 50fa76e2..38b80b7a 100644 --- a/README.md +++ b/README.md @@ -43,16 +43,42 @@ To retrieve the variables for a stage that has been previously deployed, the sec | `VEDA_DB_PGSTAC_VERSION` | **REQUIRED** version of PgStac database, i.e. 0.7.9 | | `VEDA_DB_SCHEMA_VERSION` | **REQUIRED** The version of the custom veda-backend schema, i.e. 0.1.1 | | `VEDA_DB_SNAPSHOT_ID` | **Once used always REQUIRED** Optional RDS snapshot identifier to initialize RDS from a snapshot | -> **Note** See [Advanced Configuration](docs/advanced_configuration.md) for details about custom configuration options. + +### Advanced configuration +The constructs and applications in this project are configured using pydantic. The settings are defined in config.py files stored alongside the associated construct or application--for example the settings for the RDS PostgreSQL construct are defined in database/infrastructure/config.py. For custom configuration, use environment variables to override the pydantic defaults. + +| Construct | Env Prefix | Configuration | +| --- | --- | --- | +| Database | `VEDA_DB` | [database/infrastructure/config.py](database/infrastructure/config.py) | +| Domain | `VEDA_DOMAIN` | [domain/infrastructure/config.py](domain/infrastructure/config.py) | +| Network | `N/A` | [network/infrastructure/config.py](network/infrastructure/config.py) | +| Raster API (TiTiler) | `VEDA_RASTER` | [raster_api/infrastructure/config.py](raster_-_api/infrastructure/config.py) | +| STAC API | `VEDA` | [stac_api/infrastructure/config.py](stac_api/infrastructure/config.py) | +| Routes | `VEDA` | [routes/infrastructure/config.py](routes/infrastructure/config.py) | ### Deploying to the cloud -#### Install pre-requisites +#### Install deployment pre-requisites +- [Node](https://nodejs.org/) +- [NVM](https://github.com/nvm-sh/nvm#node-version-manager---) +- [jq](https://jqlang.github.io/jq/) (used for exporting environment variable secrets to `.env` in [scripts/sync-env-local.sh](/scripts/sync-env-local.sh)) + +These can be installed with [homebrew](https://brew.sh/) on MacOS +``` +brew install node +brew install nvm +brew install jq +``` + +#### Virtual environment example +``` +python3 -m venv .venv +source .venv/bin/activate +``` +#### Install requirements ```bash -nvm install 17 -nvm use 17 -node --version +nvm use --lts npm install --location=global aws-cdk python3 -m pip install --upgrade pip python3 -m pip install -e ".[dev,deploy,test]" @@ -72,8 +98,9 @@ cdk deploy If this is a development stack that is safe to delete, you can delete the stack in CloudFormation console or via `cdk destroy`, however, the additional manual steps were required to completely delete the stack resources: 1. You will need to disable deletion protection of the RDS database and delete the database. -2. Detach the Internet Gateway (IGW) from the VPC and delete it. -3. If this stack created a new VPC, delete the VPC (this should delete a subnet and security group too). +2. Identify and delete the RDS subnet group associated with the RDS database you just deleted (it will not be automatically removed because of the RDS deletion protection in place when the group was created). +3. If this stack created a new VPC, detach the Internet Gateway (IGW) from the VPC and delete it. +4. If this stack created a new VPC, delete the VPC (this should delete a subnet and security group too). ## Custom deployments @@ -98,7 +125,7 @@ docker compose down > **Warning** PgSTAC records should be loaded in the database using [pypgstac](https://github.com/stac-utils/pgstac#pypgstac) for proper indexing and partitioning. -The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-pipelines](https://github.com/NASA-IMPACT/veda-data-pipelines) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor). +The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-airflow](https://github.com/NASA-IMPACT/veda-data-airflow) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor). ## Support scripts Support scripts are provided for manual system operations. @@ -118,6 +145,7 @@ Support scripts are provided for manual system operations. ## VEDA usage examples +### [VEDA documentation](https://nasa-impact.github.io/veda-docs) ### [VEDA documentation](https://nasa-impact.github.io/veda-docs) ### [VEDA dashboard](https://www.earthdata.nasa.gov/dashboard) @@ -129,3 +157,4 @@ Radiant Earth's [stac-browser](https://github.com/radiantearth/stac-browser) is # License This project is licensed under **Apache 2**, see the [LICENSE](LICENSE) file for more details. + diff --git a/app.py b/app.py index 8b22485a..9f240055 100644 --- a/app.py +++ b/app.py @@ -9,6 +9,7 @@ from domain.infrastructure.construct import DomainConstruct from network.infrastructure.construct import VpcConstruct from raster_api.infrastructure.construct import RasterApiLambdaConstruct +from routes.infrastructure.construct import CloudfrontDistributionConstruct from stac_api.infrastructure.construct import StacApiLambdaConstruct app = App() @@ -24,7 +25,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: if veda_app_settings.permissions_boundary_policy_name: permission_boundary_policy = aws_iam.ManagedPolicy.from_managed_policy_name( self, - "permission-boundary", + "permissions-boundary", veda_app_settings.permissions_boundary_policy_name, ) aws_iam.PermissionsBoundary.of(self).apply(permission_boundary_policy) @@ -59,18 +60,29 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: raster_api = RasterApiLambdaConstruct( veda_stack, "raster-api", + stage=veda_app_settings.stage_name(), vpc=vpc.vpc, database=database, - domain_name=domain.raster_domain_name, + domain=domain, ) stac_api = StacApiLambdaConstruct( veda_stack, "stac-api", + stage=veda_app_settings.stage_name(), vpc=vpc.vpc, database=database, raster_api=raster_api, - domain_name=domain.stac_domain_name, + domain=domain, +) + +veda_routes = CloudfrontDistributionConstruct( + veda_stack, + "routes", + stage=veda_app_settings.stage_name(), + raster_api_id=raster_api.raster_api.api_id, + stac_api_id=stac_api.stac_api.api_id, + region=veda_app_settings.cdk_default_region, ) # TODO this conditional supports deploying a second set of APIs to a separate custom domain and should be removed if no longer necessary @@ -85,6 +97,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: alt_raster_api = RasterApiLambdaConstruct( veda_stack, "alt-raster-api", + stage=veda_app_settings.stage_name(), vpc=vpc.vpc, database=database, domain_name=alt_domain.raster_domain_name, @@ -93,6 +106,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: alt_stac_api = StacApiLambdaConstruct( veda_stack, "alt-stac-api", + stage=veda_app_settings.stage_name(), vpc=vpc.vpc, database=database, raster_api=raster_api, diff --git a/database/infrastructure/config.py b/database/infrastructure/config.py index f31367ef..b5d8255f 100644 --- a/database/infrastructure/config.py +++ b/database/infrastructure/config.py @@ -1,6 +1,7 @@ """Veda-backend database construct configuration.""" from typing import Optional +from aws_cdk import aws_ec2, aws_rds from pydantic import BaseSettings, Field @@ -57,12 +58,37 @@ class vedaDBSettings(BaseSettings): False, description="Boolean if the RDS should be accessed through a proxy", ) - rds_type: str = Field( - "t3.small", - description="Postgres database type", + rds_instance_class: Optional[str] = Field( + aws_ec2.InstanceClass.BURSTABLE3.value, + description=( + "The instance class of the RDS instance " + "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/InstanceClass.html" + ), + ) + rds_instance_size: Optional[str] = Field( + aws_ec2.InstanceSize.SMALL.value, + description=( + "The size of the RDS instance " + "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/InstanceSize.html" + ), + ) + rds_engine_full_version: Optional[str] = Field( + aws_rds.PostgresEngineVersion.VER_14.postgres_full_version, + description=( + "The version of the RDS Postgres engine " + "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_rds/PostgresEngineVersion.html" + ), + ) + rds_engine_major_version: Optional[str] = Field( + aws_rds.PostgresEngineVersion.VER_14.postgres_major_version, + description=( + "The version of the RDS Postgres engine " + "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_rds/PostgresEngineVersion.html" + ), ) - storage_encrypted: bool = Field( - False, description="Boolean if the RDS should be storage encrypted" + rds_encryption: Optional[bool] = Field( + False, + description="Boolean if the RDS should be encrypted", ) class Config: diff --git a/database/infrastructure/construct.py b/database/infrastructure/construct.py index f1df8273..53389be9 100644 --- a/database/infrastructure/construct.py +++ b/database/infrastructure/construct.py @@ -139,8 +139,19 @@ def __init__( # Custom parameter group engine = aws_rds.DatabaseInstanceEngine.postgres( - version=aws_rds.PostgresEngineVersion.VER_14 + version=aws_rds.PostgresEngineVersion.of( + veda_db_settings.rds_engine_full_version, + veda_db_settings.rds_engine_major_version, + ) + ) + + # RDS Instance Type + rds_instance_type = aws_ec2.InstanceType.of( + aws_ec2.InstanceClass[veda_db_settings.rds_instance_class], + aws_ec2.InstanceSize[veda_db_settings.rds_instance_size], ) + + # version=aws_rds.PostgresEngineVersion.postgres_major_version(veda_db_settings.rds_engine_version) parameter_group = aws_rds.ParameterGroup( self, "parameter-group", @@ -152,35 +163,33 @@ def __init__( }, ) + # Database Configurations database_config = { "id": "rds", "instance_identifier": f"{stack_name}-postgres", "vpc": vpc, "engine": engine, - "instance_type": aws_ec2.InstanceType( - instance_type_identifier=veda_db_settings.rds_type - ), + "instance_type": rds_instance_type, "vpc_subnets": aws_ec2.SubnetSelection(subnet_type=subnet_type), "deletion_protection": True, "removal_policy": RemovalPolicy.RETAIN, "publicly_accessible": veda_db_settings.publicly_accessible, "parameter_group": parameter_group, } - - if storage_encrypted := veda_db_settings.storage_encrypted: - database_config["storage_encrypted"] = storage_encrypted + if veda_db_settings.rds_encryption: + database_config["storage_encrypted"] = veda_db_settings.rds_encryption # Create a new database instance from snapshot if provided if veda_db_settings.snapshot_id: # For the database from snapshot we will need a new master secret - credentials = aws_rds.SnapshotCredentials.from_generated_secret( + snapshot_credentials = aws_rds.SnapshotCredentials.from_generated_secret( username=veda_db_settings.admin_user ) database = aws_rds.DatabaseInstanceFromSnapshot( self, snapshot_identifier=veda_db_settings.snapshot_id, - credentials=credentials, + credentials=snapshot_credentials, **database_config, ) # Or create/update RDS Resource diff --git a/database/runtime/Dockerfile b/database/runtime/Dockerfile index 7cc291b0..e4b32693 100644 --- a/database/runtime/Dockerfile +++ b/database/runtime/Dockerfile @@ -5,12 +5,7 @@ RUN echo "Using PGSTAC Version ${PGSTAC_VERSION}" WORKDIR /tmp -COPY database/runtime/requirements.txt /tmp/requirements.txt - -RUN pip install --upgrade pip -RUN pip install -r /tmp/requirements.txt -t /asset - -RUN pip install pypgstac==${PGSTAC_VERSION} -t /asset +RUN pip install requests "urllib3<2" psycopg[binary,pool] pypgstac==${PGSTAC_VERSION} -t /asset COPY database/runtime/handler.py /asset/handler.py diff --git a/docker-compose.yml b/docker-compose.yml index d06d1626..8aeb43eb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -84,6 +84,8 @@ services: - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} # API Config - VEDA_RASTER_ENABLE_MOSAIC_SEARCH=TRUE + - VEDA_RASTER_EXPORT_ASSUME_ROLE_CREDS_AS_ENVS=TRUE + depends_on: - database @@ -95,7 +97,7 @@ services: database: container_name: veda.db platform: linux/amd64 - image: ghcr.io/stac-utils/pgstac:v0.7.9 + image: ghcr.io/stac-utils/pgstac:v0.7.10 environment: - POSTGRES_USER=username - POSTGRES_PASSWORD=password diff --git a/docs/advanced_configuration.md b/docs/advanced_configuration.md deleted file mode 100644 index de56fc70..00000000 --- a/docs/advanced_configuration.md +++ /dev/null @@ -1,25 +0,0 @@ -# Advanced Configuration -The constructs and applications in this project are configured using [pydantic](https://docs.pydantic.dev/usage/settings/). The settings are defined in config.py files stored alongside the associated construct or application--for example the settings for the RDS PostgreSQL construct are defined in [database/infrastructure/config.py](../database/infrastructure/config.py); the settings for the TiTiler API are defined in [raster_api/runtime/src/config.py](../raster_api/runtime/src/config.py). For custom configuration, use environment variables to override the pydantic defaults. - -## Selected configuration variables -Environment variables for specific VEDA backend components are prefixed, for example database configuration variables are prefixed `VEDA_DB`. See the config.py file in each construct for the appropriate prefix. -| Name | Explanation | -| --- | --- | -| `APP_NAME` | Optional app name used to name stack and resources, defaults to `veda` | -| `STAGE` | **REQUIRED** Deployment stage used to name stack and resources, i.e. `dev`, `staging`, `prod` | -| `VPC_ID` | Optional resource identifier of VPC, if none a new VPC with public and private subnets will be provisioned. | -| `PERMISSIONS_BOUNDARY_POLICY_NAME` | Optional name of IAM policy to define stack permissions boundary | -| `CDK_DEFAULT_ACCOUNT` | When deploying from a local machine the AWS account id is required to deploy to an exiting VPC | -| `CDK_DEFAULT_REGION` | When deploying from a local machine the AWS region id is required to deploy to an exiting VPC | -| `VEDA_DB_PGSTAC_VERSION` | **REQUIRED** version of PgStac database, i.e. 0.5 | -| `VEDA_DB_SCHEMA_VERSION` | **REQUIRED** The version of the custom veda-backend schema, i.e. 0.1.1 | -| `VEDA_DB_SNAPSHOT_ID` | **Once used always REQUIRED** Optional RDS snapshot identifier to initialize RDS from a snapshot | -| `VEDA_DB_PRIVATE_SUBNETS` | Optional boolean to deploy database to private subnet | -| `VEDA_DOMAIN_HOSTED_ZONE_ID` | Optional Route53 zone identifier if using a custom domain name | -| `VEDA_DOMAIN_HOSTED_ZONE_NAME` | Optional custom domain name, i.e. veda-backend.xyz | -| `VEDA_DOMAIN_ALT_HOSTED_ZONE_ID` | Optional second Route53 zone identifier if using a custom domain name | -| `VEDA_DOMAIN_ALT_HOSTED_ZONE_NAME` | Optional second custom domain name, i.e. alt-veda-backend.xyz | -| `VEDA_DOMAIN_API_PREFIX` | Optional domain prefix override supports using a custom prefix instead of the STAGE variabe (an alternate version of the stack can be deployed with a unique STAGE=altprod and after testing prod API traffic can be cut over to the alternate version of the stack by setting the prefix to prod) | -| `VEDA_RASTER_ENABLE_MOSAIC_SEARCH` | Optional deploy the raster API with the mosaic/list endpoint TRUE/FALSE | -| `VEDA_RASTER_DATA_ACCESS_ROLE_ARN` | Optional arn of IAM Role to be assumed by raster-api for S3 bucket data access, if not provided default role for the lambda construct is used | -| `VEDA_RASTER_AWS_REQUEST_PAYER` | Set this optional global parameter to 'requester' if the requester agrees to pay S3 transfer costs | diff --git a/domain/infrastructure/config.py b/domain/infrastructure/config.py index e2b52a87..b74c4e88 100644 --- a/domain/infrastructure/config.py +++ b/domain/infrastructure/config.py @@ -14,6 +14,13 @@ class vedaDomainSettings(BaseSettings): hosted_zone_name: Optional[str] = Field( None, description="Custom domain name, i.e. veda-backend.xyz" ) + create_custom_subdomains: bool = Field( + False, + description=( + "When true and hosted zone config is provided, create a unique subdomain for stac and raster apis. " + "For example -stac. and -raster." + ), + ) api_prefix: Optional[str] = Field( None, description=( diff --git a/domain/infrastructure/construct.py b/domain/infrastructure/construct.py index 9efd0011..39faa91f 100644 --- a/domain/infrastructure/construct.py +++ b/domain/infrastructure/construct.py @@ -30,10 +30,8 @@ def __init__( self.stac_domain_name = None self.raster_domain_name = None - if ( - veda_domain_settings.hosted_zone_id - and veda_domain_settings.hosted_zone_name - ): + if veda_domain_settings.create_custom_subdomains: + # If alternative custom domain provided, use it instead of the default if alt_domain is True: hosted_zone_name = veda_domain_settings.alt_hosted_zone_name @@ -115,5 +113,7 @@ def __init__( value=f"https://{raster_url_prefix}.{hosted_zone_name}/docs", ) CfnOutput( - self, "stac-api", value=f"https://{stac_url_prefix}.{hosted_zone_name}/" + self, + "stac-api", + value=f"https://{stac_url_prefix}.{hosted_zone_name}/", ) diff --git a/raster_api/infrastructure/config.py b/raster_api/infrastructure/config.py index 6d41cfa5..8d717f69 100644 --- a/raster_api/infrastructure/config.py +++ b/raster_api/infrastructure/config.py @@ -6,17 +6,6 @@ from pydantic import BaseSettings, Field -class MyConfig(BaseSettings.Config): - """Custom config class that support multiple env_prefixes""" - - @classmethod - def prepare_field(cls, field) -> None: - """Workaround to not overwrite ENV_PREFIX""" - if "env_names" in field.field_info.extra: - return - return super().prepare_field(field) - - class vedaRasterSettings(BaseSettings): """Raster settings""" @@ -56,47 +45,45 @@ class vedaRasterSettings(BaseSettings): timeout: int = 30 # seconds memory: int = 8000 # Mb - enable_mosaic_search: bool = Field( + raster_enable_mosaic_search: bool = Field( False, description="Deploy the raster API with the mosaic/list endpoint TRUE/FALSE", ) - pgstac_secret_arn: Optional[str] = Field( + raster_pgstac_secret_arn: Optional[str] = Field( None, description="Name or ARN of the AWS Secret containing database connection parameters", ) - data_access_role_arn: Optional[str] = Field( + raster_data_access_role_arn: Optional[str] = Field( None, description="Resource name of role permitting access to specified external S3 buckets", ) - aws_request_payer: Optional[str] = Field( + raster_export_assume_role_creds_as_envs: Optional[bool] = Field( + False, + description="enables 'get_gdal_config' flow to export AWS credentials as os env vars", + ) + + raster_aws_request_payer: Optional[str] = Field( None, description="Set optional global parameter to 'requester' if the requester agrees to pay S3 transfer costs", ) - path_prefix: Optional[str] = Field( + + raster_root_path: str = Field( "", - description="Optional path prefix to add to all api endpoints", + description="Optional root path for all api endpoints", + ) + + custom_host: str = Field( + None, + description="Complete url of custom host including subdomain. When provided, override host in api integration", ) - class Config(MyConfig): + class Config: """model config""" env_file = ".env" - env_prefix = "VEDA_RASTER_" - - -class Settings(vedaRasterSettings): - """Application Settings""" - - host: Optional[str] = Field( - "", - description="Optional host to send to raster api", # propagate cf url to raster api - ) - - class Config(MyConfig): - "Model config" env_prefix = "VEDA_" -veda_raster_settings = Settings() +veda_raster_settings = vedaRasterSettings() diff --git a/raster_api/infrastructure/construct.py b/raster_api/infrastructure/construct.py index e9dbba22..3fd67154 100644 --- a/raster_api/infrastructure/construct.py +++ b/raster_api/infrastructure/construct.py @@ -1,5 +1,7 @@ """CDK Constrcut for a Lambda based TiTiler API with pgstac extension.""" import os +import typing +from typing import Optional from aws_cdk import ( CfnOutput, @@ -16,6 +18,9 @@ from .config import veda_raster_settings +if typing.TYPE_CHECKING: + from domain.infrastructure.construct import DomainConstruct + class RasterApiLambdaConstruct(Construct): """CDK Constrcut for a Lambda based TiTiler API with pgstac extension.""" @@ -24,10 +29,12 @@ def __init__( self, scope: Construct, construct_id: str, + stage: str, vpc, database, code_dir: str = "./", - domain_name: aws_apigatewayv2_alpha.DomainName = None, + # domain_name: aws_apigatewayv2_alpha.DomainName = None, + domain: Optional["DomainConstruct"] = None, **kwargs, ) -> None: """.""" @@ -62,7 +69,7 @@ def __init__( veda_raster_function.add_environment( "VEDA_RASTER_ENABLE_MOSAIC_SEARCH", - str(veda_raster_settings.enable_mosaic_search), + str(veda_raster_settings.raster_enable_mosaic_search), ) veda_raster_function.add_environment( @@ -70,30 +77,37 @@ def __init__( ) veda_raster_function.add_environment( - "VEDA_RASTER_PATH_PREFIX", veda_raster_settings.path_prefix + "VEDA_RASTER_ROOT_PATH", veda_raster_settings.raster_root_path ) # Optional AWS S3 requester pays global setting - if veda_raster_settings.aws_request_payer: + if veda_raster_settings.raster_aws_request_payer: veda_raster_function.add_environment( - "AWS_REQUEST_PAYER", veda_raster_settings.aws_request_payer + "AWS_REQUEST_PAYER", veda_raster_settings.raster_aws_request_payer ) - raster_api_integration = aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration( - construct_id, - handler=veda_raster_function, - parameter_mapping=aws_apigatewayv2_alpha.ParameterMapping().overwrite_header( + integration_kwargs = dict(handler=veda_raster_function) + if veda_raster_settings.custom_host: + integration_kwargs[ + "parameter_mapping" + ] = aws_apigatewayv2_alpha.ParameterMapping().overwrite_header( "host", - aws_apigatewayv2_alpha.MappingValue.custom(veda_raster_settings.host), + aws_apigatewayv2_alpha.MappingValue(veda_raster_settings.custom_host), + ) + + raster_api_integration = ( + aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration( + construct_id, + **integration_kwargs, ) - if veda_raster_settings.host - else None, ) domain_mapping = None - if domain_name: + # Legacy method to use a custom subdomain for this api (i.e. -raster..com) + # If using a custom root path and/or a proxy server, do not use a custom subdomain + if domain and domain.raster_domain_name: domain_mapping = aws_apigatewayv2_alpha.DomainMappingOptions( - domain_name=domain_name + domain_name=domain.raster_domain_name ) self.raster_api = aws_apigatewayv2_alpha.HttpApi( @@ -117,12 +131,12 @@ def __init__( ) # Optional use sts assume role with GetObject permissions for external S3 bucket(s) - if veda_raster_settings.data_access_role_arn: + if veda_raster_settings.raster_data_access_role_arn: # Get the role for external data access data_access_role = aws_iam.Role.from_role_arn( self, "data-access-role", - veda_raster_settings.data_access_role_arn, + veda_raster_settings.raster_data_access_role_arn, ) # Allow this lambda to assume the data access role @@ -133,5 +147,12 @@ def __init__( veda_raster_function.add_environment( "VEDA_RASTER_DATA_ACCESS_ROLE_ARN", - veda_raster_settings.data_access_role_arn, + veda_raster_settings.raster_data_access_role_arn, + ) + + # Optional configuration to export assume role session into lambda function environment + if veda_raster_settings.raster_export_assume_role_creds_as_envs: + veda_raster_function.add_environment( + "VEDA_RASTER_EXPORT_ASSUME_ROLE_CREDS_AS_ENVS", + str(veda_raster_settings.raster_export_assume_role_creds_as_envs), ) diff --git a/raster_api/runtime/Dockerfile b/raster_api/runtime/Dockerfile index c0cec90f..6d7caebb 100644 --- a/raster_api/runtime/Dockerfile +++ b/raster_api/runtime/Dockerfile @@ -3,7 +3,7 @@ FROM public.ecr.aws/sam/build-python3.9:latest WORKDIR /tmp COPY raster_api/runtime /tmp/raster -RUN pip install "mangum>=0.14,<0.15" /tmp/raster["psycopg-binary"] -t /asset --no-binary pydantic +RUN pip install "mangum>=0.14,<0.15" /tmp/raster["psycopg-binary"] -t /asset --no-binary pydantic RUN rm -rf /tmp/raster # # Reduce package size and remove useless files @@ -15,4 +15,4 @@ RUN rm -rdf /asset/numpy/doc/ /asset/boto3* /asset/botocore* /asset/bin /asset/g COPY raster_api/runtime/handler.py /asset/handler.py -CMD ["echo", "hello world"] \ No newline at end of file +CMD ["echo", "hello world"] diff --git a/raster_api/runtime/handler.py b/raster_api/runtime/handler.py index f343d7e1..b3d1c970 100644 --- a/raster_api/runtime/handler.py +++ b/raster_api/runtime/handler.py @@ -6,13 +6,24 @@ from mangum import Mangum from src.app import app +from src.config import ApiSettings from src.monitoring import logger, metrics, tracer +from titiler.pgstac.db import connect_to_db + +settings = ApiSettings() + logging.getLogger("mangum.lifespan").setLevel(logging.ERROR) logging.getLogger("mangum.http").setLevel(logging.ERROR) -handler = Mangum(app, lifespan="off") +@app.on_event("startup") +async def startup_event() -> None: + """Connect to database on startup.""" + await connect_to_db(app, settings=settings.load_postgres_settings()) + + +handler = Mangum(app, lifespan="off", api_gateway_base_path=app.root_path) if "AWS_EXECUTION_ENV" in os.environ: loop = asyncio.get_event_loop() diff --git a/raster_api/runtime/setup.py b/raster_api/runtime/setup.py index 2a9249fa..34760bc4 100644 --- a/raster_api/runtime/setup.py +++ b/raster_api/runtime/setup.py @@ -6,9 +6,12 @@ long_description = f.read() inst_reqs = [ - "titiler.pgstac==0.2.3", - "titiler.application>=0.10,<0.11", - "importlib_resources>=1.1.0;python_version<='3.9'", # https://github.com/cogeotiff/rio-tiler/pull/379 + "boto3", + "titiler.pgstac==0.8.0", + "titiler.core>=0.15.1,<0.16", + "titiler.mosaic>=0.15.1,<0.16", + "titiler.extensions[cogeo]>=0.15.1,<0.16", + "starlette-cramjam>=0.3,<0.4", "aws_xray_sdk>=2.6.0,<3", "aws-lambda-powertools>=1.18.0", ] @@ -18,16 +21,16 @@ "psycopg": ["psycopg[pool]"], # pure python implementation "psycopg-c": ["psycopg[c,pool]"], # C implementation of the libpq wrapper "psycopg-binary": ["psycopg[binary,pool]"], # pre-compiled C implementation - "test": ["pytest", "pytest-cov", "pytest-asyncio", "requests"], + "test": ["pytest", "pytest-cov", "pytest-asyncio", "requests", "brotlipy"], } setup( name="veda.raster_api", description="", - python_requires=">=3.7", + python_requires=">=3.8", packages=find_namespace_packages(exclude=["tests*"]), - package_data={"src": ["templates/*.html"]}, + package_data={"src": ["templates/*.html", "cmap_data/*.npy"]}, include_package_data=True, zip_safe=False, install_requires=inst_reqs, diff --git a/raster_api/runtime/src/algorithms.py b/raster_api/runtime/src/algorithms.py new file mode 100644 index 00000000..08566959 --- /dev/null +++ b/raster_api/runtime/src/algorithms.py @@ -0,0 +1,45 @@ +"""veda custom algorithms""" + +import math + +import numpy +from rio_tiler.models import ImageData + +from titiler.core.algorithm import Algorithms +from titiler.core.algorithm.base import BaseAlgorithm + +# https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/reader.py#L35-L37 + +# From eoAPI datasetparams edl_auth branch https://github.com/NASA-IMPACT/eoAPI/blob/edl_auth/src/eoapi/raster/eoapi/raster/datasetparams.py + + +class SWIR(BaseAlgorithm): + """SWIR Custom Algorithm.""" + + low_value: float = math.e + high_value: float = 255 + low_threshold: float = math.log(1000) + high_threshold: float = math.log(7500) + + def __call__(self, img: ImageData) -> ImageData: + """Apply processing.""" + data = numpy.log(img.array) + data[numpy.where(data <= self.low_threshold)] = self.low_value + data[numpy.where(data >= self.high_threshold)] = self.high_value + indices = numpy.where((data > self.low_value) & (data < self.high_value)) + data[indices] = ( + self.high_value + * (data[indices] - self.low_threshold) + / (self.high_threshold - self.low_threshold) + ) + img.array = data.astype("uint8") + return img + + +algorithms = Algorithms( + { + "swir": SWIR, + } +) + +PostProcessParams = algorithms.dependency diff --git a/raster_api/runtime/src/app.py b/raster_api/runtime/src/app.py index f8f5ebb3..29b42114 100644 --- a/raster_api/runtime/src/app.py +++ b/raster_api/runtime/src/app.py @@ -1,126 +1,125 @@ """TiTiler+PgSTAC FastAPI application.""" import logging +from contextlib import asynccontextmanager from aws_lambda_powertools.metrics import MetricUnit -from rio_cogeo.cogeo import cog_info as rio_cogeo_info -from rio_cogeo.models import Info +from src.algorithms import PostProcessParams from src.config import ApiSettings -from src.datasetparams import DatasetParams -from src.factory import MultiBaseTilerFactory +from src.dependencies import ColorMapParams, ItemPathParams +from src.extensions import stacViewerExtension +from src.monitoring import LoggerRouteHandler, logger, metrics, tracer from src.version import __version__ as veda_raster_version -from fastapi import APIRouter, Depends, FastAPI, Query +from fastapi import APIRouter, FastAPI from starlette.middleware.cors import CORSMiddleware from starlette.requests import Request -from starlette.responses import HTMLResponse -from starlette.templating import Jinja2Templates from starlette_cramjam.middleware import CompressionMiddleware -from titiler.core.dependencies import DatasetPathParams from titiler.core.errors import DEFAULT_STATUS_CODES, add_exception_handlers -from titiler.core.factory import TilerFactory, TMSFactory +from titiler.core.factory import MultiBaseTilerFactory, TilerFactory, TMSFactory from titiler.core.middleware import CacheControlMiddleware from titiler.core.resources.enums import OptionalHeader from titiler.core.resources.responses import JSONResponse +from titiler.extensions import cogValidateExtension, cogViewerExtension from titiler.mosaic.errors import MOSAIC_STATUS_CODES from titiler.pgstac.db import close_db_connection, connect_to_db -from titiler.pgstac.dependencies import ItemPathParams from titiler.pgstac.factory import MosaicTilerFactory from titiler.pgstac.reader import PgSTACReader -try: - from importlib.resources import files as resources_files # type: ignore -except ImportError: - # Try backported to PY<39 `importlib_resources`. - from importlib_resources import files as resources_files # type: ignore - -from .monitoring import LoggerRouteHandler, logger, metrics, tracer - logging.getLogger("botocore.credentials").disabled = True logging.getLogger("botocore.utils").disabled = True logging.getLogger("rio-tiler").setLevel(logging.ERROR) settings = ApiSettings() -templates = Jinja2Templates(directory=str(resources_files(__package__) / "templates")) # type: ignore if settings.debug: optional_headers = [OptionalHeader.server_timing, OptionalHeader.x_assets] else: optional_headers = [] -path_prefix = settings.path_prefix + +@asynccontextmanager +async def lifespan(app: FastAPI): + """FastAPI Lifespan.""" + # Create Connection Pool + await connect_to_db(app, settings=settings.load_postgres_settings()) + yield + # Close the Connection Pool + await close_db_connection(app) + + app = FastAPI( title=settings.name, version=veda_raster_version, - openapi_url=f"{path_prefix}/openapi.json", - docs_url=f"{path_prefix}/docs", + openapi_url="/openapi.json", + docs_url="/docs", + lifespan=lifespan, + root_path=settings.root_path, ) + # router to be applied to all titiler route factories (improves logs with FastAPI context) router = APIRouter(route_class=LoggerRouteHandler) add_exception_handlers(app, DEFAULT_STATUS_CODES) add_exception_handlers(app, MOSAIC_STATUS_CODES) -# Custom PgSTAC mosaic tiler +############################################################################### +# /mosaic - PgSTAC Mosaic titiler endpoint +############################################################################### mosaic = MosaicTilerFactory( - add_mosaic_list=settings.enable_mosaic_search, - router_prefix=f"{path_prefix}/mosaic", + router_prefix="/mosaic", optional_headers=optional_headers, environment_dependency=settings.get_gdal_config, - dataset_dependency=DatasetParams, + process_dependency=PostProcessParams, router=APIRouter(route_class=LoggerRouteHandler), + # add /list (default to False) + add_mosaic_list=settings.enable_mosaic_search, + # add /statistics [POST] (default to False) + add_statistics=True, + # add /map viewer (default to False) + add_viewer=False, + # add /bbox [GET] and /feature [POST] (default to False) + add_part=True, + colormap_dependency=ColorMapParams, ) -app.include_router(mosaic.router, prefix=f"{path_prefix}/mosaic", tags=["Mosaic"]) +app.include_router(mosaic.router, prefix="/mosaic", tags=["Mosaic"]) +# TODO +# prefix will be replaced by `/mosaics/{search_id}` in titiler-pgstac 1.0.0 -# Custom STAC titiler endpoint (not added to the openapi docs) +############################################################################### +# /stac - Custom STAC titiler endpoint +############################################################################### stac = MultiBaseTilerFactory( reader=PgSTACReader, path_dependency=ItemPathParams, optional_headers=optional_headers, + router_prefix="/stac", environment_dependency=settings.get_gdal_config, - router_prefix=f"{path_prefix}/stac", router=APIRouter(route_class=LoggerRouteHandler), + extensions=[ + stacViewerExtension(), + ], + colormap_dependency=ColorMapParams, ) -app.include_router(stac.router, tags=["Items"], prefix=f"{path_prefix}/stac") +app.include_router(stac.router, tags=["Items"], prefix="/stac") +# TODO +# in titiler-pgstac we replaced the prefix to `/collections/{collection_id}/items/{item_id}` +############################################################################### +# /cog - External Cloud Optimized GeoTIFF endpoints +############################################################################### cog = TilerFactory( - router_prefix=f"{path_prefix}/cog", + router_prefix="/cog", optional_headers=optional_headers, environment_dependency=settings.get_gdal_config, router=APIRouter(route_class=LoggerRouteHandler), + extensions=[ + cogValidateExtension(), + cogViewerExtension(), + ], + colormap_dependency=ColorMapParams, ) - -@cog.router.get( - "/validate", - response_model=Info, - response_class=JSONResponse, -) -def cog_validate( - src_path: str = Depends(DatasetPathParams), - strict: bool = Query(False, description="Treat warnings as errors"), -): - """Validate a COG""" - return rio_cogeo_info(src_path, strict=strict, config=settings.get_gdal_config()) - - -@cog.router.get("/viewer", response_class=HTMLResponse) -def cog_demo(request: Request): - """COG Viewer.""" - return templates.TemplateResponse( - name="viewer.html", - context={ - "request": request, - "tilejson_endpoint": cog.url_for(request, "tilejson"), - "info_endpoint": cog.url_for(request, "info"), - "statistics_endpoint": cog.url_for(request, "statistics"), - }, - media_type="text/html", - ) - - -app.include_router( - cog.router, tags=["Cloud Optimized GeoTIFF"], prefix=f"{path_prefix}/cog" -) +app.include_router(cog.router, tags=["Cloud Optimized GeoTIFF"], prefix="/cog") @app.get("/healthz", description="Health Check", tags=["Health Check"]) @@ -133,7 +132,6 @@ def ping(): tms = TMSFactory() app.include_router(tms.router, tags=["Tiling Schemes"]) - # Set all CORS enabled origins if settings.cors_origins: app.add_middleware( @@ -151,6 +149,7 @@ def ping(): ) app.add_middleware( CompressionMiddleware, + minimum_size=0, exclude_mediatype={ "image/jpeg", "image/jpg", @@ -174,8 +173,10 @@ async def add_correlation_id(request: Request, call_next): except KeyError: # If empty, use uuid corr_id = "local" + # Add correlation id to logs logger.set_correlation_id(corr_id) + # Add correlation id to traces tracer.put_annotation(key="correlation_id", value=corr_id) @@ -192,15 +193,3 @@ async def validation_exception_handler(request, err): metrics.add_metric(name="UnhandledExceptions", unit=MetricUnit.Count, value=1) logger.exception("Unhandled exception") return JSONResponse(status_code=500, content={"detail": "Internal Server Error"}) - - -@app.on_event("startup") -async def startup_event() -> None: - """Connect to database on startup.""" - await connect_to_db(app, settings=settings.load_postgres_settings()) - - -@app.on_event("shutdown") -async def shutdown_event() -> None: - """Close database connection.""" - await close_db_connection(app) diff --git a/raster_api/runtime/src/cmap_data/README.md b/raster_api/runtime/src/cmap_data/README.md new file mode 100644 index 00000000..821038d0 --- /dev/null +++ b/raster_api/runtime/src/cmap_data/README.md @@ -0,0 +1,30 @@ +## Additional colormap for VEDA + +##### EPA colormap + +ref: https://github.com/NASA-IMPACT/veda-config-ghg/issues/203 + +```python +from matplotlib import colors +import numpy as np + +my_cmap = colors.LinearSegmentedColormap.from_list(name='my_cmap',colors=['#6F4C9B', '#6059A9', '#5568B8', '#4E79C5', '#4D8AC6', +'#4E96BC', '#549EB3', '#59A5A9', '#60AB9E', '#69B190', +'#77B77D', '#8CBC68', '#A6BE54', '#BEBC48', '#D1B541', +'#DDAA3C', '#E49C39', '#E78C35', '#E67932', '#E4632D', +'#DF4828', '#DA2222', '#B8221E', '#95211B', '#721E17', +'#521A13'], N=3000) +my_cmap._init() + +slopen = 200 +alphas_slope = np.abs(np.linspace(0, 1.0, slopen)) +alphas_stable = np.ones(3003-slopen) +alphas = np.concatenate((alphas_slope, alphas_stable)) +my_cmap._lut[:,-1] = alphas +my_cmap.set_under('white', alpha=0) + +x = np.linspace(0, 1, 256) +cmap_vals = my_cmap(x)[:, :] +cmap_uint8 = (cmap_vals * 255).astype('uint8') +np.save("epa-ghgi-ch4.npy", cmap_uint8) +``` diff --git a/raster_api/runtime/src/cmap_data/epa-ghgi-ch4.npy b/raster_api/runtime/src/cmap_data/epa-ghgi-ch4.npy new file mode 100644 index 0000000000000000000000000000000000000000..6929ef016bc68d462a3e6a9f57ae7ae8d169ca54 GIT binary patch literal 1152 zcmbWt>sQkS90qWD+Y8@x(Nti}a-hS4#N?tjEBjlY96Y^?#|1)ow zUA-Sr!VDfL7MTJI#b#ka;&5D^#BxQGG?J#0S_^VhY+w; zu@Lnv{zu%q0CArW3H|fLi^PFBNbY+e8FVAb^u{ctLo<+?T}U=hBYAiVDHbQ0MCyYH zq>eZsvyPK7q}fIxx7i_o_z>xK8#3%xWQ>j=bJPOG*f131W@L>IA=_bs(lLmfi2>wJ z_LDxSoIOxGyP=sfB5$e-`O}>!aNR|r>kf)$T2V67g3{UBD0erZ!urc-e2Uzz^V}DBL4NUkj^EtzcsU!7S2GvzyXyk}m_Co!Q|Ixg^Bn$~ zJcreZIIKD1us#-x_0d>t*kiG2Bi0yfjl^KvLWZNUGZc+o6B&$#f8Z?q{iOFZ?Ddd) zr0Wd!JIUQB?6;FP(sCO6&Eyto{1p2Qq@G+SwUO92knbb0r;os16{-9LdljVYWB5x* z(JA=zi6$I=RXBEYgxJjzVkh$?cH}3qoq7V>$tSQS3B#5+44X_SHm`(Y=;&$^Rde3<2647ugI@_9$t{=qU zh?6)%j>1l?#6nCVM=(HoiII009i)vk2Y-M@ax?f4>WCre5cEL@p(7Q6@1v9yk-Vet UK}D2D-i3ms6WQT+ASDv=FNINvuK)l5 literal 0 HcmV?d00001 diff --git a/raster_api/runtime/src/config.py b/raster_api/runtime/src/config.py index 388845b5..03cf86ac 100644 --- a/raster_api/runtime/src/config.py +++ b/raster_api/runtime/src/config.py @@ -2,12 +2,14 @@ import base64 import json +import os from typing import Optional import boto3 -import pydantic -from pydantic import BaseSettings, Field +from pydantic import Field, field_validator +from pydantic_settings import BaseSettings from rasterio.session import AWSSession +from typing_extensions import Annotated from titiler.pgstac.settings import PostgresSettings @@ -52,13 +54,20 @@ class ApiSettings(BaseSettings): cors_origins: str = "*" cachecontrol: str = "public, max-age=3600" debug: bool = False + root_path: Optional[str] = None # MosaicTiler settings enable_mosaic_search: bool = False - pgstac_secret_arn: Optional[str] + pgstac_secret_arn: Optional[str] = None - @pydantic.validator("cors_origins") + model_config = { + "env_file": ".env", + "extra": "ignore", + "env_prefix": "VEDA_RASTER_", + } + + @field_validator("cors_origins") def parse_cors_origin(cls, v): """Parse CORS origins.""" return [origin.strip() for origin in v.split(",")] @@ -72,16 +81,25 @@ def load_postgres_settings(self) -> "PostgresSettings": postgres_user=secret["username"], postgres_pass=secret["password"], postgres_host=secret["host"], - postgres_port=str(secret["port"]), + postgres_port=int(secret["port"]), postgres_dbname=secret["dbname"], ) else: return PostgresSettings() - data_access_role_arn: Optional[str] = Field( - None, - description="Resource name of role permitting access to specified external S3 buckets", - ) + data_access_role_arn: Annotated[ + Optional[str], + Field( + description="Resource name of role permitting access to specified external S3 buckets" + ), + ] = None + + export_assume_role_creds_as_envs: Annotated[ + bool, + Field( + description="enables 'get_gdal_config' flow to export AWS credentials as os env vars", + ), + ] = False def get_gdal_config(self): """return default aws session config or assume role data_access_role_arn credentials session""" @@ -91,6 +109,22 @@ def get_gdal_config(self): data_access_credentials = get_role_credentials( self.data_access_role_arn ) + + # hack for issue https://github.com/NASA-IMPACT/veda-backend/issues/192 + # which forces any nested `rasterio.Env` context managers (which run in separate threads) + # to pick up the assume-role `AWS_*` os env vars and re-init from there via: + # https://github.com/rasterio/rasterio/blob/main/rasterio/env.py#L204-L205 + if self.export_assume_role_creds_as_envs: + os.environ["AWS_ACCESS_KEY_ID"] = data_access_credentials[ + "AccessKeyId" + ] + os.environ["AWS_SECRET_ACCESS_KEY"] = data_access_credentials[ + "SecretAccessKey" + ] + os.environ["AWS_SESSION_TOKEN"] = data_access_credentials[ + "SessionToken" + ] + return { "session": AWSSession( aws_access_key_id=data_access_credentials["AccessKeyId"], @@ -108,11 +142,3 @@ def get_gdal_config(self): else: # Use the default role of this lambda return {} - - path_prefix: str = "" - - class Config: - """model config""" - - env_file = ".env" - env_prefix = "VEDA_RASTER_" diff --git a/raster_api/runtime/src/datasetparams.py b/raster_api/runtime/src/datasetparams.py deleted file mode 100644 index 1105d8bc..00000000 --- a/raster_api/runtime/src/datasetparams.py +++ /dev/null @@ -1,50 +0,0 @@ -"""From eoAPI datasetparams edl_auth branch https://github.com/NASA-IMPACT/eoAPI/blob/edl_auth/src/eoapi/raster/eoapi/raster/datasetparams.py""" -import math -from dataclasses import dataclass -from typing import Optional, Tuple - -import numpy - -from fastapi import Query -from titiler.core import dependencies - -# https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/reader.py#L35-L37 - -# From eoAPI datasetparams edl_auth branch https://github.com/NASA-IMPACT/eoAPI/blob/edl_auth/src/eoapi/raster/eoapi/raster/datasetparams.py - - -def swir(data, mask) -> Tuple[numpy.ndarray, numpy.ndarray]: - """SWIR""" - low_value = math.e - high_value = 255 - - low_threshold = math.log(1000) - high_threshold = math.log(7500) - - data = numpy.log(data) - data[numpy.where(data <= low_threshold)] = low_value - data[numpy.where(data >= high_threshold)] = high_value - indices = numpy.where((data > low_value) & (data < high_value)) - data[indices] = ( - high_value * (data[indices] - low_threshold) / (high_threshold - low_threshold) - ) - return data.astype("uint8"), mask - - -pp_methods = { - "swir": swir, -} - - -@dataclass -class DatasetParams(dependencies.DatasetParams): - """Post processing parameters for map layers""" - - post_process: Optional[str] = Query(None, description="Post Process Name.") - - def __post_init__(self): - """.""" - super().__post_init__() - - if self.post_process is not None: - self.post_process = pp_methods.get(self.post_process) # type: ignore diff --git a/raster_api/runtime/src/dependencies.py b/raster_api/runtime/src/dependencies.py index c4914138..c41ef552 100644 --- a/raster_api/runtime/src/dependencies.py +++ b/raster_api/runtime/src/dependencies.py @@ -1,76 +1,38 @@ """veda.raster.dependencies.""" -import json -from base64 import b64decode -from typing import Dict, Union -from urllib.parse import urlparse - -from cachetools import LRUCache, cached -from cachetools.keys import hashkey +import pystac +from rio_tiler.colormap import cmap as default_cmap +from typing_extensions import Annotated from fastapi import Query from starlette.requests import Request - - -@cached( - LRUCache(maxsize=512), - key=lambda pool, collection_id, item_id: hashkey(collection_id, item_id), -) -def get_item(pool, collection_id, item_id): - """Get STAC Item from PGStac.""" - - print("COLLECTION ID: ", collection_id) - print("ITEM ID: ", item_id) - - req = dict( - filter={ - "op": "and", - "args": [ - { - "op": "eq", - "args": [{"property": "collection"}, collection_id], - }, - {"op": "eq", "args": [{"property": "id"}, item_id]}, - ], - }, - ) - print("REQUEST: ", req) - with pool.connection() as conn: - with conn.cursor() as cursor: - cursor.execute( - "SELECT * FROM search(%s);", - (json.dumps(req),), - ) - resp = cursor.fetchone()[0] - features = resp.get("features", []) - if not len(features): - raise Exception( - "No item '{item_id}' found in '{collection_id}' collection" - ) - - return features[0] - - -def DatasetPathParams( - request: Request, url: str = Query(..., description="Dataset URL") -) -> Union[str, Dict]: - """Custom Dataset Param for the custom STAC Reader""" - parsed = urlparse(url) - - # stac://{base 64 encoded item} - if parsed.scheme == "stac": - return json.loads(b64decode(url.replace("stac://", ""))) - - # pgstac://{collectionId}/{itemId} - elif parsed.scheme == "pgstac": - collection_id = parsed.netloc - item_id = parsed.path.strip("/") - return get_item( - request.app.state.dbpool, - collection_id, - item_id, - ) - - # Default to passing the URL - else: - return url +from titiler.core.dependencies import create_colormap_dependency +from titiler.pgstac.dependencies import get_stac_item + +try: + from importlib.resources import files as resources_files # type: ignore +except ImportError: + # Try backported to PY<39 `importlib_resources`. + from importlib_resources import files as resources_files # type: ignore + + +def ItemPathParams( + request: Request, + collection: Annotated[ + str, + Query(description="STAC Collection ID"), + ], + item: Annotated[ + str, + Query(description="STAC Item ID"), + ], +) -> pystac.Item: + """STAC Item dependency.""" + return get_stac_item(request.app.state.dbpool, collection, item) + + +VEDA_CMAPS_FILES = { + f.stem: str(f) for f in (resources_files(__package__) / "cmap_data").glob("*.npy") # type: ignore +} +cmap = default_cmap.register(VEDA_CMAPS_FILES) +ColorMapParams = create_colormap_dependency(cmap) diff --git a/raster_api/runtime/src/extensions.py b/raster_api/runtime/src/extensions.py new file mode 100644 index 00000000..51271d98 --- /dev/null +++ b/raster_api/runtime/src/extensions.py @@ -0,0 +1,43 @@ +"""Stac Viewer Extension.""" + +from dataclasses import dataclass + +import jinja2 + +from fastapi import Depends +from starlette.requests import Request +from starlette.responses import HTMLResponse +from starlette.templating import Jinja2Templates +from titiler.core.factory import BaseTilerFactory, FactoryExtension + +DEFAULT_TEMPLATES = Jinja2Templates( + directory="", + loader=jinja2.ChoiceLoader([jinja2.PackageLoader(__package__, "templates")]), +) # type:ignore + + +@dataclass +class stacViewerExtension(FactoryExtension): + """Add /viewer endpoint to the TilerFactory.""" + + templates: Jinja2Templates = DEFAULT_TEMPLATES + + def register(self, factory: BaseTilerFactory): + """Register endpoint to the tiler factory.""" + + @factory.router.get("/viewer", response_class=HTMLResponse) + def stac_viewer( + request: Request, + item=Depends(factory.path_dependency), + ): + """STAC Viewer.""" + return self.templates.TemplateResponse( + name="stac-viewer.html", + context={ + "request": request, + "endpoint": request.url.path.replace("/viewer", ""), + "collection": item.collection_id, + "item": item.id, + }, + media_type="text/html", + ) diff --git a/raster_api/runtime/src/templates/stac-viewer.html b/raster_api/runtime/src/templates/stac-viewer.html index dacf18e4..957bf947 100644 --- a/raster_api/runtime/src/templates/stac-viewer.html +++ b/raster_api/runtime/src/templates/stac-viewer.html @@ -723,4 +723,4 @@ }) - \ No newline at end of file + diff --git a/routes/infrastructure/config.py b/routes/infrastructure/config.py new file mode 100755 index 00000000..97d90231 --- /dev/null +++ b/routes/infrastructure/config.py @@ -0,0 +1,50 @@ +"""Settings for Cloudfront distribution - any environment variables starting with +`VEDA_` will overwrite the values of variables in this file +""" +from typing import Optional + +from pydantic import BaseSettings, Field + + +class vedaRouteSettings(BaseSettings): + """Veda Route settings""" + + cloudfront: Optional[bool] = Field( + False, + description="Boolean if Cloudfront Distribution should be deployed", + ) + + # STAC S#3 browser bucket name + stac_browser_bucket: Optional[str] = Field( + "", description="STAC browser S3 bucket name" + ) + + # API Gateway URLs + ingest_url: Optional[str] = Field( + "", + description="URL of ingest API", + ) + + domain_hosted_zone_name: Optional[str] = Field( + None, + description="Domain name for the cloudfront distribution", + ) + + domain_hosted_zone_id: Optional[str] = Field( + None, description="Domain ID for the cloudfront distribution" + ) + + cert_arn: Optional[str] = Field( + None, + description="Certificate’s ARN", + ) + + class Config: + """model config""" + + env_prefix = "VEDA_" + case_sentive = False + env_file = ".env" + + +veda_route_settings = vedaRouteSettings() diff --git a/routes/infrastructure/construct.py b/routes/infrastructure/construct.py new file mode 100755 index 00000000..8efa5f36 --- /dev/null +++ b/routes/infrastructure/construct.py @@ -0,0 +1,114 @@ +"""CDK Construct for a Cloudfront Distribution.""" +from typing import Optional +from urllib.parse import urlparse + +from aws_cdk import CfnOutput, Stack +from aws_cdk import aws_certificatemanager as certificatemanager +from aws_cdk import aws_cloudfront as cf +from aws_cdk import aws_cloudfront_origins as origins +from aws_cdk import aws_route53, aws_route53_targets +from aws_cdk import aws_s3 as s3 +from constructs import Construct + +from .config import veda_route_settings + + +class CloudfrontDistributionConstruct(Construct): + """CDK Construct for a Cloudfront Distribution.""" + + def __init__( + self, + scope: Construct, + construct_id: str, + stage: str, + raster_api_id: str, + stac_api_id: str, + region: Optional[str], + **kwargs, + ) -> None: + """.""" + super().__init__(scope, construct_id) + + stack_name = Stack.of(self).stack_name + + if veda_route_settings.cloudfront: + s3Bucket = s3.Bucket.from_bucket_name( + self, + "stac-browser-bucket", + bucket_name=veda_route_settings.stac_browser_bucket, + ) + + # Certificate must be in zone us-east-1 + domain_cert = ( + certificatemanager.Certificate.from_certificate_arn( + self, "domainCert", veda_route_settings.cert_arn + ) + if veda_route_settings.cert_arn + else None + ) + + self.distribution = cf.Distribution( + self, + stack_name, + comment=stack_name, + default_behavior=cf.BehaviorOptions( + origin=origins.HttpOrigin( + s3Bucket.bucket_website_domain_name, + protocol_policy=cf.OriginProtocolPolicy.HTTP_ONLY, + origin_id="stac-browser", + ), + cache_policy=cf.CachePolicy.CACHING_DISABLED, + ), + certificate=domain_cert, + domain_names=[f"{stage}.{veda_route_settings.domain_hosted_zone_name}"] + if veda_route_settings.domain_hosted_zone_name + else None, + additional_behaviors={ + "/api/stac*": cf.BehaviorOptions( + origin=origins.HttpOrigin( + f"{stac_api_id}.execute-api.{region}.amazonaws.com", + origin_id="stac-api", + ), + cache_policy=cf.CachePolicy.CACHING_DISABLED, + allowed_methods=cf.AllowedMethods.ALLOW_ALL, + origin_request_policy=cf.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + ), + "/api/raster*": cf.BehaviorOptions( + origin=origins.HttpOrigin( + f"{raster_api_id}.execute-api.{region}.amazonaws.com", + origin_id="raster-api", + ), + cache_policy=cf.CachePolicy.CACHING_DISABLED, + allowed_methods=cf.AllowedMethods.ALLOW_ALL, + origin_request_policy=cf.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + ), + "/api/ingest*": cf.BehaviorOptions( + origin=origins.HttpOrigin( + urlparse(veda_route_settings.ingest_url).hostname, + origin_id="ingest-api", + origin_path=f"/{stage}", + ), + cache_policy=cf.CachePolicy.CACHING_DISABLED, + allowed_methods=cf.AllowedMethods.ALLOW_ALL, + ), + }, + ) + + hosted_zone = aws_route53.HostedZone.from_hosted_zone_attributes( + self, + "hosted-zone", + hosted_zone_id=veda_route_settings.domain_hosted_zone_id, + zone_name=veda_route_settings.domain_hosted_zone_name, + ) + + aws_route53.ARecord( + self, + "cloudfront-dns-record", + zone=hosted_zone, + target=aws_route53.RecordTarget.from_alias( + aws_route53_targets.CloudFrontTarget(self.distribution) + ), + record_name=stage, + ) + + CfnOutput(self, "Endpoint", value=self.distribution.domain_name) diff --git a/setup.py b/setup.py index 704a431c..7bc91eda 100644 --- a/setup.py +++ b/setup.py @@ -5,7 +5,6 @@ with open("README.md") as f: long_description = f.read() - extra_reqs = { "dev": ["pre-commit", "python-dotenv"], "deploy": [ @@ -13,7 +12,7 @@ "constructs>=10.0.0,<11.0.0", "aws-cdk.aws_apigatewayv2_alpha~=2.15.0a0", "aws_cdk.aws_apigatewayv2_integrations_alpha~=2.15.0a0", - "pydantic", + "pydantic~=1.0", ], "test": [ "pytest", @@ -27,7 +26,6 @@ setup( name="veda-backend", - version="0.6.2", description="", long_description=long_description, long_description_content_type="text/markdown", diff --git a/stac_api/infrastructure/config.py b/stac_api/infrastructure/config.py index 196b36dd..0f2a8eb9 100644 --- a/stac_api/infrastructure/config.py +++ b/stac_api/infrastructure/config.py @@ -4,17 +4,6 @@ from pydantic import BaseSettings, Field -class MyConfig(BaseSettings.Config): - """Custom config class that support multiple env_prefixes""" - - @classmethod - def prepare_field(cls, field) -> None: - """Workaround to not overwrite ENV_PREFIX""" - if "env_names" in field.field_info.extra: - return - return super().prepare_field(field) - - class vedaSTACSettings(BaseSettings): """STAC settings""" @@ -24,34 +13,26 @@ class vedaSTACSettings(BaseSettings): memory: int = 8000 # Mb # Secret database credentials - pgstac_secret_arn: Optional[str] = Field( + stac_pgstac_secret_arn: Optional[str] = Field( None, description="Name or ARN of the AWS Secret containing database connection parameters", ) - path_prefix: Optional[str] = Field( + stac_root_path: str = Field( "", description="Optional path prefix to add to all api endpoints", ) - class Config(MyConfig): + custom_host: str = Field( + None, + description="Complete url of custom host including subdomain. When provided, override host in api integration", + ) + + class Config: """model config""" env_file = ".env" - env_prefix = "VEDA_STAC_" - - -class Settings(vedaSTACSettings): - """Application Settings""" - - host: Optional[str] = Field( - "", - description="Optional host to send to stac api", # stac api populates the urls in the catalog based on this - ) - - class Config(MyConfig): - "Model config" env_prefix = "VEDA_" -veda_stac_settings = Settings() +veda_stac_settings = vedaSTACSettings() diff --git a/stac_api/infrastructure/construct.py b/stac_api/infrastructure/construct.py index 867196d3..c4d0012d 100644 --- a/stac_api/infrastructure/construct.py +++ b/stac_api/infrastructure/construct.py @@ -1,5 +1,7 @@ """CDK Construct for a Lambda backed API implementing stac-fastapi.""" import os +import typing +from typing import Optional from aws_cdk import ( CfnOutput, @@ -15,6 +17,9 @@ from .config import veda_stac_settings +if typing.TYPE_CHECKING: + from domain.infrastructure.construct import DomainConstruct + class StacApiLambdaConstruct(Construct): """CDK Construct for a Lambda backed API implementing stac-fastapi.""" @@ -23,11 +28,12 @@ def __init__( self, scope: Construct, construct_id: str, + stage: str, vpc, database, raster_api, # TODO: typing! code_dir: str = "./", - domain_name: aws_apigatewayv2_alpha.DomainName = None, + domain: Optional["DomainConstruct"] = None, **kwargs, ) -> None: """.""" @@ -71,31 +77,37 @@ def __init__( ) lambda_function.add_environment( - "VEDA_STAC_PATH_PREFIX", veda_stac_settings.path_prefix + "VEDA_STAC_ROOT_PATH", veda_stac_settings.stac_root_path ) - stac_api_integration = aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration( - construct_id, - handler=lambda_function, - parameter_mapping=aws_apigatewayv2_alpha.ParameterMapping().overwrite_header( + integration_kwargs = dict(handler=lambda_function) + if veda_stac_settings.custom_host: + integration_kwargs[ + "parameter_mapping" + ] = aws_apigatewayv2_alpha.ParameterMapping().overwrite_header( "host", - aws_apigatewayv2_alpha.MappingValue.custom(veda_stac_settings.host), + aws_apigatewayv2_alpha.MappingValue(veda_stac_settings.custom_host), + ) + stac_api_integration = ( + aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration( + construct_id, + **integration_kwargs, ) - if veda_stac_settings.host - else None, ) domain_mapping = None - if domain_name: + # Legacy method to use a custom subdomain for this api (i.e. -stac..com) + # If using a custom root path and/or a proxy server, do not use a custom subdomain + if domain and domain.stac_domain_name: domain_mapping = aws_apigatewayv2_alpha.DomainMappingOptions( - domain_name=domain_name + domain_name=domain.stac_domain_name ) - stac_api = aws_apigatewayv2_alpha.HttpApi( + self.stac_api = aws_apigatewayv2_alpha.HttpApi( self, f"{stack_name}-{construct_id}", default_integration=stac_api_integration, default_domain_mapping=domain_mapping, ) - CfnOutput(self, "stac-api", value=stac_api.url) + CfnOutput(self, "stac-api", value=self.stac_api.url) diff --git a/stac_api/runtime/handler.py b/stac_api/runtime/handler.py index 176be13b..9da7ad6b 100644 --- a/stac_api/runtime/handler.py +++ b/stac_api/runtime/handler.py @@ -4,12 +4,15 @@ from mangum import Mangum from src.app import app +from src.config import ApiSettings from src.monitoring import logger, metrics, tracer +settings = ApiSettings() + logging.getLogger("mangum.lifespan").setLevel(logging.ERROR) logging.getLogger("mangum.http").setLevel(logging.ERROR) -handler = Mangum(app, lifespan="auto") +handler = Mangum(app, lifespan="auto", api_gateway_base_path=app.root_path) # Add tracing handler.__name__ = "handler" # tracer requires __name__ to be set diff --git a/stac_api/runtime/src/api.py b/stac_api/runtime/src/api.py index 75a15d59..529cc470 100644 --- a/stac_api/runtime/src/api.py +++ b/stac_api/runtime/src/api.py @@ -34,6 +34,7 @@ def register_post_search(self): CollectionSearchPost, self.response_class, ), + include_in_schema=False, ) def register_get_search(self): @@ -55,4 +56,5 @@ def register_get_search(self): CollectionSearchGet, self.response_class, ), + include_in_schema=False, ) diff --git a/stac_api/runtime/src/app.py b/stac_api/runtime/src/app.py index eca070dc..9d1249b0 100644 --- a/stac_api/runtime/src/app.py +++ b/stac_api/runtime/src/app.py @@ -8,7 +8,7 @@ from src.config import post_request_model as POSTModel from src.extension import TiTilerExtension -from fastapi import APIRouter, FastAPI +from fastapi import FastAPI from fastapi.responses import ORJSONResponse from stac_fastapi.pgstac.db import close_db_connection, connect_to_db from starlette.middleware.cors import CORSMiddleware @@ -33,13 +33,12 @@ api_settings = ApiSettings() tiles_settings = TilesApiSettings() -path_prefix = api_settings.path_prefix - api = VedaStacApi( app=FastAPI( title=api_settings.name, - openapi_url=f"{path_prefix}/openapi.json", - docs_url=f"{path_prefix}/docs", + openapi_url="/openapi.json", + docs_url="/docs", + root_path=api_settings.root_path, ), title=api_settings.name, description=api_settings.name, @@ -50,7 +49,6 @@ search_post_request_model=POSTModel, response_class=ORJSONResponse, middlewares=[CompressionMiddleware], - router=APIRouter(prefix=f"{path_prefix}"), ) app = api.app diff --git a/stac_api/runtime/src/config.py b/stac_api/runtime/src/config.py index 3be0b4ea..7a2edf01 100644 --- a/stac_api/runtime/src/config.py +++ b/stac_api/runtime/src/config.py @@ -58,9 +58,8 @@ class _ApiSettings(pydantic.BaseSettings): cors_origins: str = "*" cachecontrol: str = "public, max-age=3600" debug: bool = False - + root_path: Optional[str] = None pgstac_secret_arn: Optional[str] - path_prefix: str = "" @pydantic.validator("cors_origins") def parse_cors_origin(cls, v): diff --git a/stac_api/runtime/src/extension.py b/stac_api/runtime/src/extension.py index 76e9d5d2..c9ad256f 100644 --- a/stac_api/runtime/src/extension.py +++ b/stac_api/runtime/src/extension.py @@ -14,7 +14,6 @@ from .monitoring import LoggerRouteHandler, tracer api_settings = ApiSettings() -path_prefix = api_settings.path_prefix MAX_B64_ITEM_SIZE = 2000 @@ -31,7 +30,7 @@ def register(self, app: FastAPI, titiler_endpoint: str) -> None: None """ - router = APIRouter(route_class=LoggerRouteHandler, prefix=path_prefix) + router = APIRouter(route_class=LoggerRouteHandler) @tracer.capture_method @router.get( diff --git a/standalone_base_infrastructure/README.md b/standalone_base_infrastructure/README.md index 0e559072..86f86551 100644 --- a/standalone_base_infrastructure/README.md +++ b/standalone_base_infrastructure/README.md @@ -1,6 +1,6 @@ # Standalone Base Infrastructure -Optional shared base infrastructure provisioning. This CloudFormation stack is intended to simulate controlled deployment environments. It also useful for deploying a long-standing VPC that can be shared across stacks. This VPC is deployed with an EC2 NAT Instance that is configured as the NAT gateway provider for the private subnets. +Optional shared base infrastructure provisioning. This CloudFormation stack is intended to simulate controlled deployment environments. It also useful for deploying a long-standing VPC that can be shared across stacks. This VPC is deployed with a NAT Gateway Service for the private subnets b/c environments like SMCE require us to manage patches and scale it ourselves which don't want to do. In MCP we can request to use a NAT Gateway Service or use their existing EC2 NAT Instance ## Deployment @@ -26,4 +26,3 @@ See main app [deployment instructions](../README.md#deployment). | `CDK_DEFAULT_REGION` | The AWS region id is required to deploy to an exiting VPC | | `VPC_CIDR` | The CIDR range to use for the VPC. Default is 10.100.0.0/16 | | `VPC_MAX_AZS` | Maximum number of availability zones per region. Default is 2. | -| `VPC_NAT_GATEWAYS` | Number of NAT gateways to create. Default is 1. |