From 0482b6713d315894b60ee3728900a70bdd9e3812 Mon Sep 17 00:00:00 2001 From: Blayne Chard Date: Thu, 2 Nov 2023 15:49:03 +1300 Subject: [PATCH] refactor: apply formatter to docs --- docs/README.md | 25 +++++++++++-------------- docs/api/empty-tiles.md | 2 +- docs/configuration.md | 6 +++--- docs/deployment.md | 7 ++++--- docs/imagery/README.md | 2 +- docs/imagery/cog.quality.md | 9 +++------ docs/imagery/relief.shade.md | 7 +++---- docs/imagery/texutre.shade.md | 13 +++++-------- 8 files changed, 31 insertions(+), 40 deletions(-) diff --git a/docs/README.md b/docs/README.md index 0727f3f7e..0dbe8c6c3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,11 +4,11 @@ Basemaps currently supports both [Imagery](#aerial--satellite-imagery) and [Vector data](#vector-data) -## Background +## Background Land Information New Zealand has a large archive of aerial and satellite imagery freely aviable from [LINZ Data Service](https://data.linz.govt.nz) and [linz/imagery](https://github.com/linz/imagery) this imagery was a primary driver for the creation of the linz basemaps product and service. For more information on how LINZ acquires, ingests and publishes Aerial and satellite imagery see [linz/imagery](https://github.com/linz/imagery) -While linz/basemaps is public, its primary reason is to [work in the open](), so that we can share knoweldge and foster collaboration with peers across the world. While LINZ Basemaps serivce and product can be customised and deployed it is primarily focused on LINZ's usecases of sharing LINZ's aerial imagery archive and Topographic Vector Data +While linz/basemaps is public, its primary reason is to [work in the open](), so that we can share knoweldge and foster collaboration with peers across the world. While LINZ Basemaps serivce and product can be customised and deployed it is primarily focused on LINZ's usecases of sharing LINZ's aerial imagery archive and Topographic Vector Data ## Basemaps Service @@ -23,7 +23,7 @@ While linz/basemaps is public, its primary reason is to [work in the open](), s A tile request flows a number of basemaps packages: -Given a request `/v1/tiles/aerial/WebMercatorQuad/6/1/40.webp`, +Given a request `/v1/tiles/aerial/WebMercatorQuad/6/1/40.webp`, Breaking down this request url `lambda-tiler` needs to serve a `webp` image that for the tile `{z: 6, x: 1, y: 40}` from the `WebMercatorQuad` tile matrix from the tile set named `aerial` in the default configuration. Which will then utlize the following packages @@ -34,27 +34,25 @@ Which will then utlize the following packages - [@basemaps/tiler](../packages/tiler/README.md) Determine which parts of the tiffs need to be loaded and resampled - [@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) - Load the bytes from the tiff and render it as a output image. -Imagery is stored as cloud optimised geotiffs by survey then layered together dynamically based on a [configured priority order](./configuration.md). +Imagery is stored as cloud optimised geotiffs by survey then layered together dynamically based on a [configured priority order](./configuration.md). [@basemaps/tiler](../packages/tiler/README.md) takes the collection of tiffs and determines how resample (resize/crop) and paint them onto the output tile. ![Tiff Operations](./static/tile-resize.png) - [@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) uses [sharp/libvips](https://github.com/lovell/sharp) to take the output of [@basemaps/tiler](../packages/tiler/README.md) then load the raw tiff bytes and render them into output image. ### Vector tile requests Vector tiles pre-rendered as [mapbox vector tiles](https://docs.mapbox.com/data/tilesets/guides/vector-tiles-introduction/) and served directly from a [cloud optimised tar](https://github.com/linz/cotar) since there is no merging or dynamic configuration the tile server will just request the specified tile directly from the tar `` +### Additional Processing -### Additional Processing - -#### Aerial & Satellite Imagery Processing +#### Aerial & Satellite Imagery Processing To efficently serve Aerial Imagery to huge of consumers, it should be optimised into a format that makes it easy to serve. All of LINZ's imagery is stored as [cloud optimised geotiffs (COG)](https://www.cogeo.org/), to ensure they are served as efficently as possible some additional processing steps are generally pre-applied to the imagery. -- Reprojection: reprojecting the imagery into common consumption formats (EPSG:3857 and EPSG:2193) will greatly reduce the amount of load and complexity of the basemaps service service +- Reprojection: reprojecting the imagery into common consumption formats (EPSG:3857 and EPSG:2193) will greatly reduce the amount of load and complexity of the basemaps service service - Tile Alignment: By aligning the COGs to the output XYZ tile grid - Additional overviews: It is hard to fetch data from 1,000s of tiffs to create a single tile, so we create additional overviews up to z0. @@ -62,26 +60,25 @@ These additional processing steps are part of the basemaps [import process](#TOD The additional steps are optional but do **greatly improve** the tile serving performance of the service. - #### Vector Data Vector data is processed from the LINZ topographic vector datasets using tippecanoe, this process is still a work in progress and will. - ## Development Deployments are handled by Github Actions see [Deployments](./deployment.md) for more information. -## Research +## Research ### Imagery + - [COG Quality](./imagery/cog.quality.md) - Resampling methods and their affect on output quality COG - [Relief Shade](./imagery/relief.shade.md) - Relief shading for vector tiles using only imagery "overlay" ### API -- [Empty Tiles API](./api/empty-tiles.md) - Tile API specification research HTTP 204 vs HTTP 200 for non existing tiles +- [Empty Tiles API](./api/empty-tiles.md) - Tile API specification research HTTP 204 vs HTTP 200 for non existing tiles ### Diagrams -All diagrams across this repository are created with [excalidraw](https://excalidraw.com/) and embed the excalidraw configuration inside them, so can be re-edited on https://excalidraw.com/ \ No newline at end of file +All diagrams across this repository are created with [excalidraw](https://excalidraw.com/) and embed the excalidraw configuration inside them, so can be re-edited on https://excalidraw.com/ diff --git a/docs/api/empty-tiles.md b/docs/api/empty-tiles.md index 4548dbf16..2d3c36799 100644 --- a/docs/api/empty-tiles.md +++ b/docs/api/empty-tiles.md @@ -62,7 +62,7 @@ Pros: - ✔️ Supported in ArcGIS Pro and QGIS - ✔️ Supported by Leaflet and OpenLayers -- ✔️ Supported by [HTTP RFC](https://www.rfc-editor.org/rfc/rfc9110.html#name-200-ok): "server ought to return 204 No content" +- ✔️ Supported by [HTTP RFC](https://www.rfc-editor.org/rfc/rfc9110.html#name-200-ok): "server ought to return 204 No content" - ✔️ Recommended option from OGC Tile API Cons: diff --git a/docs/configuration.md b/docs/configuration.md index bf9f2c31d..5d5e0c150 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -9,7 +9,7 @@ The configuration can be stored as individual AWS DynamoDB objects or can be sto The configuration is generally generated directly from the source tiffs and associated STAC metadata using the [@basemaps/cli](../packages/cli/README.md) import process, with LINZ's specific configuration being stored in [linz/basemaps](https://github.com/linz/basemaps). -This allows the configuration objects to be somewhat short, below is a snippet from the [aerial](https://github.com/linz/basemaps-config/blob/master/config/tileset/aerial.json) tile set configuration which layers 100+ aerial imagery layers together. +This allows the configuration objects to be somewhat short, below is a snippet from the [aerial](https://github.com/linz/basemaps-config/blob/master/config/tileset/aerial.json) tile set configuration which layers 100+ aerial imagery layers together. ```json5 { @@ -41,8 +41,8 @@ This allows the configuration objects to be somewhat short, below is a snippet f } ``` -When this configuration is imported it will list all of the source locations looking for `collection.json` and `*.tiff|*.tif` files. +When this configuration is imported it will list all of the source locations looking for `collection.json` and `*.tiff|*.tif` files. ### Dynamic configuration -Configuration can be stored as single JSON object in s3 and then passed directly to the basemaps service with `?config=s3://linz-basemaps/config/config-latest.json.gz` this allows basemaps to preview a configuration before being deployed. \ No newline at end of file +Configuration can be stored as single JSON object in s3 and then passed directly to the basemaps service with `?config=s3://linz-basemaps/config/config-latest.json.gz` this allows basemaps to preview a configuration before being deployed. diff --git a/docs/deployment.md b/docs/deployment.md index 80012a7df..19a1062d1 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -1,4 +1,5 @@ # Deployment + Deployments of Basemaps are managed with github actions. To trigger a deployment, make sure your branch is up to date and run the version bump script [./scripts/version.bump.sh](./scripts/version.bump.sh) @@ -13,9 +14,9 @@ If a Deployment Breaks Badly, don't try to fix that on the fly, this risk of int As Basemaps deployments are managed with github actions, every release will bundle the release packages and deployment in the github action automatically. So, it is very simple for use to roll back to previous release as all the previous deployment are remained in the github actions history. Please use the following steps to trigger a deployment roll when needed. - Open the basemaps deployments page - https://github.com/linz/basemaps/deployments -- Choose the environment that needs to be rolled back -- Open the workflow run for a older release +- Choose the environment that needs to be rolled back +- Open the workflow run for a older release ![Workflow runs for older releases](./static/workflow-run.png | width=400) -- Select `Build / deploy-prod (push)` workflow and rerun the it to roll back to previous release. \ No newline at end of file +- Select `Build / deploy-prod (push)` workflow and rerun the it to roll back to previous release. diff --git a/docs/imagery/README.md b/docs/imagery/README.md index 38c412726..4640f432b 100644 --- a/docs/imagery/README.md +++ b/docs/imagery/README.md @@ -1,4 +1,4 @@ # Image processing notes - [COG Quality](./cog.quality.md) - Resampling methods and their affect on output quality COG -- [Relief Shade](./relief.shade.md) - Relief shading for vector tiles using only imagery "overlay" \ No newline at end of file +- [Relief Shade](./relief.shade.md) - Relief shading for vector tiles using only imagery "overlay" diff --git a/docs/imagery/cog.quality.md b/docs/imagery/cog.quality.md index 176262872..55a2dbcfb 100644 --- a/docs/imagery/cog.quality.md +++ b/docs/imagery/cog.quality.md @@ -28,7 +28,6 @@ gdal_translate \ done ``` - ## Overviews When creating COGs using the different resampling methods the output quality differs a lot. @@ -37,13 +36,14 @@ Here are some overviews created using some of the different resampling methods: ![Resampling Quality](./images/quality__resampling-overview.webp) -### Lanczos +### Lanczos Provides quite a nice and crisp output ![Lanczos Overview](./images/quality__i6.lanczos.webp) ### Cubic + Creates weird artifacts around the edges ![Cubic Overview](./images/quality__i6.cubic.webp) @@ -54,7 +54,6 @@ looks blury ![Bilinear Overview](./images/quality__i6.bilinear.webp) - Using lanczos looks best for our overview creation. ## Reprojection resampling @@ -75,12 +74,10 @@ Creates a nice image ![Bilinear](./images/quality__005_006_0_bilinear.webp) - For our aerial imagery reprojection seems to be best done as bilinear, where as Overview creation seems to be best done as lanczos. - Using GDAL 3.2.x+ this can be configured with as it was fixed in [osgeo/gdal#2671](https://github.com/OSGeo/gdal/issues/2671) ```bash gdal_translate -co WARP_RESAMPLING=bilinear -co OVERVIEW_RESAMPLING=lanczos -``` \ No newline at end of file +``` diff --git a/docs/imagery/relief.shade.md b/docs/imagery/relief.shade.md index 18183af47..9a274bc8f 100644 --- a/docs/imagery/relief.shade.md +++ b/docs/imagery/relief.shade.md @@ -5,8 +5,8 @@ - 8M DEM of NZ - [lds:51768](https://data.linz.govt.nz/layer/51768-nz-8m-digital-elevation-model-2012/) - GDAL - Create a VRT of all the imagery + ```bash gdalbuildvrt dem/*.tif dem.vrt ``` @@ -37,7 +37,6 @@ Base Image: gdaldem hillshade: ![Aerial imagery](./images/relief__base.webp) - Using default color ramp applys whites ![Lighten](./images/relief__lighten.webp) @@ -51,6 +50,6 @@ nv 0 0 0 0 255 0 0 0 0 ``` -`nv` - No value or NO_DATA set full alpha +`nv` - No value or NO_DATA set full alpha -![Lighten](./images/relief__darken.webp) \ No newline at end of file +![Lighten](./images/relief__darken.webp) diff --git a/docs/imagery/texutre.shade.md b/docs/imagery/texutre.shade.md index dd109a6cc..dc387f2ad 100644 --- a/docs/imagery/texutre.shade.md +++ b/docs/imagery/texutre.shade.md @@ -4,7 +4,6 @@ See http://www.textureshading.com/Home.html - ## Process ```bash @@ -14,7 +13,7 @@ See http://www.textureshading.com/Home.html # # This process requires a huge amount of memory, for the 8M geographx DEM it requires around 100GB of free memory # This process has been tested on a AWS R5D with 32VCPUs and 256GB of ram, starting with a base AWS centos image -# +# ################# # Install required software sudo yum groupinstall 'Development Tools' @@ -24,7 +23,7 @@ sudo yum install tmux git sudo systemctl start docker sudo docker ps -# Helper function to run docker +# Helper function to run docker function gdal_docker() { sudo docker run --rm -it -v $PWD:$PWD --workdir $PWD osgeo/gdal:ubuntu-small-3.3.3 "$@" } @@ -70,7 +69,7 @@ gdal_docker gdalinfo Geographx-NZ-DEM-FLT.flt sudo chown ssm-user:ssm-user n1 -R # Texture shading takes a while run inside of tmux so the session can be resumed with `tmux attach` -tmux +tmux ../texture-shade/bin/texture 1 Geographx-NZ-DEM-FLT.flt Geographx-NZ-DEM-FLT.detail_1.flt ../texture-shade/bin/texture_image +1.5 Geographx-NZ-DEM-FLT.detail_1.flt Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.tif # texture-shade only allows ".tif" @@ -84,11 +83,10 @@ gdal_docker gdal_translate -of Gtiff -co COMPRESS=lzw -co BIGTIFF=yes -co NUM_TH aws s3 cp Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.tiff s3://linz-basemaps-source/Geographx-NZ-DEM-FLT/ ``` - Once complete this gives a uint16 tiff that has values between 0-65,536 to make this image usage a color ramp is applied to create a semi transparent image - At 0 create a 25% opacity black, `rgba(0,0,0,0.25)` then slowly ramp to 0% opacity at 32,768 + ``` nv 0 0 0 0 0 0 0 0 63 @@ -97,7 +95,6 @@ nv 0 0 0 0 Using gdal apply the color-relief - ```bash gdaldem color-relief \ -co COMPRESS=lzw \ @@ -107,4 +104,4 @@ gdaldem color-relief \ -co BIGTIFF=yes \ -co ALPHA=yes -alpha \ Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.tiff color.ramp Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.ramp.tiff -``` \ No newline at end of file +```