Skip to content

Commit

Permalink
refactor: apply formatter to docs
Browse files Browse the repository at this point in the history
  • Loading branch information
blacha committed Nov 2, 2023
1 parent f8c0693 commit 0482b67
Show file tree
Hide file tree
Showing 8 changed files with 31 additions and 40 deletions.
25 changes: 11 additions & 14 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@

Basemaps currently supports both [Imagery](#aerial--satellite-imagery) and [Vector data](#vector-data)

## Background
## Background

Land Information New Zealand has a large archive of aerial and satellite imagery freely aviable from [LINZ Data Service](https://data.linz.govt.nz) and [linz/imagery](https://github.com/linz/imagery) this imagery was a primary driver for the creation of the linz basemaps product and service. For more information on how LINZ acquires, ingests and publishes Aerial and satellite imagery see [linz/imagery](https://github.com/linz/imagery)

While linz/basemaps is public, its primary reason is to [work in the open](), so that we can share knoweldge and foster collaboration with peers across the world. While LINZ Basemaps serivce and product can be customised and deployed it is primarily focused on LINZ's usecases of sharing LINZ's aerial imagery archive and Topographic Vector Data
While linz/basemaps is public, its primary reason is to [work in the open](), so that we can share knoweldge and foster collaboration with peers across the world. While LINZ Basemaps serivce and product can be customised and deployed it is primarily focused on LINZ's usecases of sharing LINZ's aerial imagery archive and Topographic Vector Data

## Basemaps Service

Expand All @@ -23,7 +23,7 @@ While linz/basemaps is public, its primary reason is to [work in the open](), s

A tile request flows a number of basemaps packages:

Given a request `/v1/tiles/aerial/WebMercatorQuad/6/1/40.webp`,
Given a request `/v1/tiles/aerial/WebMercatorQuad/6/1/40.webp`,
Breaking down this request url `lambda-tiler` needs to serve a `webp` image that for the tile `{z: 6, x: 1, y: 40}` from the `WebMercatorQuad` tile matrix from the tile set named `aerial` in the default configuration.

Which will then utlize the following packages
Expand All @@ -34,54 +34,51 @@ Which will then utlize the following packages
- [@basemaps/tiler](../packages/tiler/README.md) Determine which parts of the tiffs need to be loaded and resampled
- [@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) - Load the bytes from the tiff and render it as a output image.

Imagery is stored as cloud optimised geotiffs by survey then layered together dynamically based on a [configured priority order](./configuration.md).
Imagery is stored as cloud optimised geotiffs by survey then layered together dynamically based on a [configured priority order](./configuration.md).

[@basemaps/tiler](../packages/tiler/README.md) takes the collection of tiffs and determines how resample (resize/crop) and paint them onto the output tile.

![Tiff Operations](./static/tile-resize.png)


[@basemaps/tiler-sharp](../packages/tiler-sharp/README.md) uses [sharp/libvips](https://github.com/lovell/sharp) to take the output of [@basemaps/tiler](../packages/tiler/README.md) then load the raw tiff bytes and render them into output image.

### Vector tile requests

Vector tiles pre-rendered as [mapbox vector tiles](https://docs.mapbox.com/data/tilesets/guides/vector-tiles-introduction/) and served directly from a [cloud optimised tar](https://github.com/linz/cotar) since there is no merging or dynamic configuration the tile server will just request the specified tile directly from the tar ``

### Additional Processing

### Additional Processing

#### Aerial & Satellite Imagery Processing
#### Aerial & Satellite Imagery Processing

To efficently serve Aerial Imagery to huge of consumers, it should be optimised into a format that makes it easy to serve. All of LINZ's imagery is stored as [cloud optimised geotiffs (COG)](https://www.cogeo.org/), to ensure they are served as efficently as possible some additional processing steps are generally pre-applied to the imagery.

- Reprojection: reprojecting the imagery into common consumption formats (EPSG:3857 and EPSG:2193) will greatly reduce the amount of load and complexity of the basemaps service service
- Reprojection: reprojecting the imagery into common consumption formats (EPSG:3857 and EPSG:2193) will greatly reduce the amount of load and complexity of the basemaps service service
- Tile Alignment: By aligning the COGs to the output XYZ tile grid
- Additional overviews: It is hard to fetch data from 1,000s of tiffs to create a single tile, so we create additional overviews up to z0.

These additional processing steps are part of the basemaps [import process](#TODO) and are controlled by [@basemaps/cogify](../packages/cogify/README.md)

The additional steps are optional but do **greatly improve** the tile serving performance of the service.


#### Vector Data

Vector data is processed from the LINZ topographic vector datasets using tippecanoe, this process is still a work in progress and will.


## Development

Deployments are handled by Github Actions see [Deployments](./deployment.md) for more information.

## Research
## Research

### Imagery

- [COG Quality](./imagery/cog.quality.md) - Resampling methods and their affect on output quality COG
- [Relief Shade](./imagery/relief.shade.md) - Relief shading for vector tiles using only imagery "overlay"

### API
- [Empty Tiles API](./api/empty-tiles.md) - Tile API specification research HTTP 204 vs HTTP 200 for non existing tiles

- [Empty Tiles API](./api/empty-tiles.md) - Tile API specification research HTTP 204 vs HTTP 200 for non existing tiles

### Diagrams

All diagrams across this repository are created with [excalidraw](https://excalidraw.com/) and embed the excalidraw configuration inside them, so can be re-edited on https://excalidraw.com/
All diagrams across this repository are created with [excalidraw](https://excalidraw.com/) and embed the excalidraw configuration inside them, so can be re-edited on https://excalidraw.com/
2 changes: 1 addition & 1 deletion docs/api/empty-tiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Pros:

- ✔️ Supported in ArcGIS Pro and QGIS
- ✔️ Supported by Leaflet and OpenLayers
- ✔️ Supported by [HTTP RFC](https://www.rfc-editor.org/rfc/rfc9110.html#name-200-ok): "server ought to return 204 No content"
- ✔️ Supported by [HTTP RFC](https://www.rfc-editor.org/rfc/rfc9110.html#name-200-ok): "server ought to return 204 No content"
- ✔️ Recommended option from OGC Tile API

Cons:
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The configuration can be stored as individual AWS DynamoDB objects or can be sto

The configuration is generally generated directly from the source tiffs and associated STAC metadata using the [@basemaps/cli](../packages/cli/README.md) import process, with LINZ's specific configuration being stored in [linz/basemaps](https://github.com/linz/basemaps).

This allows the configuration objects to be somewhat short, below is a snippet from the [aerial](https://github.com/linz/basemaps-config/blob/master/config/tileset/aerial.json) tile set configuration which layers 100+ aerial imagery layers together.
This allows the configuration objects to be somewhat short, below is a snippet from the [aerial](https://github.com/linz/basemaps-config/blob/master/config/tileset/aerial.json) tile set configuration which layers 100+ aerial imagery layers together.

```json5
{
Expand Down Expand Up @@ -41,8 +41,8 @@ This allows the configuration objects to be somewhat short, below is a snippet f
}
```

When this configuration is imported it will list all of the source locations looking for `collection.json` and `*.tiff|*.tif` files.
When this configuration is imported it will list all of the source locations looking for `collection.json` and `*.tiff|*.tif` files.

### Dynamic configuration

Configuration can be stored as single JSON object in s3 and then passed directly to the basemaps service with `?config=s3://linz-basemaps/config/config-latest.json.gz` this allows basemaps to preview a configuration before being deployed.
Configuration can be stored as single JSON object in s3 and then passed directly to the basemaps service with `?config=s3://linz-basemaps/config/config-latest.json.gz` this allows basemaps to preview a configuration before being deployed.
7 changes: 4 additions & 3 deletions docs/deployment.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Deployment

Deployments of Basemaps are managed with github actions.

To trigger a deployment, make sure your branch is up to date and run the version bump script [./scripts/version.bump.sh](./scripts/version.bump.sh)
Expand All @@ -13,9 +14,9 @@ If a Deployment Breaks Badly, don't try to fix that on the fly, this risk of int
As Basemaps deployments are managed with github actions, every release will bundle the release packages and deployment in the github action automatically. So, it is very simple for use to roll back to previous release as all the previous deployment are remained in the github actions history. Please use the following steps to trigger a deployment roll when needed.

- Open the basemaps deployments page - https://github.com/linz/basemaps/deployments
- Choose the environment that needs to be rolled back
- Open the workflow run for a older release
- Choose the environment that needs to be rolled back
- Open the workflow run for a older release

![Workflow runs for older releases](./static/workflow-run.png | width=400)

- Select `Build / deploy-prod (push)` workflow and rerun the it to roll back to previous release.
- Select `Build / deploy-prod (push)` workflow and rerun the it to roll back to previous release.
2 changes: 1 addition & 1 deletion docs/imagery/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Image processing notes

- [COG Quality](./cog.quality.md) - Resampling methods and their affect on output quality COG
- [Relief Shade](./relief.shade.md) - Relief shading for vector tiles using only imagery "overlay"
- [Relief Shade](./relief.shade.md) - Relief shading for vector tiles using only imagery "overlay"
9 changes: 3 additions & 6 deletions docs/imagery/cog.quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ gdal_translate \
done
```


## Overviews

When creating COGs using the different resampling methods the output quality differs a lot.
Expand All @@ -37,13 +36,14 @@ Here are some overviews created using some of the different resampling methods:

![Resampling Quality](./images/quality__resampling-overview.webp)

### Lanczos
### Lanczos

Provides quite a nice and crisp output

![Lanczos Overview](./images/quality__i6.lanczos.webp)

### Cubic

Creates weird artifacts around the edges

![Cubic Overview](./images/quality__i6.cubic.webp)
Expand All @@ -54,7 +54,6 @@ looks blury

![Bilinear Overview](./images/quality__i6.bilinear.webp)


Using lanczos looks best for our overview creation.

## Reprojection resampling
Expand All @@ -75,12 +74,10 @@ Creates a nice image

![Bilinear](./images/quality__005_006_0_bilinear.webp)


For our aerial imagery reprojection seems to be best done as bilinear, where as Overview creation seems to be best done as lanczos.


Using GDAL 3.2.x+ this can be configured with as it was fixed in [osgeo/gdal#2671](https://github.com/OSGeo/gdal/issues/2671)

```bash
gdal_translate -co WARP_RESAMPLING=bilinear -co OVERVIEW_RESAMPLING=lanczos
```
```
7 changes: 3 additions & 4 deletions docs/imagery/relief.shade.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
- 8M DEM of NZ - [lds:51768](https://data.linz.govt.nz/layer/51768-nz-8m-digital-elevation-model-2012/)
- GDAL


Create a VRT of all the imagery

```bash
gdalbuildvrt dem/*.tif dem.vrt
```
Expand Down Expand Up @@ -37,7 +37,6 @@ Base Image:
gdaldem hillshade:
![Aerial imagery](./images/relief__base.webp)


Using default color ramp applys whites
![Lighten](./images/relief__lighten.webp)

Expand All @@ -51,6 +50,6 @@ nv 0 0 0 0
255 0 0 0 0
```

`nv` - No value or NO_DATA set full alpha
`nv` - No value or NO_DATA set full alpha

![Lighten](./images/relief__darken.webp)
![Lighten](./images/relief__darken.webp)
13 changes: 5 additions & 8 deletions docs/imagery/texutre.shade.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
See http://www.textureshading.com/Home.html


## Process

```bash
Expand All @@ -14,7 +13,7 @@ See http://www.textureshading.com/Home.html
#
# This process requires a huge amount of memory, for the 8M geographx DEM it requires around 100GB of free memory
# This process has been tested on a AWS R5D with 32VCPUs and 256GB of ram, starting with a base AWS centos image
#
#
#################
# Install required software
sudo yum groupinstall 'Development Tools'
Expand All @@ -24,7 +23,7 @@ sudo yum install tmux git
sudo systemctl start docker
sudo docker ps

# Helper function to run docker
# Helper function to run docker
function gdal_docker() {
sudo docker run --rm -it -v $PWD:$PWD --workdir $PWD osgeo/gdal:ubuntu-small-3.3.3 "$@"
}
Expand Down Expand Up @@ -70,7 +69,7 @@ gdal_docker gdalinfo Geographx-NZ-DEM-FLT.flt
sudo chown ssm-user:ssm-user n1 -R

# Texture shading takes a while run inside of tmux so the session can be resumed with `tmux attach`
tmux
tmux
../texture-shade/bin/texture 1 Geographx-NZ-DEM-FLT.flt Geographx-NZ-DEM-FLT.detail_1.flt
../texture-shade/bin/texture_image +1.5 Geographx-NZ-DEM-FLT.detail_1.flt Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.tif # texture-shade only allows ".tif"

Expand All @@ -84,11 +83,10 @@ gdal_docker gdal_translate -of Gtiff -co COMPRESS=lzw -co BIGTIFF=yes -co NUM_TH
aws s3 cp Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.tiff s3://linz-basemaps-source/Geographx-NZ-DEM-FLT/
```


Once complete this gives a uint16 tiff that has values between 0-65,536 to make this image usage a color ramp is applied to create a semi transparent image


At 0 create a 25% opacity black, `rgba(0,0,0,0.25)` then slowly ramp to 0% opacity at 32,768

```
nv 0 0 0 0
0 0 0 0 63
Expand All @@ -97,7 +95,6 @@ nv 0 0 0 0

Using gdal apply the color-relief


```bash
gdaldem color-relief \
-co COMPRESS=lzw \
Expand All @@ -107,4 +104,4 @@ gdaldem color-relief \
-co BIGTIFF=yes \
-co ALPHA=yes -alpha \
Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.tiff color.ramp Geographx-NZ-DEM-FLT.detail_1.contrast_1.5.lzw.ramp.tiff
```
```

0 comments on commit 0482b67

Please sign in to comment.