From a1f491019ea6a0d737cb3d37e606e13e9bf739ce Mon Sep 17 00:00:00 2001 From: Jorge Sanz Date: Thu, 1 Feb 2024 10:27:25 +0100 Subject: [PATCH] [Docs][Maps] Troublesooting common Mapes File Upload issues (#174476) Fixes #135319 Adds a new section to the [Troubleshooting](https://www.elastic.co/guide/en/kibana/current/maps-troubleshooting.html) addressing how to improve geospatial datasets locally before working with Kibana Maps File Upload to increase the chances of a successful operation. * Converting a dataset into a GeoJSON or Shapefile with ogr2ogr * Transforming a dataset into WGS84 with ogr2ogr * Exploding a multigeometry dataset into parts with ogr2ogr * Reducing the precision of a dataset with ogr2ogr * Simplifying region datasets with mapshaper * Options to clean invalid geometries with ogr2ogr and mapshaper --------- Co-authored-by: Nick Peihl (cherry picked from commit 9b5d040e5ecfa915cc80beebeac63179a6d82830) --- docs/maps/clean-data.asciidoc | 183 ++++++++++++++++++++++++++++ docs/maps/index.asciidoc | 1 + docs/maps/trouble-shooting.asciidoc | 2 +- 3 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 docs/maps/clean-data.asciidoc diff --git a/docs/maps/clean-data.asciidoc b/docs/maps/clean-data.asciidoc new file mode 100644 index 0000000000000..538e84be53517 --- /dev/null +++ b/docs/maps/clean-data.asciidoc @@ -0,0 +1,183 @@ +[role="xpack"] +[[maps-clean-your-data]] +=== Clean your data + +// https://github.com/elastic/kibana/issues/135319 + +Geospatial fields in {es} have certain restrictions that need to be addressed before upload. On this section a few recipes will be presented to help troubleshooting common issues on this type of data. + +[float] +==== Convert to GeoJSON or Shapefile + +Use https://gdal.org/programs/ogr2ogr.html[ogr2ogr] (part of the https://gdal.org[GDAL/OGR] suite) to convert datasets into a GeoJSON or Esri Shapefile. For example, use the following commands to convert a GPX file into JSON: + +[source,sh] +---- +# Example GPX file from https://www.topografix.com/gpx_sample_files.asp +# +# Convert the GPX waypoints layer into a GeoJSON file +$ ogr2ogr \ + -f GeoJSON "waypoints.geo.json" \ # Output format and file name + "fells_loop.gpx" \ # Input File Name + "waypoints" # Input Layer (usually same as file name) + +# Extract the routes layer into a GeoJSON file +$ ogr2ogr -f "GeoJSON" "routes.geo.json" "fells_loop.gpx" "routes" +---- + +[float] +==== Convert to WGS84 Coordinate Reference System + +{es} only supports WGS84 Coordinate Reference System. Use `ogr2ogr` to convert from other coordinate systems to WGS84. + +On the following example, `ogr2ogr` transforms a shapefile from https://epsg.org/crs_4269/NAD83.html[NAD83] to https://epsg.org/crs_4326/WGS-84.html[WGS84]. The input CRS is detected automatically thanks to the `.prj` sidecar file in the source dataset. + +[source,sh] +---- +# Example NAD83 file from https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_county_5m.zip +# +# Convert the Census Counties shapefile to WGS84 (EPSG:4326) +$ ogr2ogr -f "Esri Shapefile" \ + "cb_2018_us_county_5m.4326.shp" \ # Output file + -t_srs "EPSG:4326" \ # EPSG:4326 is the code for WGS84 + "cb_2018_us_county_5m.shp" \ # Input file + "cb_2018_us_county_5m" # Input layer +---- + +[float] +==== Improve performance by breaking out complex geometries into one geometry per document + +Sometimes geospatial datasets are composed by a small amount of geometries that contain a very large amount of individual part geometries. A good example of this situation is on detailed world country boundaries datasets where records for countries like Canada or Philippines have hundreds of small island geometries. Depending on the final usage of a dataset, you may want to break out this type of dataset to keep one geometry per document, considerably increasing the performance of your index. + +[source,sh] +---- +# Example NAD83 file from www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ler_000b16a_e.zip +# +# Check the number of input features +$ ogrinfo -summary ler_000b16a_e.shp ler_000b16a_e \ + | grep "Feature Count" +Feature Count: 76 + +# Convert to WGS84 exploding the multiple geometries +$ ogr2ogr \ + -f "Esri Shapefile" \ + "ler_000b16a_e.4326-parts.shp" \ # Output file + -explodecollections \ # Convert multiparts into single records + -t_srs "EPSG:4326" \ # Transform to WGS84 + "ler_000b16a_e.shp" \ # Input file + "ler_000b16a_e" # Input layer + +# Check the number of geometries in the output file +# to confirm the 76 records are exploded into 27 thousand rows +$ ogrinfo -summary ler_000b16a_e.4326-parts.shp ler_000b16a_e.4326 \ + | grep "Feature Count" +Feature Count: 27059 +---- + +[WARNING] +==== +A dataset containing records with a very large amount of parts as the one from the example above may even hang in {kib} Maps file uploader. +==== + +[float] +==== Reduce the precision + +Some machine generated datasets are stored with more decimals than are strictly necessary. For reference, the GeoJSON RFC 7946 https://datatracker.ietf.org/doc/html/rfc7946#section-11.2[coordinate precision section] specifies six digits to be a common default to around 10 centimeters on the ground. The file uploader in the Maps application will automatically reduce the precision to 6 decimals but for big datasets it is better to do this before uploading. + +`ogr2ogr` generates GeoJSON files with 7 decimal degrees when requesting `RFC7946` compliant files but using the `COORDINATE_PRECISION` https://gdal.org/drivers/vector/geojson.html#layer-creation-options[GeoJSON layer creation option] it can be downsized even more if that is OK for the usage of the data. + +[source,sh] +---- +# Example NAD83 file from https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_county_5m.zip +# +# Generate a 2008 GeoJSON file +$ ogr2ogr \ + -f GeoJSON \ + "cb_2018_us_county_5m.4326.geo.json" \ # Output file + -t_srs "EPSG:4326" \ # Convert to WGS84 + -lco "RFC7946=NO" \ # Request a 2008 GeoJSON file + "cb_2018_us_county_5m.shp" \ + "cb_2018_us_county_5m" + +# Generate a RFC7946 GeoJSON file +$ ogr2ogr \ + -f GeoJSON \ + "cb_2018_us_county_5m.4326.RFC7946.geo.json" \ # Output file + -t_srs "EPSG:4326" \ # Convert to WGS84 + -lco "RFC7946=YES" \ # Request a RFC7946 GeoJSON file + "cb_2018_us_county_5m.shp" \ + "cb_2018_us_county_5m" + +# Generate a RFC7946 GeoJSON file with just 5 decimal figures +$ ogr2ogr \ + -f GeoJSON \ + "cb_2018_us_county_5m.4326.RFC7946_mini.geo.json" \ # Output file + -t_srs "EPSG:4326" \ # Convert to WGS84 + -lco "RFC7946=YES" \ # Request a RFC7946 GeoJSON file + -lco "COORDINATE_PRECISION=5" \ # Downsize to just 5 decimal positions + "cb_2018_us_county_5m.shp" \ + "cb_2018_us_county_5m" + +# Compare the disk size of the three output files +$ du -h cb_2018_us_county_5m.4326*.geo.json +7,4M cb_2018_us_county_5m.4326.geo.json +6,7M cb_2018_us_county_5m.4326.RFC7946.geo.json +6,1M cb_2018_us_county_5m.4326.RFC7946_mini.geo.json +---- + + +[float] +==== Simplifying region datasets + +Region datasets are polygon datasets where the boundaries of the documents don't overlap. This is common for administrative boundaries, land usage, and other continuous datasets. This type of datasets has the special feature that any geospatial operation modifying the lines of the polygons needs to be applied in the same way to the common sides of the polygons to avoid the generation of thin gap and overlap artifacts. + +https://github.com/mbloch/mapshaper[`mapshaper`] is an excellent tool to work with this type of datasets as it understands datasets of this nature and works with them accordingly. + +Depending on the usage of a region dataset, different geospatial precisions may be adequate. A world countries dataset that is displayed for the entire planet does not need the same precision as a map of the countries in the South Asian continent. + +`mapshaper` offers a https://github.com/mbloch/mapshaper/wiki/Command-Reference#-simplify[`simplify`] command that accepts percentages, resolutions, and different simplification algorithms. + +[source,sh] +---- +# Example NAD83 file from https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_county_5m.zip +# +# Generate a baseline GeoJSON file from OGR +$ ogr2ogr \ + -f GeoJSON "cb_2018_us_county_5m.ogr.geo.json" \ + -t_srs "EPSG:4326" \ + -lco RFC7946=YES \ + "cb_2018_us_county_5m.shp" \ + "cb_2018_us_county_5m" + +# Simplify at different percentages with mapshaper +$ for pct in 10 50 75 99; do \ + mapshaper \ + -i "cb_2018_us_county_5m.shp" \ # Input file + -proj "EPSG:4326" \ # Output projection + -simplify "${pct}%" \ # Simplification + -o cb_2018_us_county_5m.mapshaper_${pct}.geo.json; \ # Output file + done + +# Compare the size of the output files +$ du -h cb_2018_us_county_5m*.geo.json +2,0M cb_2018_us_county_5m.mapshaper_10.geo.json +4,1M cb_2018_us_county_5m.mapshaper_50.geo.json +5,3M cb_2018_us_county_5m.mapshaper_75.geo.json +6,7M cb_2018_us_county_5m.mapshaper_99.geo.json +6,7M cb_2018_us_county_5m.ogr.geo.json +---- + + +[float] +==== Fixing incorrect geometries + +The Maps application expects valid GeoJSON or Shapefile datasets. Apart from the mentioned CRS requirement, geometries need to be valid. Both `ogr2ogr` and `mapshaper` have options to try to fix invalid geometries: + +* OGR https://gdal.org/programs/ogr2ogr.html#cmdoption-ogr2ogr-makevalid[`-makevalid`] option +* Mapshaper https://github.com/mbloch/mapshaper/wiki/Command-Reference#-clean[`-clean`] command + + +[float] +==== And so much more + +`ogr2ogr` and `mapshaper` are excellent geospatial ETL (Extract Transform and Load) utilities that can do much more than viewed here. Reading the documentation in detail is worth investment to improve the quality of the datasets by removing unwanted fields, refining data types, validating value domains, etc. Finally, being command line utilities, both can be automated and added to QA pipelines. diff --git a/docs/maps/index.asciidoc b/docs/maps/index.asciidoc index f924e60cce9e6..6023cbef8a91d 100644 --- a/docs/maps/index.asciidoc +++ b/docs/maps/index.asciidoc @@ -64,5 +64,6 @@ include::search.asciidoc[] include::map-settings.asciidoc[] include::connect-to-ems.asciidoc[] include::import-geospatial-data.asciidoc[] +include::clean-data.asciidoc[] include::indexing-geojson-data-tutorial.asciidoc[] include::trouble-shooting.asciidoc[] diff --git a/docs/maps/trouble-shooting.asciidoc b/docs/maps/trouble-shooting.asciidoc index 3e4a6dfb42dc1..ebb7ec2a65aa8 100644 --- a/docs/maps/trouble-shooting.asciidoc +++ b/docs/maps/trouble-shooting.asciidoc @@ -53,4 +53,4 @@ Increase <> for large data views. [float] ==== Custom tiles are not displayed * When using a custom tile service, ensure your tile server has configured https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS[Cross-Origin Resource Sharing (CORS)] so tile requests from your {kib} domain have permission to access your tile server domain. -* Ensure custom vector and tile services have the required coordinate system. Vector data must use EPSG:4326 and tiles must use EPSG:3857. \ No newline at end of file +* Ensure custom vector and tile services have the required coordinate system. Vector data must use EPSG:4326 and tiles must use EPSG:3857.