From ddd938145d5633864a3e6eff7742f57190adf1d4 Mon Sep 17 00:00:00 2001 From: Zachary Deziel Date: Wed, 20 Dec 2023 15:20:54 -0500 Subject: [PATCH 1/4] Review cogs-examples notebook --- cloud-optimized-geotiffs/cogs-examples.ipynb | 48 +++++++++++--------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/cloud-optimized-geotiffs/cogs-examples.ipynb b/cloud-optimized-geotiffs/cogs-examples.ipynb index 2dd6b66..02ee52f 100644 --- a/cloud-optimized-geotiffs/cogs-examples.ipynb +++ b/cloud-optimized-geotiffs/cogs-examples.ipynb @@ -9,6 +9,7 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "793aee06", "metadata": {}, @@ -27,25 +28,26 @@ "mamba env create -f environment.yml\n", "```\n", "\n", - "then activate and select the kernel in the notebook (running in Jupyter)\n", + "Finally, you may activate and select the kernel in the notebook (running in Jupyter)\n", "\n", "```bash\n", "conda activate coguide-cog\n", "```\n", "\n", - "This notebook has been tested to work with the listed Conda environment." + "The notebook has been tested to work with the listed Conda environment." ] }, { + "attachments": {}, "cell_type": "markdown", "id": "5a8aa8b4", "metadata": {}, "source": [ "## Setup\n", "\n", - "For demonstrating some COG concepts, we will download a regular GeoTIFF, create a Cloud-Optimized GeoTIFF and explore how they are different.\n", + "To demonstrate some COG concepts, we will download a regular GeoTIFF, create a Cloud-Optimized GeoTIFF, and explore their differences.\n", "\n", - "First we use the [earthaccess](https://github.com/nsidc/earthaccess) library to setup credentials to fetch data from NASA's EarthData catalog." + "First, we use the [earthaccess](https://github.com/nsidc/earthaccess) library to set up credentials to fetch data from NASA's EarthData catalog." ] }, { @@ -102,13 +104,14 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "8ecd2c7c", "metadata": {}, "source": [ "## Download a GeoTIFF from EarthData\n", "\n", - "Note: The whole point of is that we _don't_ download data. So in future examples, we will demonstrate how to access just subsets of data using COG and compare that with a GeoTIFF." + "Note: The whole point is that we _don't_ download data. So, in future examples, we will demonstrate how to access just subsets of data using COG and compare that with a GeoTIFF." ] }, { @@ -188,13 +191,14 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "451dbe01", "metadata": {}, "source": [ "## Is it a valid COG?\n", "\n", - "We can use `rio_cogeo.cog_validate` to check. It returns `is_valid`, `errors` and `warnings`." + "We can use `rio_cogeo.cog_validate` to check. It returns `is_valid`, `errors`, and `warnings`:" ] }, { @@ -232,11 +236,12 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "25a82b89", "metadata": {}, "source": [ - "Return values:\n", + "Here's some more context on the output message:\n", " \n", "1. `is_valid` is `False`: this is not a valid COG.\n", "2. `errors` are `'The file is greater than 512xH or 512xW, but is not tiled'`. To be a valid COG, the file should be tiled since it has a height and width both greater than 512.\n", @@ -244,13 +249,14 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "f2826f57-6df4-458f-98b2-f639f25450ba", "metadata": {}, "source": [ "## Converting a GeoTIFF to COG\n", "\n", - "We can use `rio_cogeo.cog_create` to convert a GeoTIFF into a Cloud Optimized GeoTIFF" + "We can use `rio_cogeo.cog_create` to convert a GeoTIFF into a Cloud Optimized GeoTIFF:" ] }, { @@ -299,25 +305,25 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "35f1787d", "metadata": {}, "source": [ - "This is a valid COG, so we will use it to compare with our GeoTIFF." + "The file is a valid COG, so we will use it to compare with our GeoTIFF." ] }, { + "attachments": {}, "cell_type": "markdown", "id": "ff2cc531", "metadata": {}, "source": [ "## Data Structure\n", "\n", - "**Dimensions**\n", - "Dimensions are the number of bands, rows and columns stored in a GeoTIFF. [More Info](intro.qmd#dimensions)\n", + "**Dimensions**: the number of bands, rows, and columns stored in a GeoTIFF. [More Info](intro.qmd#dimensions)\n", "\n", - "**Internal Blocks** (aka chunks or internal tiles)\n", - "Internal blocks are required if the dimensions of data are over 512x512. [More Info](intro.qmd#internal-blocks)\n", + "**Internal Blocks (chunks or internal tiles)**: Internal blocks are required if the data dimensions are over 512x512. [More Info](intro.qmd#internal-blocks)\n", "\n", "Let's check out the dimensions and blocks of our GeoTIFF and Cloud-Optimized GeoTIFF." ] @@ -363,13 +369,14 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "51828845-3531-4813-b694-25a4563a58f5", "metadata": {}, "source": [ - "They have the same dimensions which is what we expect, so that is good!\n", + "They have the exact dimensions that we expected, which is good!\n", "\n", - "We can also print information about the GeoTIFF's IFD (Internal File Directory). Only one item is returned because the GeoTIFF doesn't have overviews. When we print the IFD info for the COG, which has overviews, we see more items returned." + "We can also print information about the GeoTIFF's IFD (Internal File Directory). Only one item is returned because the GeoTIFF needs overviews. We see more items returned when we print the IFD info for the COG, which has overviews." ] }, { @@ -426,6 +433,7 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "12c0b3e4", "metadata": {}, @@ -434,14 +442,12 @@ "\n", "Overviews are downsampled (aggregated) data intended for visualization.\n", "\n", - "The smallest size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation just aim to have at least one dimension at or slightly less than 256. \n", - "> The COG driver in GDAL, or rio cogeo tools should do this.\n", + "The most miniature size overview should match the tiling components’ fetch size, typically 256x256. Due to aspect ratio variation, aim to have at least one dimension at or slightly less than 256. \n", + "> The COG driver in GDAL or rio cogeo tools should do this.\n", "\n", - "There are many resampling algorithms for generating overviews. The best resampling algorithm depends on the range, type, and distribution of the data. When creating overviews several options should be compared before deciding which resampling method to apply. \n", + "There are many resampling algorithms for generating overviews. The best resampling algorithm depends on the data's range, type, and distribution. When creating overviews, several options should be compared before deciding which resampling method to apply. \n", "\n", - "GDAL >= 3.2 allows for the overview resampling method to be set directly.\n", - "\n", - "" + "GDAL >= 3.2 allows for the overview resampling method to be set directly." ] }, { From 1176c85d2a6f34fd587563b274145d6275ae13c3 Mon Sep 17 00:00:00 2001 From: Zachary Deziel Date: Wed, 20 Dec 2023 15:33:33 -0500 Subject: [PATCH 2/4] Review overview resampling notebook --- .../cogs-overview_resampling.ipynb | 109 +++++++++++++----- 1 file changed, 80 insertions(+), 29 deletions(-) diff --git a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb index 209bb67..6662126 100644 --- a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb +++ b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb @@ -1,19 +1,20 @@ { "cells": [ { + "attachments": {}, "cell_type": "markdown", "id": "37c4c227", "metadata": {}, "source": [ "# Cloud Optimized Geotiff (COG) Overview Resampling\n", "\n", - "When making Cloud-Optimized GeoTIFFs (COGs) you can select the resampling method used to generate the overviews. Different types of data can drastically change appearance when more zoomed out based on the method selected. This is most important when using software that renders from overviews (e.g QGIS, ArcGIS), particularly web tilers like [Titiler](https://developmentseed.org/titiler/). \n", + "When making Cloud-Optimized GeoTIFFs (COGs), you can select the resampling method to generate the overviews. Different data types can drastically change appearance when more zoomed out based on the method chosen. This is most important when using software that renders from overviews (e.g., QGIS, ArcGIS), particularly web tilers like [Titiler](https://developmentseed.org/titiler/). \n", "\n", - "Also when using a tiling service on high resolution dataset you will often have many tiles of source data. The idea behind this notebook, is that before you generate all your final output data, as COGs, or if you need to rebuild your overviews, you should first test on some representative samples. After testing you'll then know which method to use in your data pipeline.\n", + "Also, when using a tiling service on a high-resolution dataset, you will often have many tiles of source data. The idea behind this notebook is that before you generate all your final output data, as COGs, or if you need to rebuild your overviews, you should first test on some representative samples. After testing, you'll know which method to use in your data pipeline.\n", "\n", - "This notebook loops over the overview resampling methods available in [GDAL](https://gdal.org/) applying it to the same sample tile so that we can compare how the dataset will appear when zoomed out from the full resolution. \n", + "This notebook loops over the overview resampling methods available in [GDAL](https://gdal.org/) applying it to the same sample tile to compare how the dataset will appear when zoomed out from the full resolution. \n", "\n", - "> Note: GDAL 3.2 added the ability to specify the resampling method for the overviews specifically. GDAL 3.3 added a couple of new resampling methods. \n", + "> Note: GDAL 3.2 specifically added the ability to specify the resampling method for the overviews. GDAL 3.3 added a couple of new resampling ways. \n", "\n", "## Setup Environment\n", "\n", @@ -27,19 +28,19 @@ "conda env create -f environment.yml\n", "```\n", "\n", - "or\n", + "Or:\n", "\n", "```bash\n", "mamba env create -f environment.yml\n", "```\n", "\n", - "or you can install the packages you need manually\n", + "Or you can install the packages you need manually:\n", "\n", "```\n", "mamba create -q -y -n coguide-cog -c conda-forge 'gdal>=3.3' 'rasterio>=1.3.3' 'rio-cogeo=3.5.0' ipykernel matplotlib earthaccess\n", "```\n", "\n", - "then activate and select the kernel in the notebook (running in Jupyter)\n", + "Then, you may activate and select the kernel in the notebook (running in Jupyter):\n", "\n", "```bash\n", "conda activate coguide-cog\n", @@ -47,9 +48,11 @@ "\n", "This notebook has been tested to work with the listed Conda environment.\n", "\n", - "Remember to switch your notebook kernel if you made a new env, you may need to activate the new env first.\n", + "Remember to switch your notebook kernel if you made a new env; you may need to activate the new env first.\n", "\n", - "## Setup the code" + "## Setup the code\n", + "\n", + "First, we should verify our GDAL version:" ] }, { @@ -67,7 +70,6 @@ } ], "source": [ - "# Verify the GDAL version\n", "!gdalinfo --version" ] }, @@ -101,6 +103,7 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "1ffbe2a1", "metadata": {}, @@ -109,9 +112,11 @@ "\n", "The example data comes from the [Aboveground Biomass Density for High Latitude Forests from ICESat-2, 2020](https://cmr.earthdata.nasa.gov/search/concepts/C2756302505-ORNL_CLOUD.html)\n", "\n", - "You can see the full mosaic of all the data on the [MAAP Biomass dashboard](https://www.earthdata.nasa.gov/maap-biomass/products/icesat2_boreal?map=-124.0135%2C63.1543%2C2.62&layers=icesat2_boreal_biomass_reduced%2Cicesat2_boreal_biomass&lState=)\n", + "You can see the whole mosaic of all the data on the [MAAP Biomass dashboard](https://www.earthdata.nasa.gov/maap-biomass/products/icesat2_boreal?map=-124.0135%2C63.1543%2C2.62&layers=icesat2_boreal_biomass_reduced%2Cicesat2_boreal_biomass&lState=)\n", + "\n", + "For this example, we will pull a single 3000x3000 pixel tile that is part of the larger dataset of more than 3500 tiles.\n", "\n", - "For this example we will pull a single 3000x3000 pixel tile that is part of the larger dataset of more than 3500 tiles." + "We must retrieve a NASA Earth data session for downloading data:\n" ] }, { @@ -132,10 +137,18 @@ } ], "source": [ - "# Get a NASA Earth Data Session for downloading data\n", "earthaccess.login(strategy=\"interactive\")" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "16dbeee8", + "metadata": {}, + "source": [ + "We can now query NASA CMR to retrieve a download URL: " + ] + }, { "cell_type": "code", "execution_count": 4, @@ -151,7 +164,6 @@ } ], "source": [ - "# Now we query NASA CMR to get the download url\n", "short_name = 'Boreal_AGB_Density_ICESat2_2186'\n", "item = 'Boreal_AGB_Density_ICESat2.boreal_agb_202302061675666220_3741.tif'\n", "\n", @@ -163,6 +175,15 @@ ")" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "035e8746", + "metadata": {}, + "source": [ + "Finally, we can download the COG for local use:" + ] + }, { "cell_type": "code", "execution_count": 5, @@ -201,9 +222,6 @@ } ], "source": [ - "# And then download the Cloud-Optimized GeoTIFF for local use\n", - "# Note: We downloaded the file because, it's small (less than 100 MB), and we're going to use the whole file several times\n", - "\n", "test_data_dir = \"./test_data\"\n", "os.makedirs(test_data_dir, exist_ok=True)\n", "\n", @@ -211,6 +229,17 @@ "tile = f\"{sample_files[0]}\"" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "1b371a17", + "metadata": {}, + "source": [ + "> Note: We downloaded the file because, it's small (less than 100 MB), and we're going to use the whole file several times.\n", + "\n", + "Using rio cogeo, we can investigate if the file has tiles and what resampling was used:\n" + ] + }, { "cell_type": "code", "execution_count": 6, @@ -287,19 +316,18 @@ } ], "source": [ - "# Using rio cogeo we can investigate if the file has tiles and what resampling was used.\n", - "\n", "!rio cogeo info {tile}" ] }, { + "attachments": {}, "cell_type": "markdown", "id": "bec46918", "metadata": {}, "source": [ "## Generate Overviews\n", "\n", - "Now lets generate overviews with each of the resampling methods possible in GDAL." + "Now, let's generate overviews with each resampling method possible in GDAL." ] }, { @@ -344,11 +372,12 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "8577e1d8", "metadata": {}, "source": [ - "Now lets make a list of the Resampling methods offered by GDAL. Some resampling methods aren't appropriate for the data, so we are doing to drop those from the list. You can see descriptions at https://gdal.org/programs/gdalwarp.html#cmdoption-gdalwarp-r " + "Now, we can make a list of the resampling methods offered by GDAL. Some resampling methods aren't appropriate for the data, so we are doing to drop those from the list. You can see descriptions at https://gdal.org/programs/gdalwarp.html#cmdoption-gdalwarp-r " ] }, { @@ -383,6 +412,15 @@ "print(resample_methods)" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "08c7b0b3", + "metadata": {}, + "source": [ + "For each resampling method, we can create a copy of the data with it's given overview method:" + ] + }, { "cell_type": "code", "execution_count": 9, @@ -406,10 +444,18 @@ } ], "source": [ - "# For each resampling method create a copy of the data creating overviews with the method.\n", "files = [generate_overview(tile, test_data_dir, resample_method) for resample_method in resample_methods] \n" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "44d2237c", + "metadata": {}, + "source": [ + "We can validate that the overviews were created: " + ] + }, { "cell_type": "code", "execution_count": 10, @@ -488,8 +534,6 @@ } ], "source": [ - "# Check that the overviews were created\n", - "#!gdalinfo {files[0]} | grep Overviews\n", "!rio cogeo info {files[0]}" ] }, @@ -545,6 +589,15 @@ " return\n" ] }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "cc19c418", + "metadata": {}, + "source": [ + "For each variat, we will add interpretation details below the plot. Notice we are using hard-coded values for the levels:" + ] + }, { "cell_type": "code", "execution_count": 12, @@ -563,9 +616,6 @@ } ], "source": [ - "# plot all the variants\n", - "# interpretation details below the plot\n", - "# hard coding levels for now\n", "fig, ax_list = plt.subplots(ncols=4, nrows=len(files), figsize=(16,32), constrained_layout=True)\n", "row = 0\n", "for file in files:\n", @@ -574,15 +624,16 @@ ] }, { + "attachments": {}, "cell_type": "markdown", "id": "5400bd54", "metadata": {}, "source": [ "## Interpretation\n", "\n", - "With this particular example you can see the default method (CUBIC) over represents the amount of NoData cells as you zoom out (left to right). You can also see how some algorithms hide NoData when zoomed out, and that the different methods vary in their level of smoothness.\n", + "With this particular example, you can see the default method (CUBIC) represents the amount of NoData cells as you zoom out (left to right). You can also see how some algorithms hide NoData when zoomed out, and the different methods vary in their level of smoothness.\n", "\n", - "Depending on what impression you want viewers to get, comparing will help you choose which method to use.\n" + "Depending on what impression you want viewers to get, comparing will help you choose which method to use." ] } ], From 50d8cb2049432f05bce1a1e487df4c634c87c6eb Mon Sep 17 00:00:00 2001 From: Zachary Deziel Date: Fri, 22 Dec 2023 09:34:19 -0500 Subject: [PATCH 3/4] Refactor text details based on review --- cloud-optimized-geotiffs/cogs-examples.ipynb | 4 +++- cloud-optimized-geotiffs/cogs-overview_resampling.ipynb | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/cloud-optimized-geotiffs/cogs-examples.ipynb b/cloud-optimized-geotiffs/cogs-examples.ipynb index 02ee52f..5561478 100644 --- a/cloud-optimized-geotiffs/cogs-examples.ipynb +++ b/cloud-optimized-geotiffs/cogs-examples.ipynb @@ -447,7 +447,9 @@ "\n", "There are many resampling algorithms for generating overviews. The best resampling algorithm depends on the data's range, type, and distribution. When creating overviews, several options should be compared before deciding which resampling method to apply. \n", "\n", - "GDAL >= 3.2 allows for the overview resampling method to be set directly." + "GDAL >= 3.2 allows for the overview resampling method to be set directly.\n", + "\n", + "For more information on overviews, see the [COG overview resampling notebook](cogs-overview_resampling.ipynb)." ] }, { diff --git a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb index 6662126..697e3af 100644 --- a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb +++ b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb @@ -8,7 +8,7 @@ "source": [ "# Cloud Optimized Geotiff (COG) Overview Resampling\n", "\n", - "When making Cloud-Optimized GeoTIFFs (COGs), you can select the resampling method to generate the overviews. Different data types can drastically change appearance when more zoomed out based on the method chosen. This is most important when using software that renders from overviews (e.g., QGIS, ArcGIS), particularly web tilers like [Titiler](https://developmentseed.org/titiler/). \n", + "When making Cloud-Optimized GeoTIFFs (COGs), you can select the resampling method to generate the overviews. Different types of data (e.g., nominal, ordinal, discrete, continuous) can drastically change appearance when more zoomed out based on the chosen method. This is most important when using software that renders from overviews (e.g., QGIS, ArcGIS), particularly web tilers like [Titiler](https://developmentseed.org/titiler/). \n", "\n", "Also, when using a tiling service on a high-resolution dataset, you will often have many tiles of source data. The idea behind this notebook is that before you generate all your final output data, as COGs, or if you need to rebuild your overviews, you should first test on some representative samples. After testing, you'll know which method to use in your data pipeline.\n", "\n", @@ -631,7 +631,7 @@ "source": [ "## Interpretation\n", "\n", - "With this particular example, you can see the default method (CUBIC) represents the amount of NoData cells as you zoom out (left to right). You can also see how some algorithms hide NoData when zoomed out, and the different methods vary in their level of smoothness.\n", + "With this particular example, you can see the default method (CUBIC) over-represents the amount of NoData cells as you zoom out (left to right). You can also see how some algorithms hide NoData when zoomed out, and the different methods vary in their level of smoothness.\n", "\n", "Depending on what impression you want viewers to get, comparing will help you choose which method to use." ] From 5b416a7d4afe94414ac77e836e495354a2189743 Mon Sep 17 00:00:00 2001 From: Zachary Deziel Date: Wed, 3 Jan 2024 09:08:20 -0500 Subject: [PATCH 4/4] Add pointer to NASA Earth Data registration --- cloud-optimized-geotiffs/cogs-examples.ipynb | 27 +++---------------- .../cogs-overview_resampling.ipynb | 2 ++ 2 files changed, 6 insertions(+), 23 deletions(-) diff --git a/cloud-optimized-geotiffs/cogs-examples.ipynb b/cloud-optimized-geotiffs/cogs-examples.ipynb index 5561478..1023b93 100644 --- a/cloud-optimized-geotiffs/cogs-examples.ipynb +++ b/cloud-optimized-geotiffs/cogs-examples.ipynb @@ -47,6 +47,8 @@ "\n", "To demonstrate some COG concepts, we will download a regular GeoTIFF, create a Cloud-Optimized GeoTIFF, and explore their differences.\n", "\n", + "To access and integrate NASA Earth data into your Jupyter Notebook, you can create an account by visiting [NASA's Earthdata Login page](https://urs.earthdata.nasa.gov/users/new). This will enable you to register for an account and retrieve the datasets used in the notebook.\n", + "\n", "First, we use the [earthaccess](https://github.com/nsidc/earthaccess) library to set up credentials to fetch data from NASA's EarthData catalog." ] }, @@ -74,31 +76,10 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "id": "ad220208", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "EARTHDATA_USERNAME and EARTHDATA_PASSWORD are not set in the current environment, try setting them or use a different strategy (netrc, interactive)\n", - "You're now authenticated with NASA Earthdata Login\n", - "Using token with expiration date: 10/24/2023\n", - "Using .netrc file for EDL\n" - ] - }, - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "earthaccess.login()" ] diff --git a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb index 697e3af..d2642d3 100644 --- a/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb +++ b/cloud-optimized-geotiffs/cogs-overview_resampling.ipynb @@ -112,6 +112,8 @@ "\n", "The example data comes from the [Aboveground Biomass Density for High Latitude Forests from ICESat-2, 2020](https://cmr.earthdata.nasa.gov/search/concepts/C2756302505-ORNL_CLOUD.html)\n", "\n", + "To access and integrate NASA Earth data into your Jupyter Notebook, you can create an account by visiting [NASA's Earthdata Login page](https://urs.earthdata.nasa.gov/users/new). This will enable you to register for an account and retrieve the datasets used in the notebook.\n", + "\n", "You can see the whole mosaic of all the data on the [MAAP Biomass dashboard](https://www.earthdata.nasa.gov/maap-biomass/products/icesat2_boreal?map=-124.0135%2C63.1543%2C2.62&layers=icesat2_boreal_biomass_reduced%2Cicesat2_boreal_biomass&lState=)\n", "\n", "For this example, we will pull a single 3000x3000 pixel tile that is part of the larger dataset of more than 3500 tiles.\n",