1_level1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XZ_9AP55KxvB"
   },
   "source": [
    "#### Installation\n",
    "\n",
    "The pywapor package depends on several other packages, most of them get installed automatically when we install pywapor. The GDAL package needs to be installed manually however. Luckily, it is already installed on the backend computer used by Google Colab. We can verify that GDAL is installed by running the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from osgeo import gdal\n",
    "print(\"Using gdal version\", gdal.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know that Python is able to import the GDAL package, we can install pywapor by running the following command.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pywapor --quiet"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything went well, we can now import pywapor in Python, let's try it (fingers crossed)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pywapor\n",
    "print(\"Using pywapor version:\", pywapor.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ETLook Input\n",
    "\n",
    "In order to run the ETLook model, we first need to collect the necessary inputs. For your convenience, the pywapor package has a function that can collect all this data from selected sources and make sure the data is stored in the correct format. In one of the following lessons we will look more closely at how you can adjust these inputs. Here we will focus on running a basic configuration.\n",
    "\n",
    "Because some of the data portals used require a user to login with a username and a password, we first need to set those up. Most importantly, we will need a `NASA Earthdata Login` to be able to collect the MODIS and MERRA2 datasets, which can be created [over here](https://urs.earthdata.nasa.gov/users/new).\n",
    "\n",
    "> **Note**\n",
    ">\n",
    "> After creating your account, you still need to accept some 'Terms of Use', before you can continue with this notebook. To do that, login to your newly created account and go to\n",
    ">\n",
    "> `Applications > Authorized Apps > Approve More Applications`\n",
    ">\n",
    "> There, make sure the two following applications are authorized:\n",
    "> 1. `NASA GESDISC DATA ARCHIVE`\n",
    "> 2. `LP DAAC OPeNDAP`\n",
    "\n",
    "Once you have your account set up, we can enter our username and password by running the following code (if you skip this step, you'll be prompted for your username and password once the script starts downloading any product that requires them).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pywapor.collect.accounts.setup(\"NASA\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our account set up we can start defining our period and area-of-interest (AOI). \n",
    "\n",
    "First we define a time period through `timelim`, our boundingbox through `latlim` and `lonlim` and we give a `project_folder` in which all our data will be stored. Here I've chosen an period of exactly 10 days which is the default composite length of pyWAPOR. We'll talk more about composites in the next lesson.\n",
    "\n",
    "Note that for `latlim` the first value refers to the southern border of your AOI, so this value should always be smaller than the second. For `lonlim`, the first value refers to the western border of your AOI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = r\"/Users/hmcoerver/Local/1_level1\"\n",
    "timelim = [\"2021-08-11\", \"2021-09-21\"]\n",
    "lonlim = [68.3, 69.5]\n",
    "latlim = [25.3, 26.3]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> Which area is covered by these longitude and latitude coordinates?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can run `pywapor.pre_et_look.main()` to start preparing the inputs for the model. This will take a bit of time (around 15min)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds  = pywapor.pre_et_look.main(folder, latlim, lonlim, timelim)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> You can see that the function prints all kind of information. Can you list all the variables that have been downloaded (or \"collected\")?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **PRE_ET_LOOK Variables**\n",
    "> \n",
    "> * NDVI\n",
    "> * etc..\n",
    "> * \n",
    "> * "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you see the output line `< PRE_ET_LOOK`, the code has finished. The function has created a bunch of files in your `project_folder` and returned a variables `ds`. This variable contains a `xarray.Dataset`. XArray is an Python-package that let's you work with large multi-dimensional datasets. We can see the file this dataset is stored in like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fh = ds.encoding[\"source\"]\n",
    "print(fh)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> How large (in MBs) is this file?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the dataset is stored inside a file called `et_look_in.nc`, this file contains all the data we need to run the `et_look` model!\n",
    "\n",
    "We can have a closer look at the contents of the datasets by simply calling `ds` like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of calling `ds` is interactive, you can click on different parts of the output to get more informationa and expand some parts.\n",
    "\n",
    "> **Question**\n",
    "> \n",
    "> * How many dimensions does this dataset have?\n",
    "> \n",
    "> * How many variables does the dataset contain?\n",
    "> \n",
    "> * Does every variable have the same amount of dimensions?\n",
    ">\n",
    "> * How many pixels does the variable `ndvi` contain in total?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like geotiff files, this dataset also contains information like it's coordinate reference system and boundaries. We can access that information like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ds.rio.crs)\n",
    "print(ds.rio.bounds())\n",
    "print(ds.rio.resolution())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> * What is the resolution of this dataset (don't forget the units)?\n",
    "> \n",
    "> * Given the bounds and the resolution, how many pixels does the dataset contain? Does that match with what you found in the previous question?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XArray datasets are also easy to plot, for example, we can create a map of `z` like this (Note that XArray is smart enough to automatically fill in the units and the description of the variable!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.z.plot()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to make a map of a variable with more than 2 dimensions, we'll first have to select for which time (i.e. `time_bins`) we want to create the map. Here I'm selecting the first map, i.e. at index 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.r0.isel(time_bins = 0).plot()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> * Which time does the index 0 refer to?\n",
    "> \n",
    "> * Can you make a plot of the `ndvi` for another date/time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create another plot here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If for some reason you have lost the variable `ds` (maybe you restarted you computer) and don't want to run `pre_et_look` again, you can simply open the file specified by `fh` like this. (The `decode_coords` keyword is used to make sure the CRS information is loaded correctly.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = pywapor.general.processing_functions.open_ds(fh)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running ETLook\n",
    "\n",
    "Now that we have created the input file for the model, we can run the model! We simply pass the `ds` to `pywapor.et_look.main()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_out = pywapor.et_look.main(ds)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This should go pretty quickly. Again, we can see a new netCDF file has been created. Do you remember how to print the path to the file?\n",
    "\n",
    "> **Question**\n",
    "> \n",
    "> Run the command to print the paths to the file.\n",
    "> \n",
    "> How large is this file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the path to the file linked to the new dataset here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the contents of the new dataset just like we did before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_out"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> How many variables are present in the output?\n",
    "> \n",
    "> Can you find the resolution of this output dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write code to find the resolution of the output dataset here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> Can you create a map of the ET?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write some code to create a plot of the ET here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the variables exported in the output dataset, the model calculates many more (intermediate) variables. You can export all variables by specifying a keyword argument called `export_vars` set to `\"all\"` when you can run ET_Look.\n",
    "\n",
    "> **Question**\n",
    "> \n",
    "> Can you run ET_Look and export all variables?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_out_all = pywapor.et_look.main(fh, # ... finish the code to export all variables here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> * How many variables does the output contain now?\n",
    "> \n",
    "> * What is the filesize of the output?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the dataset here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now its your turn\n",
    "\n",
    "> **Question**\n",
    "> \n",
    "> Can you run `pre_et_look` for another region and/or another year ([bboxfinder](http://bboxfinder.com) can help you choose a region)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a project folder, time period and area-of-interest here.\n",
    "folder = r\"\"\n",
    "timelim = [\"\", \"\"]\n",
    "lonlim = []\n",
    "latlim = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run pre_et_look here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run et_look here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> * Does the variable `et_24_mm` contains any holes with missing data?\n",
    "> \n",
    "> * If so, can you figure out what is causing these holes (hint: check the input data)?\n",
    "> \n",
    "> * Do you have any ideas on how to fill these gaps of missing data?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Question**\n",
    "> \n",
    "> Can you think of any ways to validate if our outputs are correct?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **ET Validation methods**\n",
    "> \n",
    "> 1. ...\n",
    "> 2. ...\n",
    "> 3. ...\n",
    "> 4. etc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pywapor",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "30267a9963ef4e891f35edf7c2db672f8b38daceac54cc246d7b09d3d93c63b2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}