python-data/exercises/ex07_netcdf4_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4fc1b0c3-f3ec-4724-b24c-0e42bfdb2cb4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Exercise 7: NetCDF4 Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ac81b88-7771-4404-89cd-d9ec233651d7",
   "metadata": {},
   "source": [
    "## Aim: Introduce the netCDF4 library in Python to read and create NetCDF4 Files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6457c36c-b7ba-44a7-861e-0ae678d5412c",
   "metadata": {},
   "source": [
    "Find the teaching material here: https://unidata.github.io/netcdf4-python/"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5512bea4-24d3-415b-8572-d770111ba0b6",
   "metadata": {},
   "source": [
    "### Issues covered:\n",
    "- Importing netCDF4\n",
    "- Groups, dimensions, variables and attributes\n",
    "- Writing data and retrieving it from variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81916b14-ec1c-4e8b-af3f-93d00377e9ff",
   "metadata": {},
   "source": [
    "## Creating/opening/closing netCDF files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a47cb092-895b-411d-8ac2-2b7df0d1138d",
   "metadata": {},
   "source": [
    "Q1.\n",
    "- Import the `netCDF4` library\n",
    "- Let's create a new NetCDF file called `test.nc` in appending mode (`a`) with the `NETCDF4` format. This mode will allow us to edit the dataset later. Save this to a variable called `new_file`.\n",
    "- Inspect the new file to see what its `data_model` is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "261f60af-c7ec-4cb3-8859-5d173a534a1d",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:38.682331Z",
     "iopub.status.busy": "2024-11-08T14:55:38.681747Z",
     "iopub.status.idle": "2024-11-08T14:55:39.246105Z",
     "shell.execute_reply": "2024-11-08T14:55:39.245107Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1c4e8491-204a-4bcf-9696-3535a37c7b8d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Groups, dimensions, variables and attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4649d782-c96b-4bc5-840b-231345ed4c79",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ecfa40-dcd9-4728-aba7-57fef3dd089e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Q2. Groups act as containers for variables, dimensions and attributes.\n",
    "- Add a new group to the dataset we just made called \"forecasts\".\n",
    "- Create a new group within forecasts called `model1`.\n",
    "- List the groups of your dataset using `.groups`\n",
    "- What happens if you do `group3 = new_file.createGroup(\"/analyses/model2\")`?\n",
    "- What happens if you do `group4 = new_file.createGroup(\"analyses\")`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6c3c7a16-5d5f-4f01-b83e-a20fe1c96392",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.250340Z",
     "iopub.status.busy": "2024-11-08T14:55:39.249830Z",
     "iopub.status.idle": "2024-11-08T14:55:39.257372Z",
     "shell.execute_reply": "2024-11-08T14:55:39.256627Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b898110e-14e8-464d-9a4f-bd19a273c7cb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdf87db-5455-45a2-ae47-22febfa0a20e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Q3.\n",
    "- Create some dimensions for the `new_file` dataset:\n",
    "    - `time` dimension with unlimited size\n",
    "    - `level` dimension with unlimited size\n",
    "    - `lat` dimension with unlimited size\n",
    "    - `lon` dimension with unlimited size\n",
    "- Print out the dimensions you just created.\n",
    "- Check the length of the latitude dimension to make sure it is 0.\n",
    "- Check that the level dimension is unlimited.\n",
    "- Let's take a look at an overview using \n",
    "```\n",
    "for dim in new_file.dimensions.values():\n",
    "    print(dim)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "037ca268-58fa-4657-a527-112d08cc16b8",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.260641Z",
     "iopub.status.busy": "2024-11-08T14:55:39.260294Z",
     "iopub.status.idle": "2024-11-08T14:55:39.269039Z",
     "shell.execute_reply": "2024-11-08T14:55:39.268334Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "56511ca9-39a1-466a-9516-d801ab53407f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e7d217-ea7c-4ea2-af69-942239aebb29",
   "metadata": {},
   "source": [
    "Remember that the data types are as follows:\n",
    "- `f4`: 32-bit floting point \n",
    "- `f8`: 64-bit floating point \n",
    "- `i4`: 32-bit signed integer \n",
    "- `i2`: 16-bit signed integer\n",
    "- `i8`: 64-bit unsigned integer\n",
    "- `i1`: 8-bit signed integer\n",
    "- `u1`: 8-bit unsigned integer\n",
    "- `u2`: 16-bit unsigned integer\n",
    "- `u4`: 32-bit unsigned integer\n",
    "- `u8`: 64-bit unsigned integer\n",
    "- `S1`: single-character string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7176b7-79b9-4e38-99f5-5491d91de3a2",
   "metadata": {},
   "source": [
    "Q4.\n",
    "- Create a scalar variable called `times` with the type set to `f8`.\n",
    "- Create a scalar variable called `levels` but this time set the type to `np.float64`. (You'll need to import numpy as np)\n",
    "- Print out the variables using `new_file.variables`. What do you notice about the types?\n",
    "- Create a variable in the `model2` group we made earlier called `temp`, with the `float64` type and this time give it dimensions: (`time`, `level`, `lat`, `lon`). Print it out.\n",
    "- Create two values: \n",
    "    - `longitudes` with the name `lon`, type `float64` and dimension `lon`\n",
    "    - `latitudes` with the name `lat`, type `float64` and dimension `lat`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "292ca4fe-3681-4b5e-8b2e-c681a3aa9249",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.272376Z",
     "iopub.status.busy": "2024-11-08T14:55:39.271846Z",
     "iopub.status.idle": "2024-11-08T14:55:39.282042Z",
     "shell.execute_reply": "2024-11-08T14:55:39.281363Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cf76552d-79d8-41c4-95fc-4616c02248b3",
   "metadata": {},
   "source": [
    "### Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be859b4-16a3-48d0-9242-b8c590d8508a",
   "metadata": {},
   "source": [
    "Q5.\n",
    "- Let's create a global attribute. Create an attribute on the `new_file` object called `.description` with the value `This is a test description.`.\n",
    "- Let's create a variable attribute. Create an attribute on the `times` variable called `units` and put `hours`.\n",
    "- Take a look at the attrs on `new_file` using `new_file.ncattrs()`. What does this show?\n",
    "- To get the name AND description, use the following loop:\n",
    "```\n",
    "for name in new_file.ncattrs():\n",
    "    print(name, \":\", getattr(new_file, name))\n",
    "```\n",
    "- There is an easier way of doing this - using `new_file.__dict__`. Try it out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5ae8dcf6-0184-4f83-b75c-9684a852d140",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.284955Z",
     "iopub.status.busy": "2024-11-08T14:55:39.284674Z",
     "iopub.status.idle": "2024-11-08T14:55:39.295226Z",
     "shell.execute_reply": "2024-11-08T14:55:39.294641Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "df7ec8fe-81b3-4768-9f4b-c643d4a8c254",
   "metadata": {},
   "source": [
    "## Writing data to and receiving data from netCDF variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d636cdaa-c646-406a-96e9-85810cea39ec",
   "metadata": {},
   "source": [
    "Q6. \n",
    "- Create an array to populate a new variable `lats` with using `lats = np.arange(-100, 100, 2)` and an array to populate the `lons` variable with using `lons = np.arange(-200, 200, 2)`.\n",
    "- Print out the `latitudes` and `longitudes` variables we created earlier to see what they look like before we populate them.\n",
    "- Populate the two variables with our data using `latitudes[:] = lats` and the same for longitudes.\n",
    "- Print the data out and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f14444cf-ff2b-4b90-b4fd-3af73ad640b4",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.297995Z",
     "iopub.status.busy": "2024-11-08T14:55:39.297706Z",
     "iopub.status.idle": "2024-11-08T14:55:39.310135Z",
     "shell.execute_reply": "2024-11-08T14:55:39.309547Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ce5d22bf-d201-424e-babc-11d9d431f834",
   "metadata": {},
   "source": [
    "Q7.\n",
    "- Extend `new_file` to have the dimension `pressure` with size 10.\n",
    "- Define a 4D variable `temperature` with dimensions (time, pressure, latitude, longitude). Print the shape of the temperature variable to look at the size before populating with data.\n",
    "- Generate random temperature data for a subset of time and pressure values - start by creating `nlats` and `nlons` equal to the length of the `lat` and `lon` dimensions. Assign random data to `temperature[0:10, 0:3, :, :]` using `np.random.uniform(size=(10,3, nlats, nlons))`.\n",
    "- After assigning the data, print the shape of the `temperature` variable. Take a look at the size of it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f4c4384f-a966-4615-a2da-77168568ae77",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.312687Z",
     "iopub.status.busy": "2024-11-08T14:55:39.312397Z",
     "iopub.status.idle": "2024-11-08T14:55:39.741786Z",
     "shell.execute_reply": "2024-11-08T14:55:39.741066Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "12c17e3e-ad67-4c29-a426-e6390626ce25",
   "metadata": {},
   "source": [
    "Q8. \n",
    "- Define the `pressure` variable with type `f4` and the `pressure` dimension.\n",
    "- Populate the `pressure` variable with the values [1000, 850, 700, 500, 300, 250, 200, 150, 100, 50].\n",
    "- Extract the tempearture variable using `temperature = new_file.variables[\"temperature\"]`, the latitudes using `latitudes = new_file.variables[\"lat\"][:]` and the longitudes using `longitudes = new_file.variables[\"lon\"][:]`.\n",
    "- Use fancy indexing to slice the temperature variable: select times 0, 2 and 4. Index the 2nd, 4th and 7th values of the pressures and select only positive latitudes and longitudes.\n",
    "- Print the shape of the resulting subset array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "769c08e7-6db7-4112-b49e-8003787b6882",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:55:39.745009Z",
     "iopub.status.busy": "2024-11-08T14:55:39.744663Z",
     "iopub.status.idle": "2024-11-08T14:55:39.882617Z",
     "shell.execute_reply": "2024-11-08T14:55:39.881968Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 + Jaspy",
   "language": "python",
   "name": "jaspy"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}