2 Basic xarray - subsetting data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Subsetting a dataset by time and space (Slicing and Dicing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xarray creates labelled coordinate indexes for CF compliant data. This makes selecting subsets of the data in time and space straightforward.\n",
    "\n",
    "This tutorial covers some of the common usage patterns for subsetting data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload the library and the dataset from the previous notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xarray.open_dataset('http://dapds00.nci.org.au/thredds/dodsC/rr3/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historical/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_ACCESS1-3_historical_r1i1p1_185001-200512.nc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select just the data variable `tas` and save a refernce to it in another python variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tas = ds.tas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872, lat: 145, lon: 192)>\n",
       "[52116480 values with dtype=float32]\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xarray builds on top of numpy, and stores its data internally as numpy arrays. It supports many numpy operations, so it is possible to find out the shape of the underlying data, and use numpy style indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1872, 145, 192)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 145, lon: 192)>\n",
       "array([[240.83618, 240.83618, 240.83618, ..., 240.83191, 240.83191, 240.83191],\n",
       "       [242.67894, 242.65828, 242.63867, ..., 242.72214, 242.71056, 242.69565],\n",
       "       [243.58124, 243.52393, 243.46175, ..., 243.73363, 243.69022, 243.6365 ],\n",
       "       ...,\n",
       "       [239.76013, 239.84523, 239.93364, ..., 239.52486, 239.60957, 239.68538],\n",
       "       [239.02599, 239.03561, 239.04205, ..., 239.04428, 239.03633, 239.02672],\n",
       "       [238.45506, 238.45506, 238.45506, ..., 238.45506, 238.45506, 238.45506]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas[0,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By selecting just the first time index it has created a DataArray with no time dimension, but time is still a coordinate not associated with any variable, as indicated by no longer having `*` beside it. It now only has one value: the value of the first time index. The index selection above is equivalent to using `isel` like so"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 145, lon: 192)>\n",
       "array([[240.83618, 240.83618, 240.83618, ..., 240.83191, 240.83191, 240.83191],\n",
       "       [242.67894, 242.65828, 242.63867, ..., 242.72214, 242.71056, 242.69565],\n",
       "       [243.58124, 243.52393, 243.46175, ..., 243.73363, 243.69022, 243.6365 ],\n",
       "       ...,\n",
       "       [239.76013, 239.84523, 239.93364, ..., 239.52486, 239.60957, 239.68538],\n",
       "       [239.02599, 239.03561, 239.04205, ..., 239.04428, 239.03633, 239.02672],\n",
       "       [238.45506, 238.45506, 238.45506, ..., 238.45506, 238.45506, 238.45506]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.isel(time=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way is more compact, and one more descriptive, but they have the same result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The power of xarray comes with the close association of data with coordinates. So it is possible to use the equivalent `.sel` operator but with coordinate values. For example, to select an area that includes the Indian Ocean and Australia use `slice` to indicate the range of latitude and longitude values required and pass as key/value pairs to `sel`.  `slice` will include coordinate values less than **or equal** to the upper bound, not like `range` in basic python that excludes the upper bound"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872, lat: 85, lon: 75)>\n",
       "[11934000 values with dtype=float32]\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(lon=slice(20,160),lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operators can be chained, so multiple operations can be peformed sequentially. For example, to select the above area and the first time index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 85, lon: 75)>\n",
       "array([[244.32095, 244.06064, 243.80359, ..., 251.85716, 253.08392, 254.23578],\n",
       "       [243.11086, 242.9945 , 242.96965, ..., 250.56412, 252.03651, 253.29895],\n",
       "       [242.59998, 242.75227, 242.95288, ..., 251.33804, 252.93666, 254.82977],\n",
       "       ...,\n",
       "       [280.82947, 281.203  , 281.8401 , ..., 298.99326, 299.03836, 299.07532],\n",
       "       [280.9224 , 281.0952 , 281.43793, ..., 297.9115 , 298.03323, 298.08383],\n",
       "       [280.7901 , 281.48508, 281.8507 , ..., 296.3739 , 296.66852, 296.84888]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.isel(time=0).sel(lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case it is convenient to use `isel` to select the time, rather than specifying a date, but it is also possible to specify the date explicitly using `sel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 85, lon: 75)>\n",
       "array([[244.32095, 244.06064, 243.80359, ..., 251.85716, 253.08392, 254.23578],\n",
       "       [243.11086, 242.9945 , 242.96965, ..., 250.56412, 252.03651, 253.29895],\n",
       "       [242.59998, 242.75227, 242.95288, ..., 251.33804, 252.93666, 254.82977],\n",
       "       ...,\n",
       "       [280.82947, 281.203  , 281.8401 , ..., 298.99326, 299.03836, 299.07532],\n",
       "       [280.9224 , 281.0952 , 281.43793, ..., 297.9115 , 298.03323, 298.08383],\n",
       "       [280.7901 , 281.48508, 281.8507 , ..., 296.3739 , 296.66852, 296.84888]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(time='1850-01-16T12:00:00', lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to use `slice` for the `time` dimension. To select Mar to November of 1871:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 9, lat: 85, lon: 75)>\n",
       "array([[[232.72298, 232.41539, ..., 240.3316 , 241.3103 ],\n",
       "        [230.3484 , 230.09096, ..., 241.09224, 242.43002],\n",
       "        ...,\n",
       "        [287.8961 , 287.78564, ..., 296.20505, 296.41022],\n",
       "        [288.0455 , 288.2771 , ..., 294.80002, 294.9939 ]],\n",
       "\n",
       "       [[230.42459, 229.92857, ..., 240.06776, 241.25368],\n",
       "        [227.20078, 226.75165, ..., 239.50015, 241.08636],\n",
       "        ...,\n",
       "        [293.6292 , 293.43594, ..., 297.73218, 297.78775],\n",
       "        [293.78354, 294.1199 , ..., 296.54602, 296.6775 ]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[234.84286, 234.5873 , ..., 242.15875, 243.17744],\n",
       "        [233.36253, 233.07495, ..., 242.26463, 243.54987],\n",
       "        ...,\n",
       "        [290.4821 , 290.44135, ..., 300.78342, 300.67822],\n",
       "        [290.74014, 291.0542 , ..., 300.5218 , 300.38864]],\n",
       "\n",
       "       [[237.83063, 237.62698, ..., 248.45111, 249.83937],\n",
       "        [236.2484 , 235.9952 , ..., 247.79037, 249.23076],\n",
       "        ...,\n",
       "        [286.57666, 286.7646 , ..., 299.87238, 299.7646 ],\n",
       "        [286.386  , 287.12302, ..., 299.55652, 299.45605]]], dtype=float32)\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1871-03-16T12:00:00 1871-04-16 ... 1871-11-16\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(time=slice('1871-03','1871-11'), lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `slice` operator selects values between an upper and lower bound. If a single coordinate value is required when using `sel` it must either correspond to an *exact* value in the coordinate array, or the `method` argument specified to tell xarray how to choose a value. For example, to select out just values in the cell closest to Brisbane"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872)>\n",
       "array([296.4649 , 296.54944, 296.43167, ..., 293.6983 , 294.73325, 296.40576],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "    lat      float64 -27.5\n",
       "    lon      float64 153.8\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(lat=-27.47, lon=153.03, method='nearest')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the closest location in the data was at `lat=-27.5`, `lon=153.8`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:analysis3-19.07]",
   "language": "python",
   "name": "conda-env-analysis3-19.07-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}