notebooks/data_management.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bd8db56e-4d8a-4e43-a127-cb91881f094b",
   "metadata": {},
   "source": [
    "# Data Management\n",
    "\n",
    "*J. Runnoe* <br>\n",
    "*October, 2023*\n",
    "\n",
    "In the course of your data reduction, it will be useful to have a coding skillset for repeating tasks and working with the imaging data set. This notebook will introduce some helpful coding skills in this context."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbee14f1-03b2-47e3-bfd4-d6bdcff623f4",
   "metadata": {},
   "source": [
    "---\n",
    "## Contents\n",
    "* [Exercises](#exercises)\n",
    "* [Summary](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "586d97e8-eacd-4e75-bfa8-cba99ea7fcb0",
   "metadata": {},
   "source": [
    "---\n",
    "## Exercises <a class=\"anchor\" id=\"exercises\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f7b69fa-17dc-442c-8c88-01467ae54cdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import block\n",
    "import numpy as np\n",
    "from astropy.io import fits\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import rc\n",
    "import matplotlib as mpl\n",
    "%matplotlib inline\n",
    "from astropy.visualization import hist\n",
    "from ccdproc import ImageFileCollection\n",
    "import ccdproc as ccdp\n",
    "from astropy.modeling import fitting\n",
    "from astropy.modeling.models import Polynomial1D,Chebyshev1D,Legendre1D,Hermite1D\n",
    "from astropy.nddata import CCDData\n",
    "import glob\n",
    "from datetime import datetime\n",
    "from matplotlib.ticker import (MultipleLocator, AutoMinorLocator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "254ff05a-c86c-4fc7-9766-2517eb84c514",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import convenience plotting functions downloaded from \n",
    "# here: https://github.com/mwcraig/ccd-reduction-and-photometry-guide\n",
    "phot_tutorial_dir = '/Users/runnojc1/Software/py/ccd-reduction-and-photometry-guide/notebooks/'\n",
    "import sys\n",
    "sys.path.insert(0,phot_tutorial_dir)\n",
    "from convenience_functions import show_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "41e653b6-cc40-412f-a330-aab562940023",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for imaging, I like the photometry notebook plot defaults\n",
    "# so use their custom style for larger fonts and figures\n",
    "plt.style.use('/Users/runnojc1/Software/py/ccd-reduction-and-photometry-guide/notebooks/guide.mplstyle')\n",
    "\n",
    "# set a couple more default parameters for the plots below\n",
    "rc('font', size=20)\n",
    "rc('axes', grid=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "26513e4a-9e41-49c8-b644-b918e751b098",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '/Users/runnojc1/Dropbox/Research/teaching/F2021/ASTR8060/Imaging/'            # raw data directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cff2de0-43c4-4bc1-8096-ac9f91b8010c",
   "metadata": {},
   "source": [
    "#### 1. Collecting filenames\n",
    "\n",
    "Create a list of your bias filenames in three different ways:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2bc2dae-5637-4b30-92a5-fa45b44b80d2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "##### Using list comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "967f77ea-8aab-4d39-af37-5f56f4abb4fc",
   "metadata": {},
   "source": [
    "List comprehension is a \"Pythonic\" way of writing a for loop. It is often a little bit faster than writing a loop, and it allows you to fill your arrays in one line instead of having to declare your empty variables and then fill them in the loop. The syntax might look like this:\n",
    "\n",
    "```python\n",
    "telescope_names = ['Gemini','Keck','Magellan','WIYN']\n",
    "instrument_names = ['GMOS','KCIW','MAGE','ODI-N']\n",
    "\n",
    "# normal loop\n",
    "names = []\n",
    "for scope,instr in zip(telescope_names,instrument_names):\n",
    "    names.append(scope+'/'+instr)\n",
    "\n",
    "# list comprehension\n",
    "names = [scope+'/'+instr for scope,instr in zip(telescope_names,instrument_names)]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5b049029-768f-407b-b5ea-0c9d05f0cad4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. list comprehension\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf40df7-44e3-4aae-ab9c-defee0fdc61e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "##### Using [`glob`](https://docs.python.org/3/library/glob.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8a535a-9568-40c4-8107-cc8cb60cbf15",
   "metadata": {},
   "source": [
    "`glob` is a Python package that is good for listing the contents of a directory if you know the general pattern of filename you'd like to find (`listdir` is a good option if you'd just like to list everything in a directory). You might use it like this:\n",
    "\n",
    "```python\n",
    "import glob\n",
    "img_files = glob.glob(data_dir+'a[0-1]*.fits')\n",
    "```\n",
    "\n",
    "This will return all the raw data files with numbers starting in 0 or 1 (i.e. everything with a number <200). You may find that you need to sort them.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d4ec333f-f9b1-4a9f-ab97-62e9c4cd0f40",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. glob\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "154b16ba-c211-48d9-9e8d-3094716eb7e6",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "##### Using an [`ImageFileCollection`](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.ImageFileCollection.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02bb89da-30a6-416b-a55c-f2c3391ac653",
   "metadata": {},
   "source": [
    "An `ImageFileCollection` is a class in the `ccdproc` package that provides a useful way to represent many image files and their parameters. [Notebook [01-11]](https://github.com/astropy/ccd-reduction-and-photometry-guide/blob/main/notebooks/01-11-reading-images.ipynb) has some useful tutorials for reading images, including examples with `ImageFileCollections`.\n",
    "\n",
    "```python\n",
    "from ccdproc import ImageFileCollection\n",
    "imgs = ImageFileCollection(data_dir,glob_include='a???.fits',glob_exclude='*ot*.fits') # exclude otz files in case of rerun\n",
    "mask = imgs.summary['exptime'] == 30. # find all 30s exposures\n",
    "files_t30 = np.array(imgs.files_filtered(include_path=True))[mask]\n",
    "```\n",
    "\n",
    "This code will read all the files matching the pattern \"a???.fits\" (except those with the \"ot\" flags) into an `ImageFileCollection`. It will create a mask for those with 30s exposures and return their filenames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cdc2520d-5d2f-4fa0-82f6-f584e00886a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. ImageFilecollection\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f4fa123-0ad2-4187-af5f-704cae6747e1",
   "metadata": {},
   "source": [
    "#### 2. Extracting information from fits headers\n",
    "\n",
    "A useful exercise is to be able to extract e.g., the time an exposure was taken from its header. Try to extract the time stamp for each flat frame and plot median pixel value in the image versus time.\n",
    "\n",
    "It may be useful to express the times using the [`datetime`](https://docs.python.org/3/library/datetime.html) module. It is also helpful to know that you can get Python to mark just the time when plotting `datetime` objects with the following:\n",
    "\n",
    "```python\n",
    "fig,ax = plt.subplots(figsize=(8,6))\n",
    "ax.plot(x,y)\n",
    "ax.set_ylabel('Value')\n",
    "ax.set_xlabel('Time')\n",
    "ax.xaxis.set_major_formatter(mpl.dates.DateFormatter('%H:%M'))\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "Finally, note that when reading in many header data units and headers from fits files, you can use up a lot of your computer's available memory. Eventually, it may complain. In that case, you can delete variables you no longer need:\n",
    "\n",
    "```python\n",
    "hdu = fits.open('filename')\n",
    "del hdu\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c3c1e6c1-f7fc-407f-aff4-3897496ec28d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f836da06-700c-4c20-8027-0864c69edb2b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49e575e1-15c5-46c8-a7db-0cb5a16a6a6c",
   "metadata": {},
   "source": [
    "---\n",
    "## Summary <a class=\"anchor\" id=\"summary\"></a>\n",
    "\n",
    "At this point, all of you should have:\n",
    "* Practiced multiple ways of pulling up filenames.\n",
    "* Learned to extract and plot times from the fits headers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:astr8060] *",
   "language": "python",
   "name": "conda-env-astr8060-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}