notebooks/overscan_subtraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bd8db56e-4d8a-4e43-a127-cb91881f094b",
   "metadata": {},
   "source": [
    "# Overscan Subtraction\n",
    "\n",
    "*J. Runnoe* <br>\n",
    "*October, 2023*\n",
    "\n",
    "This notebook will take you through the steps to overscan and bias subtract your science frames. Many of these tasks are reflected in the [astropy CCD reduction and photometry guide](https://github.com/astropy/ccd-reduction-and-photometry-guide/tree/main), so reading those notebooks can be very helpful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbee14f1-03b2-47e3-bfd4-d6bdcff623f4",
   "metadata": {},
   "source": [
    "---\n",
    "## Contents\n",
    "* [Setup](#setup)\n",
    "    * [Download CCD Photometry Guide](#ccdphot)\n",
    "* [Exercises](#exercises)\n",
    "* [Summary](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "947dac1c-3304-41de-b8a7-03f46a0ba2f7",
   "metadata": {},
   "source": [
    "## Setup <a class=\"anchor\" id=\"setup\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a4a6fbb-d6f6-4abf-a255-533283779384",
   "metadata": {},
   "source": [
    "### Download CCD Photometry Guide <a class=\"anchor\" id=\"ccdphot\"></a>\n",
    "\n",
    "I have mentioned several times that the CCD reduction and photometry guide is a useful tool. If you haven't already, I recommend downloading a local copy from Github so that you can interact with the notebooks and use some of their convenience functions.\n",
    "\n",
    "To do this, start by creating a home for the guide on your local machine. I keep a `Software/` directory on my computer, with subdirectories for code in different languages. Therefore, I can just stick this guide in my existing `py` directory. <br>\n",
    "`$ cd ~/Software/py/` <br>\n",
    "`$ mkdir ccd-reduction-and-photometry-guide/` <br>\n",
    "\n",
    "Then, use `git clone` to copy the guide to your local machine. This makes it easy to download updates and changes anytime `astropy` publishes them. <br>\n",
    "`$ cd ccd-reduction-and-photometry-guide` <br>\n",
    "`$ git clone https://github.com/astropy/ccd-reduction-and-photometry-guide/ .` <br>\n",
    "\n",
    "It may take a minute to donwload everything."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "586d97e8-eacd-4e75-bfa8-cba99ea7fcb0",
   "metadata": {},
   "source": [
    "---\n",
    "## Exercises <a class=\"anchor\" id=\"exercises\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f7b69fa-17dc-442c-8c88-01467ae54cdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import block\n",
    "import numpy as np\n",
    "from astropy.io import fits\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import rc\n",
    "%matplotlib inline\n",
    "from astropy.visualization import hist\n",
    "from ccdproc import ImageFileCollection\n",
    "import ccdproc as ccdp\n",
    "from astropy.modeling import fitting\n",
    "from astropy.modeling.models import Polynomial1D,Chebyshev1D,Legendre1D,Hermite1D\n",
    "from astropy.nddata import CCDData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "254ff05a-c86c-4fc7-9766-2517eb84c514",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import convenience plotting functions downloaded from \n",
    "# here: https://github.com/mwcraig/ccd-reduction-and-photometry-guide\n",
    "phot_tutorial_dir = '/Users/runnojc1/Software/py/ccd-reduction-and-photometry-guide/notebooks/'\n",
    "import sys\n",
    "sys.path.insert(0,phot_tutorial_dir)\n",
    "from convenience_functions import show_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "41e653b6-cc40-412f-a330-aab562940023",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for imaging, I like the photometry notebook plot defaults\n",
    "# so use their custom style for larger fonts and figures\n",
    "plt.style.use('/Users/runnojc1/Software/py/ccd-reduction-and-photometry-guide/notebooks/guide.mplstyle')\n",
    "\n",
    "# set a couple more default parameters for the plots below\n",
    "rc('font', size=20)\n",
    "rc('axes', grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cff2de0-43c4-4bc1-8096-ac9f91b8010c",
   "metadata": {},
   "source": [
    "#### 1. Create a working directory\n",
    "\n",
    "You should preserve a clean copy of the `Imaging/` folder that you downloaded. That is, do not work in that folder or modify those files directly. Instead, you should create a `reduced/` directory to conduct your work and copy over files as necessary there."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b9504ba-197a-4488-a1ea-7bc22668aa90",
   "metadata": {},
   "source": [
    "#### 2. Open a bias frame in Python\n",
    "\n",
    "There are several ways to interact with images, which are outlined in [this notebook](https://github.com/astropy/ccd-reduction-and-photometry-guide/blob/main/notebooks/01-11-reading-images.ipynb). You have seen the `astropy.io.fits` approach before, but this mentions two new options. One is to use the [`CCDData`](https://ccdproc.readthedocs.io/en/latest/ccddata.html) image class, and another (useful when you have many frames) is the [`ImageFileCollection`](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.ImageFileCollection.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dd47c860-bcf9-4f6f-a4db-c9228579a12a",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '/Users/runnojc1/Dropbox/Research/teaching/F2021/ASTR8060/Imaging/'            # raw data directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4f3f8eb-420b-4d8c-b67e-b6612c389bc9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "591c5cf1-bba7-4b5f-98cb-48f59e32fec2",
   "metadata": {},
   "source": [
    "#### 3. Inspect the header\n",
    "If the BIASSEC, TRIMSEC, FILTER, and IMAGETYP header keywords are not present, [add them](https://docs.astropy.org/en/stable/io/fits/usage/headers.html). HERE, BIASSEC refers to the indices needed to select the overscan region (one of them is sufficient), and TRIMSEC refers to the region you want to extract when you trim the overscan off. Then save the file with new header keywords to `reduced/`. Open the file as an image collection and check that the header keywords are there.\n",
    "\n",
    "You can access header keywords with code like this:\n",
    "\n",
    "```python\n",
    "hdu = fits.open('filename')\n",
    "hdr = hdu[extension].header\n",
    "key = hdr['KEYWORD']\n",
    "```\n",
    "\n",
    "Here, `filename` is the file you want to read in, `extension` is a number indicating the extension of the fits file you want to access the header for, and `KEYWORD` is the keyword name you want to extract the header value for. If the keyword isn't present in the header, you'll get an error. You can create new keywords and assign values to them like this:\n",
    "\n",
    "```python\n",
    "hdr['NEW_KEYWORD'] = 5\n",
    "hdr['ANOTHER_KEY'] = '[0:10,20:30]'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e9ac536-de3b-40fe-99b8-2b0f5ca27d6e",
   "metadata": {},
   "source": [
    "A word on indexing. Python starts at 0 and goes [row,col], FITS starts at 1 and goes [col,row]. Also, Python will take a range as [low,high), where high is not included, so the upper limits will be the same for Python and FITS. The `CCDPROC` `subtract_overscan` function uses both of these, depending on how you pass it, so be careful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4101eca-3362-4330-a626-3854fa000e8b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3832562d-de65-4c37-9652-94c4241cf03a",
   "metadata": {},
   "source": [
    "### 4. Model the overscan region\n",
    "\n",
    "We need to perform a bias subtraction on the images. The best way to do this would be to subtract a value from each pixel, but we don't know the correct value. The overscan region doesn't capture all the pixel-to-pixel variations, and the bias frames have noise in them. Our best approach will perform the bias subtraction in two steps. First, we will subtract the overscan region and trim it off to reveal the science section (referred to as the TRIMSEC). Then, we will remove any two dimensional patterns that remain.\n",
    "\n",
    "When we subtract the bias level indicated in the overscan region, we can do a better job and subtract a different value for each row. Figure out waht value to use by fitting a function to the pixel values in the overscan region as a function of row number. \n",
    "\n",
    "Determine whether it is better to use a mean or median to collapse the columns, and what function and order to fit along each row.\n",
    "\n",
    "For model fitting, `astropy` has some handy packages for model fitting. I recommend the [astropy models](https://docs.astropy.org/en/stable/modeling/) for this model fitting and [these examples](https://learn.astropy.org/tutorials/Models-Quick-Fit.html) to learn how to 1D model fitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c5c5960e-8446-4767-ad19-317bd7a6ca89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean or median to combine columns?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4ab9e27c-17bf-458d-a854-b4b0452677c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model overscan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2971f883-9493-4e8e-8625-9c6c2f7f4d7a",
   "metadata": {},
   "source": [
    "#### 5. Bias subtract and trim your image\n",
    "While we can do many data reduction steps by writing our own code, it's not always time efficient to reinvent the wheel. [`CCDPROC`](https://ccdproc.readthedocs.io/en/latest/index.html) is a code with functionality for astronomical CCD data reduction.\n",
    "\n",
    "Use [`ccdproc.subtract_overscan`](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.subtract_overscan.html) and [`ccdproc.trim_image`](https://ccdproc.readthedocs.io/en/latest/api/ccdproc.trim_image.html) with the best parameters from \\#4 to subtract and trim your bias image. Display the raw bias and overscan subtracted and trimmed version in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "760bd691-f615-42ba-a9ec-8143b6989f0b",
   "metadata": {},
   "source": [
    "During the course of data reduction, it is common to give your variables and filenames flags to indicate which corrections have been applied. These are relatively standard:\n",
    "\n",
    "-o overscan subtracted <br>\n",
    "-t trimmed <br>\n",
    "-z bias corrected <br>\n",
    "-f flat field corrected <br>\n",
    "-d dark corrected <br>\n",
    "\n",
    "So for a raw frame `a093.fits`, it would be called `a093_otz.fits` after overscan, trim, and bias correction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7736c52-b881-492a-bf88-196a92e0f147",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a21d120a-2305-49f8-96ee-2992066ac269",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "---\n",
    "## Summary <a class=\"anchor\" id=\"summary\"></a>\n",
    "\n",
    "At this point, all of you should have:\n",
    "* Cloned a local copy of the CCD photomoetry guide.\n",
    "* Modified the header keywords of a bias frame.\n",
    "* Determined whether it is better to mean or median combine your biases.\n",
    "* Modeled the overscan region.\n",
    "* Overscan subtracted and trimmed a bias frame."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:astr8060] *",
   "language": "python",
   "name": "conda-env-astr8060-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}