Adding Cotrend Tutorial

docs/source/tutorials/fitting-CBVs.ipynb

@@ -0,0 +1,277 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Example 5: Fitting CBVs to Remove Long Term Trends"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### What are CBVs\n",
+    "\n",
+    "Cotrending Basis Vectors are used to remove long term trends in light curves. They are built from the most common trends observed in each channel. You can read more about CBVs in [Demystifying Kepler Data](https://arxiv.org/pdf/1207.3093.pdf). They can be used to clean lightcurves of common trends experience by all targets.\n",
+    "\n",
+    "*kepcotrend* can be used to remove CBVs from a light curve. The new light curve is put in a new extention to the fits file. \n",
+    "\n",
+    "Below is an example workflow to remove CBVs from a supernova light curve in *Kepler*. The first stage is to obtain both the light curve and the CBV file for that quarter. (You can download CBVs [here](https://archive.stsci.edu/kepler/cbv.html) for Kepler and [here](https://archive.stsci.edu/ktwo/cbv.html) for K2.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--2017-10-23 17:12:41--  https://archive.stsci.edu/missions/kepler/lightcurves/0084/008462852/kplr008462852-2011073133259_llc.fits\n",
+      "Resolving archive.stsci.edu... 130.167.252.39\n",
+      "Connecting to archive.stsci.edu|130.167.252.39|:443... connected.\n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: 357120 (349K) [text/plain]\n",
+      "Saving to: ‘kplr008462852-2011073133259_llc.fits.1’\n",
+      "\n",
+      "kplr008462852-20110 100%[===================>] 348.75K   472KB/s    in 0.7s    \n",
+      "\n",
+      "2017-10-23 17:12:43 (472 KB/s) - ‘kplr008462852-2011073133259_llc.fits.1’ saved [357120/357120]\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "!wget https://archive.stsci.edu/missions/kepler/lightcurves/0084/008462852/kplr008462852-2011073133259_llc.fits"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--2017-10-23 17:13:03--  https://archive.stsci.edu/missions/kepler/cbv/kplr2011073133259-q08-d25_lcbv.fits\n",
+      "Resolving archive.stsci.edu... 130.167.252.39\n",
+      "Connecting to archive.stsci.edu|130.167.252.39|:443... connected.\n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: 21925440 (21M) [application/fits]\n",
+      "Saving to: ‘kplr2011073133259-q08-d25_lcbv.fits.1’\n",
+      "\n",
+      "kplr2011073133259-q 100%[===================>]  20.91M   297KB/s    in 62s     \n",
+      "\n",
+      "2017-10-23 17:14:06 (343 KB/s) - ‘kplr2011073133259-q08-d25_lcbv.fits.1’ saved [21925440/21925440]\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "!wget https://archive.stsci.edu/missions/kepler/cbv/kplr2011073133259-q08-d25_lcbv.fits"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can plot up the light curve using kepdraw. We've downloaded quarter 8 of KIC 8462852, which undergos large flux dips over time periods of a few days. kepdraw will plot the light curve and save it to *kplr008462852-2011073133259_llc-kepdraw.png*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 68,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found the following flux columns: SAP_FLUX, SAP_FLUX_ERR, PDCSAP_FLUX, PDCSAP_FLUX_ERR\n",
+      "Using data column PDCSAP_FLUX on the plot...\n",
+      "Writing output file kplr008462852-2011073133259_llc-kepdraw.png...\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%bash\n",
+    "kepdraw kplr008462852-2011073133259_llc.fits --non-interactive"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![](kplr008462852-2011073133259_llc-kepdraw.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's use kepcotrend to remove the CBVs from the light curve. It is up to the user to determine how many CBVs should be fit; too few and there will be residual trends, too many and the lightcurve will be overfit, removing astrophysical data. We'll use the first two for this star. \n",
+    "\n",
+    "kepcotrend takes in the lightcurve file name, the CBV file name and a list of CBVs to use. (We are using '1 2'.) The results are two coefficients for each of the CBVs and a png image of the fit."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 105,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Writing output file kplr008462852-2011073133259_llc-kepcotrend.fits...\n",
+      "      -----      \n",
+      "reduced chi2: 51658.12955052195\n",
+      "rms: 3357.82526764298\n",
+      "Coefficient of CBV #1: [ 0.03127181]\n",
+      "Coefficient of CBV #2: [-0.21461379]\n",
+      "      -----      \n"
+     ]
+    }
+   ],
+   "source": [
+    "%%bash\n",
+    "kepcotrend kplr008462852-2011073133259_llc.fits kplr2011073133259-q08-d25_lcbv.fits '1 2' --plot \\\n",
+    "--overwrite --non-interactive"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![](kplr008462852-2011073133259_llc-kepcotrend.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "That seems to have fit very poorly. Unfortunately the CBVs are being fit to the large dip in flux and this is causing a very poor fit. We can mask out this feature using keprange. Using the call below a window will pop up and allow the user to select a time range. keprange will save a text file containing those time stamps.\n",
+    "\n",
+    "You should see an interface pop up that looks like the following\n",
+    "![](keprange.png)\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 72,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2455617.56890755,2455630.0185563867\n",
+      "\n",
+      "Wrote ASCII file kplr008462852-2011073133259_llc-keprange.txt\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%bash\n",
+    "keprange kplr008462852-2011073133259_llc.fits --overwrite"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we can pass kepcotrend the range file and it will fit the CBVs ignoring these data points. It will then apply the correction to all datapoints. This is shown below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 104,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Writing output file out.fits...\n",
+      "      -----      \n",
+      "reduced chi2: 7079.055789270921\n",
+      "rms: 1278.6256990911806\n",
+      "Coefficient of CBV #1: [-0.09308594]\n",
+      "Coefficient of CBV #2: [-0.09540007]\n",
+      "      -----      \n"
+     ]
+    }
+   ],
+   "source": [
+    "%%bash\n",
+    "kepcotrend kplr008462852-2011073133259_llc.fits kplr2011073133259-q08-d25_lcbv.fits '1 2' --plot \\\n",
+    "--overwrite --non-interactive --outfile out.fits --maskfile kplr008462852-2011073133259_llc-keprange.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![](out.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "kepcotrend has made a new fits file and added a new flux column to the first extention called 'CBVSAP_FLUX'. We can plot up this new, CBV corrected flux using kepdraw and check that it has been corrected. This is below and the long term trends have been successfully removed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 74,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Writing output file out-kepdraw.png...\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%bash\n",
+    "kepdraw out.fits --non-interactive --datacol CBVSAP_FLUX"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![](out-kepdraw.png)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}