Jupyter/HIRES_PRV_Service.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Reduction Tutorial\n",
    "\n",
    "<b><i>This service enables reduction and analysis of precision radial velocity (PRV) data from the HIRES Keck instrument. </i></b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i><b><font style=\"color: red;\">This notebook is meant to be a template to set up your own processing using the code snippets provided here. The notebook may not run exactly as presented if you attempt to run the full notebook directly. The code has not been rigorously tested from within a notebook environemnt.</font></b></i>\n",
    "\n",
    "This notebook introduces the Keck HIRES Precision Radial Velocity (PRV) pipeline service and works through one specific example. There are number of variations, mostly having to do with planning processing, which will be covered in more detail by other notebooks but here you will see all the basics.\n",
    "\n",
    "The notebook is kept with the HIRES PRV Python access toolkit in <a href=\"https://github.com/Caltech-IPAC/hiresprv\">GitHub: https://github.com/Caltech-IPAC/hiresprv</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Login\n",
    "\n",
    "Logging in the first time creates a workspace for the user and associates it with a KOA account. This workspace is initialized containing pre-processed products for all HIRESprv compatible data in the archive. Please have a look around and search for existing observations of your target stars before collecting new observations.\n",
    "\n",
    "Users of this service must have <a href=\"https://koa.ipac.caltech.edu\">Keck Observatory Archive (KOA)</a> accounts and use that login here to gain access to their data.  Even researchers planning to use only public data will need a KOA login as this service is maintaining persistent storage under that ID.\n",
    "\n",
    "The login is persisted through the use of HTTP cookies and logging in from multiple clients will connect the user to the same account, storage, and processing history.  This environment (user workspace) is permanent as we expect some on-going research to span years.  The login for a give client machine need only be done once, assuming the cookie file in the local storage is not deleted.  If it is, logging in again will reconstruct it.\n",
    "\n",
    "The cookie file, processing state information, and downloaded results like 1D spectra and RV curve tables will be kept locally in the same space as this notebook.  If you wish to change that, simply add in whatever directory management and navigation you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successful login as bfulton\n"
     ]
    }
   ],
   "source": [
    "from hiresprv.auth import login\n",
    "\n",
    "login('prv.cookies')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KOA Data Retrieval\n",
    "\n",
    "The PRV workspace first needs to be populated with your data from KOA.  This can be done all at once, if the data exists, or incrementally as the data are taken/identified. If the requested night is already contained in the pre-processed dataset or if you have already processed the night it will be skipped over.\n",
    "\n",
    "This step is more than a simple data transfer.  \"Raw reduction\" of the data, which converts the 2D CCD echelle images to 1D spectra, is done up-front as the data are retrieved a night at a time.  The UT dates you give here are actually shifted a few hours to catch any calibration data collected in the afternoon of the same (Hawaii-local) day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"status\": \"ok\",\n",
      "    \"msg\": \"Processing dates in background.\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from hiresprv.archive import Archive\n",
    "\n",
    "koa = Archive('prv.cookies')\n",
    "\n",
    "# These nights are known to work, but were not included in the pre-processed dataset\n",
    "rtn = koa.by_dates(\"\"\"2018-09-02\n",
    "2018-09-03\n",
    "2018-09-04\n",
    "2018-09-05\n",
    "2018-09-22\n",
    "\"\"\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Note that since the data in the workspace are permanent, repeated request for the same data would not change anything and so those dates will be ignored.  Therefore, you can add to the above list or replace it with new dates as you choose.  Dates must be formatted as YYYY-MM-DD.\n",
    "\n",
    "The above service responds immediately with an acknowledgement of the request and starts the actual transfer and raw data reduction (which can take some time) as a background job.  The job status can be checked by polling or can be monitored using the function below.  While one retrieval job (or processing job below) is running, no others can be initiated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PRV Processing Monitor\n",
    "\n",
    "Some steps in the PRV processing can take quite a long time (hours) and we do not want to tie up this notebook page waiting for it to finish.  Below we show how to retrieve a snapshot of the status (and you would have to poll manually to track the progress of the job) but the preferred approach is to start a real-time monitor in a custom page/tab which uses Javascript and an HTTP Event stream.  Run the next cell to generate a link to this monitor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Launch <a href=\"http://hiresprv.ipac.caltech.edu/applications/prvMonitor/monitor.html?workspace=prv_20190926_061945_122750\" target=\"_blank\">real-time monitor</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hiresprv.status import Status\n",
    "from IPython.display import IFrame, HTML, display\n",
    "\n",
    "monitor = Status('prv.cookies')\n",
    "\n",
    "link = monitor.generate_link()\n",
    "\n",
    "HTML(link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metadata\n",
    "\n",
    "Once data have been retrieved and the \"nightly\" raw reduction performed, a set of records is added to a persistent metadata table, one row is added per observation.  These observations are all taken through the HIRES PRV instrument (2D CCD) and will have been reduced to 1D spectra by the raw reduction.  They fall into five classes:\n",
    "\n",
    "<ul>\n",
    "<li><b>RV observations</b> -- Multiple observations of a star with the iodine cell in the light path.  Precision, relative radial velocities are calculated for this type of observation. <p/></li>\n",
    "    \n",
    "<li><b>Templates</b> -- One long observation of the same star without iodine, for reference.<p/></li>\n",
    "    \n",
    "<li><b>B stars</b> -- A set of observations of rapidly rotating B stars bracketing the template observation and used to reduce it.<p/></li>\n",
    "    \n",
    "<li><b>Iodine</b> -- Reference observation of iodine for nightly calibration.<p/></li>\n",
    "    \n",
    "<li>Miscellaneous other calibration observations (labelled as \"<b>Unknown</b>\").<p/></li>\n",
    "</ul>\n",
    "\n",
    "By inspecting this table, the user can determine what objects were observed, whether there are template observations for them (and adequate B star data to reduce a template), and whether there are enough RV measurements to generate a final RV curve.\n",
    "\n",
    "With a small metadata table, this is simple enough to do by inspection but a typical workspace can easily have thousands of files covering tens or hundreds of objects.  Furthermore, since observations for a single object are frequently spread out over years, the metadata table is often fairly thorougly mixed in time.\n",
    "\n",
    "Luckily, there are a number of tools available in client-side Python subset and organize the metadata, so we provide it for download as a CSV table or an SQLite binary file or even, as here, as a simple HTML table.  The workspace copy of the data is maintained in an SQLite database so we also provide a basic filtering mechanism as an optional addition to the download.  This filtering is often adequate for basic processing scenarios.  \n",
    "\n",
    "Note that metadata retrieval can't be done while the system is \"busy\" (downloading additional data or further reducing data in the workspace). Otherwise, metadata downloads can be done at any time.\n",
    "\n",
    "Also note that the client-side file will become out-of-date once new data download or processing requests are submitted.  It is up to the user to re-request the new metadata.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>DEACTIVATED</th>\n",
       "      <th>OBTYPE</th>\n",
       "      <th>FILENAME</th>\n",
       "      <th>TARGET</th>\n",
       "      <th>MJD</th>\n",
       "      <th>BJD</th>\n",
       "      <th>BCVEL</th>\n",
       "      <th>RADVEL</th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "      <th>EPOCH</th>\n",
       "      <th>HRANG</th>\n",
       "      <th>RA_MOTION</th>\n",
       "      <th>DEC_MOTION</th>\n",
       "      <th>PARALLAX</th>\n",
       "      <th>ORIGFILENAME</th>\n",
       "      <th>KOAID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.1</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820001.fits</td>\n",
       "      <td>HI.20091231.06409.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.2</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820002.fits</td>\n",
       "      <td>HI.20091231.06653.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.3</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820003.fits</td>\n",
       "      <td>HI.20091231.07296.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.4</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820004.fits</td>\n",
       "      <td>HI.20091231.07594.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.5</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820005.fits</td>\n",
       "      <td>HI.20091231.07645.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.6</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820006.fits</td>\n",
       "      <td>HI.20091231.07801.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.7</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820007.fits</td>\n",
       "      <td>HI.20091231.07848.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.8</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820008.fits</td>\n",
       "      <td>HI.20091231.07964.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>r20091231.9</td>\n",
       "      <td>th-ar</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820009.fits</td>\n",
       "      <td>HI.20091231.08011.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Iodine calibration</td>\n",
       "      <td>r20091231.10</td>\n",
       "      <td>iodine</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820010.fits</td>\n",
       "      <td>HI.20091231.08160.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>Iodine calibration</td>\n",
       "      <td>r20091231.11</td>\n",
       "      <td>iodine</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820011.fits</td>\n",
       "      <td>HI.20091231.08233.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.66</td>\n",
       "      <td>HD204277</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-20657.870</td>\n",
       "      <td>9320.0</td>\n",
       "      <td>21 27 06.53</td>\n",
       "      <td>+16 07 25.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>3.250</td>\n",
       "      <td>-0.0783</td>\n",
       "      <td>-0.0962</td>\n",
       "      <td>0.030195</td>\n",
       "      <td>j820066.fits</td>\n",
       "      <td>HI.20091231.15897.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.67</td>\n",
       "      <td>HD210460</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-23885.341</td>\n",
       "      <td>20630.0</td>\n",
       "      <td>22 10 19.12</td>\n",
       "      <td>+19 36 57.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>2.557</td>\n",
       "      <td>0.0917</td>\n",
       "      <td>-0.0935</td>\n",
       "      <td>0.017691</td>\n",
       "      <td>j820067.fits</td>\n",
       "      <td>HI.20091231.15988.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.68</td>\n",
       "      <td>HD201091</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-16012.432</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21 06 53.74</td>\n",
       "      <td>+38 44 29.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>3.646</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820068.fits</td>\n",
       "      <td>HI.20091231.16113.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>20091231</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.69</td>\n",
       "      <td>HD201092</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-16013.712</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21 06 53.74</td>\n",
       "      <td>+38 44 29.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>3.675</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>j820069.fits</td>\n",
       "      <td>HI.20091231.16208.fits</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        DATE  DEACTIVATED              OBTYPE      FILENAME    TARGET  \\\n",
       "0   20091231            0             Unknown   r20091231.1     th-ar   \n",
       "1   20091231            0             Unknown   r20091231.2     th-ar   \n",
       "2   20091231            0             Unknown   r20091231.3     th-ar   \n",
       "3   20091231            0             Unknown   r20091231.4     th-ar   \n",
       "4   20091231            0             Unknown   r20091231.5     th-ar   \n",
       "5   20091231            0             Unknown   r20091231.6     th-ar   \n",
       "6   20091231            0             Unknown   r20091231.7     th-ar   \n",
       "7   20091231            0             Unknown   r20091231.8     th-ar   \n",
       "8   20091231            0             Unknown   r20091231.9     th-ar   \n",
       "9   20091231            0  Iodine calibration  r20091231.10    iodine   \n",
       "10  20091231            0  Iodine calibration  r20091231.11    iodine   \n",
       "11  20091231            0      RV observation  r20091231.66  HD204277   \n",
       "12  20091231            0      RV observation  r20091231.67  HD210460   \n",
       "13  20091231            0      RV observation  r20091231.68  HD201091   \n",
       "14  20091231            0      RV observation  r20091231.69  HD201092   \n",
       "\n",
       "             MJD           BJD      BCVEL   RADVEL           RA           DEC  \\\n",
       "0   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "1   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "2   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "3   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "4   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "5   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "6   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "7   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "8   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "9   2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "10  2.455197e+06  2.455197e+06        NaN      NaN          NaN           NaN   \n",
       "11  2.455197e+06  2.455197e+06 -20657.870   9320.0  21 27 06.53  +16 07 25.00   \n",
       "12  2.455197e+06  2.455197e+06 -23885.341  20630.0  22 10 19.12  +19 36 57.00   \n",
       "13  2.455197e+06  2.455197e+06 -16012.432      NaN  21 06 53.74  +38 44 29.00   \n",
       "14  2.455197e+06  2.455197e+06 -16013.712      NaN  21 06 53.74  +38 44 29.00   \n",
       "\n",
       "     EPOCH  HRANG  RA_MOTION  DEC_MOTION  PARALLAX  ORIGFILENAME  \\\n",
       "0      NaN  0.000        NaN         NaN       NaN  j820001.fits   \n",
       "1      NaN  0.000        NaN         NaN       NaN  j820002.fits   \n",
       "2      NaN  0.000        NaN         NaN       NaN  j820003.fits   \n",
       "3      NaN  0.000        NaN         NaN       NaN  j820004.fits   \n",
       "4      NaN  0.000        NaN         NaN       NaN  j820005.fits   \n",
       "5      NaN  0.000        NaN         NaN       NaN  j820006.fits   \n",
       "6      NaN  0.000        NaN         NaN       NaN  j820007.fits   \n",
       "7      NaN  0.000        NaN         NaN       NaN  j820008.fits   \n",
       "8      NaN  0.000        NaN         NaN       NaN  j820009.fits   \n",
       "9      NaN  0.000        NaN         NaN       NaN  j820010.fits   \n",
       "10     NaN  0.000        NaN         NaN       NaN  j820011.fits   \n",
       "11  2015.5  3.250    -0.0783     -0.0962  0.030195  j820066.fits   \n",
       "12  2015.5  2.557     0.0917     -0.0935  0.017691  j820067.fits   \n",
       "13  2015.5  3.646        NaN         NaN       NaN  j820068.fits   \n",
       "14  2015.5  3.675        NaN         NaN       NaN  j820069.fits   \n",
       "\n",
       "                     KOAID  \n",
       "0   HI.20091231.06409.fits  \n",
       "1   HI.20091231.06653.fits  \n",
       "2   HI.20091231.07296.fits  \n",
       "3   HI.20091231.07594.fits  \n",
       "4   HI.20091231.07645.fits  \n",
       "5   HI.20091231.07801.fits  \n",
       "6   HI.20091231.07848.fits  \n",
       "7   HI.20091231.07964.fits  \n",
       "8   HI.20091231.08011.fits  \n",
       "9   HI.20091231.08160.fits  \n",
       "10  HI.20091231.08233.fits  \n",
       "11  HI.20091231.15897.fits  \n",
       "12  HI.20091231.15988.fits  \n",
       "13  HI.20091231.16113.fits  \n",
       "14  HI.20091231.16208.fits  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from hiresprv.database import Database\n",
    "import pandas as pd\n",
    "\n",
    "state = Database('prv.cookies')\n",
    "\n",
    "# We limit to 100 entries so that the query returns quickly in this tutorial\n",
    "search_string = \"select * from FILES LIMIT 100;\"\n",
    "url = state.search(sql=search_string)\n",
    "\n",
    "df = pd.read_html(url, header=0)[0]\n",
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reducing RV Measurements for a Star\n",
    "\n",
    "### Subsetting the Metadata: Single Target\n",
    "\n",
    "Ultimately, to make an RV curve for one star we need to reduce its observations into RV measurements.  Assuming there are adequate B star observations to reduce the template, we can isolate appropriate records in the metadata above by simply filtering on TARGET name.  There are many ways to do this; in our case we we used the remote SQLite query capability and filtered it with <p/>\n",
    "\n",
    "<tt>select DATE, OBTYPE, FILENAME, TARGET, BJD, BCVEL from FILES where TARGET like 'HD185144';</tt> \n",
    "\n",
    "The resulting records are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>OBTYPE</th>\n",
       "      <th>FILENAME</th>\n",
       "      <th>TARGET</th>\n",
       "      <th>BJD</th>\n",
       "      <th>BCVEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20091231</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.72</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4620.095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20091231</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.73</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4620.211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20091231</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20091231.74</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4620.321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.79</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4621.229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.80</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4621.324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.81</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4621.420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.82</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4621.511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.83</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "      <td>-4621.597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>20150606</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20150606.145</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.457180e+06</td>\n",
       "      <td>3189.795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>20150606</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20150606.146</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.457180e+06</td>\n",
       "      <td>3189.328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>20150606</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20150606.147</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.457180e+06</td>\n",
       "      <td>3188.863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>20080516</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20080516.177</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.454603e+06</td>\n",
       "      <td>2061.135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>20080517</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20080517.181</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.454604e+06</td>\n",
       "      <td>2124.858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>20080617</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20080617.132</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.454635e+06</td>\n",
       "      <td>3884.757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>20080617</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20080617.133</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.454635e+06</td>\n",
       "      <td>3884.242</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        DATE          OBTYPE       FILENAME    TARGET           BJD     BCVEL\n",
       "0   20091231  RV observation   r20091231.72  HD185144  2.455197e+06 -4620.095\n",
       "1   20091231  RV observation   r20091231.73  HD185144  2.455197e+06 -4620.211\n",
       "2   20091231  RV observation   r20091231.74  HD185144  2.455197e+06 -4620.321\n",
       "3   20091231        Template   r20091231.79  HD185144  2.455197e+06 -4621.229\n",
       "4   20091231        Template   r20091231.80  HD185144  2.455197e+06 -4621.324\n",
       "5   20091231        Template   r20091231.81  HD185144  2.455197e+06 -4621.420\n",
       "6   20091231        Template   r20091231.82  HD185144  2.455197e+06 -4621.511\n",
       "7   20091231        Template   r20091231.83  HD185144  2.455197e+06 -4621.597\n",
       "8   20150606  RV observation  r20150606.145  HD185144  2.457180e+06  3189.795\n",
       "9   20150606  RV observation  r20150606.146  HD185144  2.457180e+06  3189.328\n",
       "10  20150606  RV observation  r20150606.147  HD185144  2.457180e+06  3188.863\n",
       "11  20080516  RV observation  r20080516.177  HD185144  2.454603e+06  2061.135\n",
       "12  20080517  RV observation  r20080517.181  HD185144  2.454604e+06  2124.858\n",
       "13  20080617  RV observation  r20080617.132  HD185144  2.454635e+06  3884.757\n",
       "14  20080617  RV observation  r20080617.133  HD185144  2.454635e+06  3884.242"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "search_string = \"select DATE,OBTYPE,FILENAME,TARGET,BJD,BCVEL from FILES where TARGET like 'HD185144';\"\n",
    "\n",
    "url = state.search(sql=search_string)\n",
    "df = pd.read_html(url, header=0)[0]\n",
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Templates and B-stars\n",
    "\n",
    "Another subset that comes up is matching B-Star observations with the template observations they will be used with.  This can be many to many so the easiest quick look is just to list out all B-star and Template observations in time order and then match visually:<p/>\n",
    "\n",
    "<tt>select DATE, OBTYPE, FILENAME, TARGET, BJD from FILES \n",
    "    where OBTYPE like 'TEMPLATE' or OBTYPE like 'B Star';</tt> \n",
    "\n",
    "The resulting records are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>OBTYPE</th>\n",
       "      <th>FILENAME</th>\n",
       "      <th>TARGET</th>\n",
       "      <th>BJD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.75</td>\n",
       "      <td>HR9071</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.76</td>\n",
       "      <td>HR9071</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.77</td>\n",
       "      <td>HR9071</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.78</td>\n",
       "      <td>HR9071</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.79</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.80</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.81</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.82</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.83</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.84</td>\n",
       "      <td>HR8047</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.85</td>\n",
       "      <td>HR8047</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.86</td>\n",
       "      <td>HR8047</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>20091231</td>\n",
       "      <td>B star</td>\n",
       "      <td>r20091231.87</td>\n",
       "      <td>HR8047</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.88</td>\n",
       "      <td>HD216520</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>20091231</td>\n",
       "      <td>Template</td>\n",
       "      <td>r20091231.89</td>\n",
       "      <td>HD216520</td>\n",
       "      <td>2.455197e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        DATE    OBTYPE      FILENAME    TARGET           BJD\n",
       "0   20091231    B star  r20091231.75    HR9071  2.455197e+06\n",
       "1   20091231    B star  r20091231.76    HR9071  2.455197e+06\n",
       "2   20091231    B star  r20091231.77    HR9071  2.455197e+06\n",
       "3   20091231    B star  r20091231.78    HR9071  2.455197e+06\n",
       "4   20091231  Template  r20091231.79  HD185144  2.455197e+06\n",
       "5   20091231  Template  r20091231.80  HD185144  2.455197e+06\n",
       "6   20091231  Template  r20091231.81  HD185144  2.455197e+06\n",
       "7   20091231  Template  r20091231.82  HD185144  2.455197e+06\n",
       "8   20091231  Template  r20091231.83  HD185144  2.455197e+06\n",
       "9   20091231    B star  r20091231.84    HR8047  2.455197e+06\n",
       "10  20091231    B star  r20091231.85    HR8047  2.455197e+06\n",
       "11  20091231    B star  r20091231.86    HR8047  2.455197e+06\n",
       "12  20091231    B star  r20091231.87    HR8047  2.455197e+06\n",
       "13  20091231  Template  r20091231.88  HD216520  2.455197e+06\n",
       "14  20091231  Template  r20091231.89  HD216520  2.455197e+06"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_string = \"select DATE, OBTYPE, FILENAME, TARGET, BJD from FILES where OBTYPE like 'TEMPLATE' or OBTYPE like 'B Star';\"\n",
    "\n",
    "url = state.search(sql=search_string)\n",
    "\n",
    "df = pd.read_html(url, header=0)[0]\n",
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RV Pipeline Processing\n",
    "\n",
    "This shows that on 12/31/2009 three separate RV observations were made of HD185144 followed by five template observations (which the pipeline will combine into a single template).  Five years later, another three RV observations were made.\n",
    "\n",
    "As with the data download, the further reduction steps in the pipeline can be quite lengthy (minutes to hours each), so rather than have the user monitor each one, we provide a scripting mechanism so complex reduction jobs can be submitted in on shot.\n",
    "\n",
    "In order to turn any of the RV observations into an RV value, we need the template.  So we will generate that first.  Since it is possible to repeat the template observations on more than one day, we need to explicitly state which object and which day.  The script command for this is:\n",
    "\n",
    "<pre>template 185144 20091231</pre>\n",
    "\n",
    "To reduce an RV measurement, we have to refer to this template (the target name is enough) and specify which file to reduce.  For example:\n",
    "\n",
    "<pre>rv 185144 r20091231.7</pre>\n",
    "\n",
    "Finally, once we have a set of RV measurements for an object, we a generate an RV curve (the pipeline finds all the appropriate RV measurements):\n",
    "\n",
    "<pre>rvcurve 185144</pre>\n",
    "\n",
    "As long as we follow the general rules that we need a template before we can reduce an RV measurement and we need at least three RV measurements before we can generate an RV curve, we can otherwise scripte things in whatever order we wish (<i>e.g.</i> all the templates first).\n",
    "\n",
    "All of this is submitted to the pipeline as a text script, but first lets figure out which observations need to be processed.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>DEACTIVATED</th>\n",
       "      <th>OBTYPE</th>\n",
       "      <th>FILENAME</th>\n",
       "      <th>TARGET</th>\n",
       "      <th>MJD</th>\n",
       "      <th>BJD</th>\n",
       "      <th>BCVEL</th>\n",
       "      <th>RADVEL</th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "      <th>EPOCH</th>\n",
       "      <th>HRANG</th>\n",
       "      <th>RA_MOTION</th>\n",
       "      <th>DEC_MOTION</th>\n",
       "      <th>PARALLAX</th>\n",
       "      <th>ORIGFILENAME</th>\n",
       "      <th>KOAID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20180922</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20180922.60</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2405.133</td>\n",
       "      <td>26580.0</td>\n",
       "      <td>19 32 23.37</td>\n",
       "      <td>+69 39 13.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>-1.023</td>\n",
       "      <td>0.5979</td>\n",
       "      <td>-1.7383</td>\n",
       "      <td>0.17324</td>\n",
       "      <td>j3040160.fits</td>\n",
       "      <td>HI.20180922.17329.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20180922</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20180922.61</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2404.570</td>\n",
       "      <td>26580.0</td>\n",
       "      <td>19 32 23.37</td>\n",
       "      <td>+69 39 13.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>-1.009</td>\n",
       "      <td>0.5979</td>\n",
       "      <td>-1.7383</td>\n",
       "      <td>0.17324</td>\n",
       "      <td>j3040161.fits</td>\n",
       "      <td>HI.20180922.17378.fits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20180922</td>\n",
       "      <td>0</td>\n",
       "      <td>RV observation</td>\n",
       "      <td>r20180922.62</td>\n",
       "      <td>HD185144</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2.458384e+06</td>\n",
       "      <td>2404.007</td>\n",
       "      <td>26580.0</td>\n",
       "      <td>19 32 23.37</td>\n",
       "      <td>+69 39 13.00</td>\n",
       "      <td>2015.5</td>\n",
       "      <td>-0.996</td>\n",
       "      <td>0.5979</td>\n",
       "      <td>-1.7383</td>\n",
       "      <td>0.17324</td>\n",
       "      <td>j3040162.fits</td>\n",
       "      <td>HI.20180922.17427.fits</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DATE  DEACTIVATED          OBTYPE      FILENAME    TARGET  \\\n",
       "0  20180922            0  RV observation  r20180922.60  HD185144   \n",
       "1  20180922            0  RV observation  r20180922.61  HD185144   \n",
       "2  20180922            0  RV observation  r20180922.62  HD185144   \n",
       "\n",
       "            MJD           BJD     BCVEL   RADVEL           RA           DEC  \\\n",
       "0  2.458384e+06  2.458384e+06  2405.133  26580.0  19 32 23.37  +69 39 13.00   \n",
       "1  2.458384e+06  2.458384e+06  2404.570  26580.0  19 32 23.37  +69 39 13.00   \n",
       "2  2.458384e+06  2.458384e+06  2404.007  26580.0  19 32 23.37  +69 39 13.00   \n",
       "\n",
       "    EPOCH  HRANG  RA_MOTION  DEC_MOTION  PARALLAX   ORIGFILENAME  \\\n",
       "0  2015.5 -1.023     0.5979     -1.7383   0.17324  j3040160.fits   \n",
       "1  2015.5 -1.009     0.5979     -1.7383   0.17324  j3040161.fits   \n",
       "2  2015.5 -0.996     0.5979     -1.7383   0.17324  j3040162.fits   \n",
       "\n",
       "                    KOAID  \n",
       "0  HI.20180922.17329.fits  \n",
       "1  HI.20180922.17378.fits  \n",
       "2  HI.20180922.17427.fits  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_string = \"select * from FILES where TARGET like 'HD185144' and DATE = '20180922';\"\n",
    "\n",
    "url = state.search(sql=search_string)\n",
    "df = pd.read_html(url, header=0)[0]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "status= ok\n",
      "msg= Script running in background. Consult monitor for status.\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from hiresprv.idldriver import Idldriver\n",
    "\n",
    "idl = Idldriver('prv.cookies')\n",
    "\n",
    "# This example would create the template, reduce the RV observations, and then construct the\n",
    "# RV time series\n",
    "full_example = \"\"\"\n",
    "template 185144 20091231\n",
    "rv 185144 r20091231.72\n",
    "rv 185144 r20091231.73\n",
    "rv 185144 r20091231.74\n",
    "rv 185144 r20150606.145\n",
    "rv 185144 r20150606.146\n",
    "rv 185144 r20150606.147\n",
    "rvcurve 185144\n",
    "\"\"\"\n",
    "\n",
    "# Since the template for HD185144 has already been created in the pre-processed dataset\n",
    "# we can start by calculating the RVs only for the new observations.\n",
    "new_rvs = \"\"\"\n",
    "rv 185144 r20180922.60\n",
    "rv 185144 r20180922.61\n",
    "rv 185144 r20180922.62\n",
    "\"\"\"\n",
    "\n",
    "rtn = idl.run_script(new_rvs)\n",
    "\n",
    "print(rtn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring (again)\n",
    "\n",
    "To monitor the pipeline processing request, the best idea is to use the same monitor page from above.  It stops whenever a given script is finished but you can restart it any time to see the currently-running job.  You can also insert a monitor start-up or status polling call here as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Product Retrieval\n",
    "\n",
    "There is a utility function for retrieving the RV curves (CSV files) for each target\n",
    "(similarly, there is a function -- data.spectrum -- for retrieving the 1D FITS spectrum files)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BJD_TDB,RV,RV_ERR,BC,ADU,CHI2\n",
      "15196.69208800001,-2.860674413602231,0.789710,-4620.095214843750,52362,1.05080\n",
      "15196.69270200003,1.730315543645411,0.812607,-4620.210937500000,51591,1.04891\n",
      "15196.69329199987,1.431450932849171,0.803164,-4620.320800781250,48950,1.05804\n",
      "17180.10972899990,-2.845576383390323,0.890091,3189.794921875000,55029,1.10542\n",
      "17180.11030799989,1.235438116773508,0.825978,3189.327880859375,55717,1.10771\n",
      "17180.11088699987,1.047083913731319,0.828699,3188.863037109375,48769,1.10120\n"
     ]
    }
   ],
   "source": [
    "from hiresprv.download import Download\n",
    "\n",
    "# second argument is path where files will be downloaded into\n",
    "data = Download('prv.cookies', './')\n",
    "\n",
    "rtn = data.rvcurve('185144')\n",
    "\n",
    "with open('vst185144.csv', 'r') as file:\n",
    "  for line in file:\n",
    "    print(line, end='')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hiresprv",
   "language": "python",
   "name": "hiresprv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}