{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "---\n",
    "title: \"Plotting and Pandas\"\n",
    "teaching: 3000\n",
    "exercises: 0\n",
    "\n",
    "questions:\n",
    "\n",
    "- \"How do we make scatter plots in Matplotlib? How do we store data in a Pandas `DataFrame`?\"\n",
    "\n",
    "objectives:\n",
    "\n",
    "- \"Select rows and columns from an Astropy `Table`.\"\n",
    "\n",
    "- \"Use Matplotlib to make a scatter plot.\"\n",
    "\n",
    "- \"Use Gala to transform coordinates.\"\n",
    "\n",
    "- \"Make a Pandas `DataFrame` and use a Boolean `Series` to select rows.\"\n",
    "\n",
    "- \"Save a `DataFrame` in an HDF5 file.\"\n",
    "\n",
    "keypoints:\n",
    "\n",
    "- \"When you make a scatter plot, adjust the size of the markers and their transparency so the figure is not overplotted; otherwise it can misrepresent the data badly.\"\n",
    "\n",
    "- \"For simple scatter plots in Matplotlib, `plot` is faster than `scatter`.\"\n",
    "\n",
    "- \"An Astropy `Table` and a Pandas `DataFrame` are similar in many ways and they provide many of the same functions.  They have pros and cons, but for many projects, either one would be a reasonable choice.\"\n",
    "\n",
    "---\n",
    "\n",
    "{% include links.md %}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Proper Motion\n",
    "\n",
    "This is the third in a series of notebooks related to astronomy data.\n",
    "\n",
    "As a running example, we are replicating parts of the analysis in a recent paper, \"[Off the beaten path: Gaia reveals GD-1 stars outside of the main stream](https://arxiv.org/abs/1805.00425)\" by Adrian M. Price-Whelan and Ana Bonaca.\n",
    "\n",
    "In the first lesson, we wrote ADQL queries and used them to select and download data from the Gaia server.\n",
    "\n",
    "In the second lesson, we wrote a query to select stars from the region of the sky where we expect GD-1 to be, and saved the results in a FITS file.\n",
    "\n",
    "Now we'll read that data back and implement the next step in the analysis, identifying stars with the proper motion we expect for GD-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "Here are the steps in this lesson:\n",
    "\n",
    "1. We'll read back the results from the previous lesson, which we saved in a FITS file.\n",
    "\n",
    "2. Then we'll transform the coordinates and proper motion data from ICRS back to the coordinate frame of GD-1.\n",
    "\n",
    "3. We'll put those results into a Pandas `DataFrame`, which we'll use to select stars near the centerline of GD-1.\n",
    "\n",
    "4. Plotting the proper motion of those stars, we'll identify a region of proper motion for stars that are likely to be in GD-1.\n",
    "\n",
    "5. Finally, we'll select and plot the stars whose proper motion is in that region.\n",
    "\n",
    "After completing this lesson, you should be able to\n",
    "\n",
    "* Select rows and columns from an Astropy `Table`.\n",
    "\n",
    "* Use Matplotlib to make a scatter plot.\n",
    "\n",
    "* Use Gala to transform coordinates.\n",
    "\n",
    "* Make a Pandas `DataFrame` and use a Boolean `Series` to select rows.\n",
    "\n",
    "* Save a `DataFrame` in an HDF5 file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "## Installing libraries\n",
    "\n",
    "If you are running this notebook on Colab, you can run the following cell to install Astroquery and the other libraries we'll use.\n",
    "\n",
    "If you are running this notebook on your own computer, you might have to install these libraries yourself.  See the instructions in the preface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# If we're running on Colab, install libraries\n",
    "\n",
    "import sys\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if IN_COLAB:\n",
    "    !pip install astroquery astro-gala wget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reload the data\n",
    "\n",
    "In the previous lesson, we ran a query on the Gaia server and downloaded data for roughly 100,000 stars.  We saved the data in a FITS file so that now, picking up where we left off, we can read the data from a local file rather than running the query again.\n",
    "\n",
    "If you ran the previous lesson successfully, you should already have a file called `gd1_results.fits` that contains the data we downloaded.\n",
    "\n",
    "If not, you can run the following cell, which downloads the data from our repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from wget import download\n",
    "\n",
    "filename = 'gd1_results.fits'\n",
    "path = 'https://github.com/AllenDowney/AstronomicalData/raw/main/data/'\n",
    "\n",
    "if not os.path.exists(filename):\n",
    "    print(download(path+filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now here's how we can read the data from the file back into an Astropy `Table`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from astropy.table import Table\n",
    "\n",
    "results = Table.read(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is an Astropy `Table`.\n",
    "\n",
    "We can use `info` to refresh our memory of the contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table length=140340>\n",
       "      name       dtype    unit                              description                            \n",
       "--------------- ------- -------- ------------------------------------------------------------------\n",
       "      source_id   int64          Unique source identifier (unique within a particular Data Release)\n",
       "             ra float64      deg                                                    Right ascension\n",
       "            dec float64      deg                                                        Declination\n",
       "           pmra float64 mas / yr                         Proper motion in right ascension direction\n",
       "          pmdec float64 mas / yr                             Proper motion in declination direction\n",
       "       parallax float64      mas                                                           Parallax\n",
       " parallax_error float64      mas                                         Standard error of parallax\n",
       "radial_velocity float64   km / s                                                    Radial velocity"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting rows and columns\n",
    "\n",
    "In this section we'll see operations for selecting columns and rows from an Astropy `Table`.  You can find more information about these operations in the [Astropy documentation](https://docs.astropy.org/en/stable/table/access_table.html).\n",
    "\n",
    "We can get the names of the columns like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['source_id',\n",
       " 'ra',\n",
       " 'dec',\n",
       " 'pmra',\n",
       " 'pmdec',\n",
       " 'parallax',\n",
       " 'parallax_error',\n",
       " 'radial_velocity']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.colnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And select an individual column like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "&lt;Column name=&apos;ra&apos; dtype=&apos;float64&apos; unit=&apos;deg&apos; description=&apos;Right ascension&apos; length=140340&gt;\n",
       "<table>\n",
       "<tr><td>142.48301935991023</td></tr>\n",
       "<tr><td>142.25452941346344</td></tr>\n",
       "<tr><td>142.64528557468074</td></tr>\n",
       "<tr><td>142.57739430926034</td></tr>\n",
       "<tr><td>142.58913564478618</td></tr>\n",
       "<tr><td>141.81762228999614</td></tr>\n",
       "<tr><td>143.18339801317677</td></tr>\n",
       "<tr><td>142.9347319464589</td></tr>\n",
       "<tr><td>142.26769745823267</td></tr>\n",
       "<tr><td>142.89551292869012</td></tr>\n",
       "<tr><td>142.2780935768316</td></tr>\n",
       "<tr><td>142.06138786534987</td></tr>\n",
       "<tr><td>...</td></tr>\n",
       "<tr><td>143.05456487172972</td></tr>\n",
       "<tr><td>144.0436496516182</td></tr>\n",
       "<tr><td>144.06566578919313</td></tr>\n",
       "<tr><td>144.13177563215973</td></tr>\n",
       "<tr><td>143.77696341662764</td></tr>\n",
       "<tr><td>142.945956347594</td></tr>\n",
       "<tr><td>142.97282480557786</td></tr>\n",
       "<tr><td>143.4166017695258</td></tr>\n",
       "<tr><td>143.64484588686904</td></tr>\n",
       "<tr><td>143.41554585481808</td></tr>\n",
       "<tr><td>143.6908739159247</td></tr>\n",
       "<tr><td>143.7702681295401</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Column name='ra' dtype='float64' unit='deg' description='Right ascension' length=140340>\n",
       "142.48301935991023\n",
       "142.25452941346344\n",
       "142.64528557468074\n",
       "142.57739430926034\n",
       "142.58913564478618\n",
       "141.81762228999614\n",
       "143.18339801317677\n",
       " 142.9347319464589\n",
       "142.26769745823267\n",
       "142.89551292869012\n",
       " 142.2780935768316\n",
       "142.06138786534987\n",
       "               ...\n",
       "143.05456487172972\n",
       " 144.0436496516182\n",
       "144.06566578919313\n",
       "144.13177563215973\n",
       "143.77696341662764\n",
       "  142.945956347594\n",
       "142.97282480557786\n",
       " 143.4166017695258\n",
       "143.64484588686904\n",
       "143.41554585481808\n",
       " 143.6908739159247\n",
       " 143.7702681295401"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results['ra']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a `Column` object that contains the data, and also the data type, units, and name of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.table.column.Column"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(results['ra'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rows in the `Table` are numbered from 0 to `n-1`, where `n` is the number of rows.  We can select the first row like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<i>Row index=0</i>\n",
       "<table id=\"table140569154410096\">\n",
       "<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>parallax</th><th>parallax_error</th><th>radial_velocity</th></tr></thead>\n",
       "<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th>mas</th><th>mas</th><th>km / s</th></tr></thead>\n",
       "<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>\n",
       "<tr><td>637987125186749568</td><td>142.48301935991023</td><td>21.75771616932985</td><td>-2.5168384683875766</td><td>2.941813096629439</td><td>-0.2573448962333354</td><td>0.823720794509811</td><td>1e+20</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Row index=0>\n",
       "    source_id              ra                dec                pmra              pmdec             parallax        parallax_error  radial_velocity\n",
       "                          deg                deg              mas / yr           mas / yr             mas                mas             km / s    \n",
       "      int64             float64            float64            float64            float64            float64            float64          float64    \n",
       "------------------ ------------------ ----------------- ------------------- ----------------- ------------------- ----------------- ---------------\n",
       "637987125186749568 142.48301935991023 21.75771616932985 -2.5168384683875766 2.941813096629439 -0.2573448962333354 0.823720794509811           1e+20"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might have guessed, the result is a `Row` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.table.row.Row"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(results[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the bracket operator selects both columns and rows.  You might wonder how it knows which to select.\n",
    "\n",
    "If the expression in brackets is a string, it selects a column; if the expression is an integer, it selects a row.\n",
    "\n",
    "If you apply the bracket operator twice, you can select a column and then an element from the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "142.48301935991023"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results['ra'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can select a row and then an element from the row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "142.48301935991023"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0]['ra']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You get the same result either way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scatter plot\n",
    "\n",
    "To see what the results look like, we'll use a scatter plot.  The library we'll use is [Matplotlib](https://matplotlib.org/), which is the most widely-used plotting library for Python.\n",
    "\n",
    "The Matplotlib interface is based on MATLAB (hence the name), so if you know MATLAB, some of it will be familiar.\n",
    "\n",
    "We'll import like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pyplot part of the Matplotlib library.  It is conventional to import it using the shortened name `plt`.\n",
    "\n",
    "Pyplot provides two functions that can make scatterplots, [plt.scatter](https://matplotlib.org/3.3.0/api/_as_gen/matplotlib.pyplot.scatter.html) and [plt.plot](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html).\n",
    "\n",
    "* `scatter` is more versatile; for example, you can make every point in a scatter plot a different color.\n",
    "\n",
    "* `plot` is more limited, but for simple cases, it can be substantially faster.  \n",
    "\n",
    "Jake Vanderplas explains these differences in [The Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/04.02-simple-scatter-plots.html)\n",
    "\n",
    "Since we are plotting more than 100,000 points and they are all the same size and color, we'll use `plot`.\n",
    "\n",
    "Here's a scatter plot with right ascension on the x-axis and declination on the y-axis, both ICRS coordinates in degrees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 13044 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = results['ra']\n",
    "y = results['dec']\n",
    "plt.plot(x, y, 'ko')\n",
    "\n",
    "plt.xlabel('ra (degree ICRS)')\n",
    "plt.ylabel('dec (degree ICRS)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The arguments to `plt.plot` are `x`, `y`, and a string that specifies the style.  In this case, the letters `ko` indicate that we want a black, round marker (`k` is for black because `b` is for blue).\n",
    "\n",
    "The functions `xlabel` and `ylabel` put labels on the axes.\n",
    "\n",
    "This scatter plot has a problem.  It is \"[overplotted](https://python-graph-gallery.com/134-how-to-avoid-overplotting-with-python/)\", which means that there are so many overlapping points, we can't distinguish between high and low density areas.\n",
    "\n",
    "To fix this, we can provide optional arguments to control the size and transparency of the points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "In the call to `plt.plot`, use the keyword argument `markersize` to make the markers smaller.\n",
    "\n",
    "Then add the keyword argument `alpha` to make the markers partly transparent.\n",
    "\n",
    "Adjust these arguments until you think the figure shows the data most clearly.\n",
    "\n",
    "Note: Once you have made these changes, you might notice that the figure shows stripes with lower density of stars.  These stripes are caused by the way Gaia scans the sky, which [you can read about here](https://www.cosmos.esa.int/web/gaia/scanning-law).  The dataset we are using, [Gaia Data Release 2](https://www.cosmos.esa.int/web/gaia/dr2), covers 22 months of observations; during this time, some parts of the sky were scanned more than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "\n",
    "# x = results['ra']\n",
    "# y = results['dec']\n",
    "# plt.plot(x, y, 'ko', markersize=0.1, alpha=0.1)\n",
    "\n",
    "# plt.xlabel('ra (degree ICRS)')\n",
    "# plt.ylabel('dec (degree ICRS)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform back\n",
    "\n",
    "Remember that we selected data from a rectangle of coordinates in the `GD1Koposov10` frame, then transformed them to ICRS when we constructed the query.\n",
    "The coordinates in `results` are in ICRS.\n",
    "\n",
    "To plot them, we will transform them back to the `GD1Koposov10` frame; that way, the axes of the figure are aligned with the orbit of GD-1, which is useful for two reasons:\n",
    "\n",
    "* We can identify stars that are likely to be in GD-1 by selecting stars near the centerline of the stream, where $\\phi_2$ is close to 0.\n",
    "\n",
    "* We expect stars in GD-1 to have similar proper motion along the $\\phi_1$ axis.\n",
    "\n",
    "To do the transformation, we'll put the results into a `SkyCoord` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from astropy.coordinates import SkyCoord\n",
    "import astropy.units as u\n",
    "\n",
    "skycoord = SkyCoord(\n",
    "               ra=results['ra'], \n",
    "               dec=results['dec'],\n",
    "               pm_ra_cosdec=results['pmra'],\n",
    "               pm_dec=results['pmdec'], \n",
    "               distance=8*u.kpc, \n",
    "               radial_velocity=0*u.km/u.s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the arguments we send to `SkyCoord` come directly from `results`.\n",
    "\n",
    "We provide `distance` and `radial_velocity` to prepare the data for reflex correction, which we explain below.\n",
    "\n",
    "The result is an Astropy `SkyCoord` object, which we can transform to the GD-1 frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.coordinates.sky_coordinate.SkyCoord"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gala.coordinates import GD1Koposov10\n",
    "\n",
    "gd1_frame = GD1Koposov10()\n",
    "transformed = skycoord.transform_to(gd1_frame)\n",
    "type(transformed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is another `SkyCoord` object, now in the `GD1Koposov10` frame.\n",
    "\n",
    "The next step is to correct the proper motion measurements from Gaia for reflex due to the motion of our solar system around the Galactic center.\n",
    "\n",
    "When we created `skycoord`, we provided `distance` and `radial_velocity` as arguments, but we did not use the measurements provided by Gaia.  Instead, we use fixed values for these parameters.\n",
    "\n",
    "That might seem like a strange thing to do, but here's the motivation:\n",
    "\n",
    "* Because the stars in GD-1 are so far away, the distance estimates we get from Gaia, which are based on parallax, are not very precise.  So we replace them with our current best estimate of the mean distance to GD-1, about 8 kpc.  See [Koposov, Rix, and Hogg, 2010](https://ui.adsabs.harvard.edu/abs/2010ApJ...712..260K/abstract).\n",
    "\n",
    "* For the other stars in the table, this distance estimate will be inaccurate, so reflex correction will not be correct.  But that should have only a small effect on our ability to identify stars with the proper motion we expect for GD-1.\n",
    "\n",
    "* The measurement of radial velocity has no effect on the correction for proper motion; the value we provide is arbitrary, but we have to provide a value to avoid errors in the reflex correction calculation.\n",
    "\n",
    "We are grateful to Adrian Price-Whelen for his help explaining this step in the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this preparation, we can use `reflex_correct` from Gala ([documentation here](https://gala-astro.readthedocs.io/en/latest/api/gala.coordinates.reflex_correct.html)) to correct for solar reflex motion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.coordinates.sky_coordinate.SkyCoord"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gala.coordinates import reflex_correct\n",
    "\n",
    "gd1_coord = reflex_correct(transformed)\n",
    "type(gd1_coord)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a `SkyCoord` object that contains \n",
    "\n",
    "* `phi1` and `phi2`, which represent the transformed coordinates in the `GD1Koposov10` frame.\n",
    "\n",
    "* `pm_phi1_cosphi2` and `pm_phi2`, which represent the transformed and corrected proper motions.\n",
    "\n",
    "We can select the coordinates like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "phi1 = gd1_coord.phi1\n",
    "phi2 = gd1_coord.phi2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And plot them like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 153808 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(phi1, phi2, 'ko', markersize=0.1, alpha=0.2)\n",
    "\n",
    "plt.xlabel('ra (degree GD1)')\n",
    "plt.ylabel('dec (degree GD1)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we started with a rectangle in GD-1 coordinates.  When transformed to ICRS, it's a non-rectangular polygon.  Now that we have transformed back to GD-1 coordinates, it's a rectangle again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas DataFrame\n",
    "\n",
    "At this point we have two objects containing different subsets of the data.  `results` is the Astropy `Table` we downloaded from Gaia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.table.table.Table"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `gd1_coord` is a `SkyCoord` object that contains the transformed coordinates and proper motions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.coordinates.sky_coordinate.SkyCoord"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(gd1_coord)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On one hand, this division of labor makes sense because each object provides different capabilities.  But working with multiple object types can be awkward.\n",
    "\n",
    "It will be more convenient to choose one object and get all of the data into it.  We'll use a Pandas DataFrame, for two reasons:\n",
    "\n",
    "1. It provides capabilities that are pretty much a superset of the other data structures, so it's the all-in-one solution.\n",
    "\n",
    "2. Pandas is a general-purpose tool that is useful in many domains, especially data science.  If you are going to develop expertise in one tool, Pandas is a good choice.\n",
    "\n",
    "However, compared to an Astropy `Table`, Pandas has one big drawback: it does not keep the metadata associated with the table, including the units for the columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's easy to convert a `Table` to a Pandas `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140340, 8)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "results_df = results.to_pandas()\n",
    "results_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataFrame` provides `shape`, which shows the number of rows and columns.\n",
    "\n",
    "It also provides `head`, which displays the first few rows.  It is useful for spot-checking large results as you go along."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_id</th>\n",
       "      <th>ra</th>\n",
       "      <th>dec</th>\n",
       "      <th>pmra</th>\n",
       "      <th>pmdec</th>\n",
       "      <th>parallax</th>\n",
       "      <th>parallax_error</th>\n",
       "      <th>radial_velocity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>637987125186749568</td>\n",
       "      <td>142.483019</td>\n",
       "      <td>21.757716</td>\n",
       "      <td>-2.516838</td>\n",
       "      <td>2.941813</td>\n",
       "      <td>-0.257345</td>\n",
       "      <td>0.823721</td>\n",
       "      <td>1.000000e+20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>638285195917112960</td>\n",
       "      <td>142.254529</td>\n",
       "      <td>22.476168</td>\n",
       "      <td>2.662702</td>\n",
       "      <td>-12.165984</td>\n",
       "      <td>0.422728</td>\n",
       "      <td>0.297472</td>\n",
       "      <td>1.000000e+20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>638073505568978688</td>\n",
       "      <td>142.645286</td>\n",
       "      <td>22.166932</td>\n",
       "      <td>18.306747</td>\n",
       "      <td>-7.950660</td>\n",
       "      <td>0.103640</td>\n",
       "      <td>0.544584</td>\n",
       "      <td>1.000000e+20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>638086386175786752</td>\n",
       "      <td>142.577394</td>\n",
       "      <td>22.227920</td>\n",
       "      <td>0.987786</td>\n",
       "      <td>-2.584105</td>\n",
       "      <td>-0.857327</td>\n",
       "      <td>1.059607</td>\n",
       "      <td>1.000000e+20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>638049655615392384</td>\n",
       "      <td>142.589136</td>\n",
       "      <td>22.110783</td>\n",
       "      <td>0.244439</td>\n",
       "      <td>-4.941079</td>\n",
       "      <td>0.099625</td>\n",
       "      <td>0.486224</td>\n",
       "      <td>1.000000e+20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            source_id          ra        dec       pmra      pmdec  parallax  \\\n",
       "0  637987125186749568  142.483019  21.757716  -2.516838   2.941813 -0.257345   \n",
       "1  638285195917112960  142.254529  22.476168   2.662702 -12.165984  0.422728   \n",
       "2  638073505568978688  142.645286  22.166932  18.306747  -7.950660  0.103640   \n",
       "3  638086386175786752  142.577394  22.227920   0.987786  -2.584105 -0.857327   \n",
       "4  638049655615392384  142.589136  22.110783   0.244439  -4.941079  0.099625   \n",
       "\n",
       "   parallax_error  radial_velocity  \n",
       "0        0.823721     1.000000e+20  \n",
       "1        0.297472     1.000000e+20  \n",
       "2        0.544584     1.000000e+20  \n",
       "3        1.059607     1.000000e+20  \n",
       "4        0.486224     1.000000e+20  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python detail: `shape` is an attribute, so we display its value without calling it as a function; `head` is a function, so we need the parentheses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can extract the columns we want from `gd1_coord` and add them as columns in the `DataFrame`.  `phi1` and `phi2` contain the transformed coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140340, 10)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['phi1'] = gd1_coord.phi1\n",
    "results_df['phi2'] = gd1_coord.phi2\n",
    "results_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pm_phi1_cosphi2` and `pm_phi2` contain the components of proper motion in the transformed frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140340, 12)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['pm_phi1'] = gd1_coord.pm_phi1_cosphi2\n",
    "results_df['pm_phi2'] = gd1_coord.pm_phi2\n",
    "results_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Detail:** If you notice that `SkyCoord` has an attribute called `proper_motion`, you might wonder why we are not using it.\n",
    "\n",
    "We could have: `proper_motion` contains the same data as `pm_phi1_cosphi2` and `pm_phi2`, but in a different format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring data\n",
    "\n",
    "One benefit of using Pandas is that it provides functions for exploring the data and checking for problems.\n",
    "\n",
    "One of the most useful of these functions is `describe`, which computes summary statistics for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_id</th>\n",
       "      <th>ra</th>\n",
       "      <th>dec</th>\n",
       "      <th>pmra</th>\n",
       "      <th>pmdec</th>\n",
       "      <th>parallax</th>\n",
       "      <th>parallax_error</th>\n",
       "      <th>radial_velocity</th>\n",
       "      <th>phi1</th>\n",
       "      <th>phi2</th>\n",
       "      <th>pm_phi1</th>\n",
       "      <th>pm_phi2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.403400e+05</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>1.403400e+05</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "      <td>140340.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>6.792378e+17</td>\n",
       "      <td>143.822971</td>\n",
       "      <td>26.780161</td>\n",
       "      <td>-2.484410</td>\n",
       "      <td>-6.100784</td>\n",
       "      <td>0.179474</td>\n",
       "      <td>0.518068</td>\n",
       "      <td>9.931167e+19</td>\n",
       "      <td>-50.091337</td>\n",
       "      <td>-1.803264</td>\n",
       "      <td>-0.868980</td>\n",
       "      <td>1.409215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.792015e+16</td>\n",
       "      <td>3.697824</td>\n",
       "      <td>3.052639</td>\n",
       "      <td>5.913923</td>\n",
       "      <td>7.202013</td>\n",
       "      <td>0.759622</td>\n",
       "      <td>0.505558</td>\n",
       "      <td>8.267982e+18</td>\n",
       "      <td>2.892321</td>\n",
       "      <td>3.444439</td>\n",
       "      <td>6.657700</td>\n",
       "      <td>6.518573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>6.214900e+17</td>\n",
       "      <td>135.425699</td>\n",
       "      <td>19.286617</td>\n",
       "      <td>-106.755260</td>\n",
       "      <td>-138.065163</td>\n",
       "      <td>-15.287602</td>\n",
       "      <td>0.020824</td>\n",
       "      <td>-1.792684e+02</td>\n",
       "      <td>-54.999989</td>\n",
       "      <td>-8.029159</td>\n",
       "      <td>-115.275637</td>\n",
       "      <td>-161.150142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>6.443515e+17</td>\n",
       "      <td>140.967807</td>\n",
       "      <td>24.592348</td>\n",
       "      <td>-5.038746</td>\n",
       "      <td>-8.341641</td>\n",
       "      <td>-0.035983</td>\n",
       "      <td>0.141108</td>\n",
       "      <td>1.000000e+20</td>\n",
       "      <td>-52.603097</td>\n",
       "      <td>-4.750410</td>\n",
       "      <td>-2.948851</td>\n",
       "      <td>-1.107074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6.888056e+17</td>\n",
       "      <td>143.734183</td>\n",
       "      <td>26.746169</td>\n",
       "      <td>-1.834971</td>\n",
       "      <td>-4.689570</td>\n",
       "      <td>0.362705</td>\n",
       "      <td>0.336103</td>\n",
       "      <td>1.000000e+20</td>\n",
       "      <td>-50.147567</td>\n",
       "      <td>-1.671497</td>\n",
       "      <td>0.585038</td>\n",
       "      <td>1.987196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.976578e+17</td>\n",
       "      <td>146.607180</td>\n",
       "      <td>28.990490</td>\n",
       "      <td>0.452995</td>\n",
       "      <td>-1.937833</td>\n",
       "      <td>0.657636</td>\n",
       "      <td>0.751171</td>\n",
       "      <td>1.000000e+20</td>\n",
       "      <td>-47.593466</td>\n",
       "      <td>1.160632</td>\n",
       "      <td>3.001761</td>\n",
       "      <td>4.628859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.974418e+17</td>\n",
       "      <td>152.777393</td>\n",
       "      <td>34.285481</td>\n",
       "      <td>104.319923</td>\n",
       "      <td>20.981070</td>\n",
       "      <td>0.999957</td>\n",
       "      <td>4.171221</td>\n",
       "      <td>1.000000e+20</td>\n",
       "      <td>-45.000086</td>\n",
       "      <td>4.014794</td>\n",
       "      <td>39.802471</td>\n",
       "      <td>79.275199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          source_id             ra            dec           pmra  \\\n",
       "count  1.403400e+05  140340.000000  140340.000000  140340.000000   \n",
       "mean   6.792378e+17     143.822971      26.780161      -2.484410   \n",
       "std    3.792015e+16       3.697824       3.052639       5.913923   \n",
       "min    6.214900e+17     135.425699      19.286617    -106.755260   \n",
       "25%    6.443515e+17     140.967807      24.592348      -5.038746   \n",
       "50%    6.888056e+17     143.734183      26.746169      -1.834971   \n",
       "75%    6.976578e+17     146.607180      28.990490       0.452995   \n",
       "max    7.974418e+17     152.777393      34.285481     104.319923   \n",
       "\n",
       "               pmdec       parallax  parallax_error  radial_velocity  \\\n",
       "count  140340.000000  140340.000000   140340.000000     1.403400e+05   \n",
       "mean       -6.100784       0.179474        0.518068     9.931167e+19   \n",
       "std         7.202013       0.759622        0.505558     8.267982e+18   \n",
       "min      -138.065163     -15.287602        0.020824    -1.792684e+02   \n",
       "25%        -8.341641      -0.035983        0.141108     1.000000e+20   \n",
       "50%        -4.689570       0.362705        0.336103     1.000000e+20   \n",
       "75%        -1.937833       0.657636        0.751171     1.000000e+20   \n",
       "max        20.981070       0.999957        4.171221     1.000000e+20   \n",
       "\n",
       "                phi1           phi2        pm_phi1        pm_phi2  \n",
       "count  140340.000000  140340.000000  140340.000000  140340.000000  \n",
       "mean      -50.091337      -1.803264      -0.868980       1.409215  \n",
       "std         2.892321       3.444439       6.657700       6.518573  \n",
       "min       -54.999989      -8.029159    -115.275637    -161.150142  \n",
       "25%       -52.603097      -4.750410      -2.948851      -1.107074  \n",
       "50%       -50.147567      -1.671497       0.585038       1.987196  \n",
       "75%       -47.593466       1.160632       3.001761       4.628859  \n",
       "max       -45.000086       4.014794      39.802471      79.275199  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Review the summary statistics in this table.\n",
    "\n",
    "* Do the values makes senses based on what you know about the context?\n",
    "\n",
    "* Do you see any values that seem problematic, or evidence of other data issues?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "\n",
    "# A few issues that are likely to come up:\n",
    "\n",
    "# 1. Why are some of the parallax values negative?\n",
    "#    Some parallax measurements are inaccurate, especially\n",
    "#    stars that are far away.\n",
    "\n",
    "# 2. Why are some of the radial velocities 1e20?\n",
    "#    It seems like this value is used to indicate invalid data.\n",
    "#    Notice that the 25th percentile is 1e20, which indicates\n",
    "#    that at least 75% of these values are invalid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot proper motion\n",
    "\n",
    "Now we are ready to replicate one of the panels in Figure 1 of the Price-Whelan and Bonaca paper, the one that shows the components of proper motion as a scatter plot:\n",
    "\n",
    "<img width=\"300\" src=\"https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-1.png\">\n",
    "\n",
    "In this figure, the shaded area is a high-density region of stars with the proper motion we expect for stars in GD-1.  \n",
    "\n",
    "* Due to the nature of tidal streams, we expect the proper motion for most stars to be along the axis of the stream; that is, we expect motion in the direction of `phi2` to be near 0.\n",
    "\n",
    "* In the direction of `phi1`, we don't have a prior expectation for proper motion, except that it should form a cluster at a non-zero value. \n",
    "\n",
    "To locate this cluster, we will select stars near the centerline_df of GD-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting the centerline\n",
    "\n",
    "As we can see in the following figure, many stars in GD-1 are less than 1 degree from the line `phi2=0`.\n",
    "\n",
    "<img src=\"https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-4.png\">\n",
    "\n",
    "So stars near this line have the highest probability of being in GD-1.\n",
    "\n",
    "To select them, we will use a \"Boolean mask\".  We'll start by selecting the `phi2` column from the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phi2 = results_df['phi2']\n",
    "type(phi2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a `Series`, which is the structure Pandas uses to represent columns.\n",
    "\n",
    "We can use a comparison operator, `>`, to compare the values in a `Series` to a constant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phi2_min = -1.0 * u.deg\n",
    "phi2_max = 1.0 * u.deg\n",
    "\n",
    "mask = (phi2 > phi2_min)\n",
    "type(mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a `Series` of Boolean values, that is, `True` and `False`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1    False\n",
       "2    False\n",
       "3    False\n",
       "4    False\n",
       "Name: phi2, dtype: bool"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `&` operator computes \"logical AND\", which means the result is true where elements from both Boolean `Series` are true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = (phi2 > phi2_min) & (phi2 < phi2_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python note: We need the parentheses around the conditions; otherwise the order of operations is incorrect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sum of a Boolean `Series` is the number of `True` values, so we can use `sum` to see how many stars are in the selected region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25084"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Boolean `Series` is sometimes called a \"mask\" because we can use it to mask out some of the rows in a `DataFrame` and select the rest, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "centerline_df = results_df[mask]\n",
    "type(centerline_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`centerline_df` is a `DataFrame` that contains only the rows from `results_df` that correspond to `True` values in `mask`; that is, in contains the stars near the centerline of GD-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting proper motion\n",
    "\n",
    "Here's a scatter plot of proper motion for the selected stars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20880 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm1 = centerline_df['pm_phi1']\n",
    "pm2 = centerline_df['pm_phi2']\n",
    "\n",
    "plt.plot(pm1, pm2, 'ko', markersize=0.1, alpha=0.1)\n",
    "    \n",
    "plt.xlabel('Proper motion phi1 (GD1 frame)')\n",
    "plt.ylabel('Proper motion phi2 (GD1 frame)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these results, we see a large cluster around (0, 0), and a smaller cluster near (0, -10).\n",
    "\n",
    "We can use `xlim` and `ylim` to set the limits on the axes and zoom in on the region near (0, 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 108244 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm1 = centerline_df['pm_phi1']\n",
    "pm2 = centerline_df['pm_phi2']\n",
    "\n",
    "plt.plot(pm1, pm2, 'ko', markersize=0.3, alpha=0.3)\n",
    "    \n",
    "plt.xlabel('Proper motion phi1 (GD1 frame)')\n",
    "plt.ylabel('Proper motion phi2 (GD1 frame)')\n",
    "\n",
    "plt.xlim(-12, 8)\n",
    "plt.ylim(-10, 10);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see the smaller cluster more clearly.\n",
    "\n",
    "You might notice that our figure is less dense than the one in the paper.  That's because we started with a set of stars from a relatively small region.  The figure in the paper is based on a region about 10 times bigger.\n",
    "\n",
    "In the next lesson we'll go back and select stars from a larger region.  But first we'll use the proper motion data to identify stars likely to be in GD-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering based on proper motion\n",
    "\n",
    "The next step is to select stars in the \"overdense\" region of proper motion, which are candidates to be in GD-1.\n",
    "\n",
    "In the original paper, Price-Whelan and Bonaca used a polygon to cover this region, as shown in this figure.\n",
    "\n",
    "<img width=\"300\" src=\"https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-1.png\">\n",
    "\n",
    "We'll use a simple rectangle for now, but in a later lesson we'll see how to select a polygonal region as well.\n",
    "\n",
    "Here are bounds on proper motion we chose by eye:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm1_min = -8.9\n",
    "pm1_max = -6.9\n",
    "pm2_min = -2.2\n",
    "pm2_max =  1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To draw these bounds, we'll make two lists containing the coordinates of the corners of the rectangle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_rectangle(x1, x2, y1, y2):\n",
    "    \"\"\"Return the corners of a rectangle.\"\"\"\n",
    "    xs = [x1, x1, x2, x2, x1]\n",
    "    ys = [y1, y2, y2, y1, y1]\n",
    "    return xs, ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm1_rect, pm2_rect = make_rectangle(\n",
    "    pm1_min, pm1_max, pm2_min, pm2_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the plot looks like with the bounds we chose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 108856 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(pm1, pm2, 'ko', markersize=0.3, alpha=0.3)\n",
    "plt.plot(pm1_rect, pm2_rect, '-')\n",
    "    \n",
    "plt.xlabel('Proper motion phi1 (GD1 frame)')\n",
    "plt.ylabel('Proper motion phi2 (GD1 frame)')\n",
    "\n",
    "plt.xlim(-12, 8)\n",
    "plt.ylim(-10, 10);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select rows that fall within these bounds, we'll use the following function, which uses Pandas operators to make a mask that selects rows where `series` falls between `low` and `high`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def between(series, low, high):\n",
    "    \"\"\"Make a Boolean Series.\n",
    "    \n",
    "    series: Pandas Series\n",
    "    low: lower bound\n",
    "    high: upper bound\n",
    "    \n",
    "    returns: Boolean Series\n",
    "    \"\"\"\n",
    "    return (series > low) & (series < high)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following mask select stars with proper motion in the region we chose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm1 = results_df['pm_phi1']\n",
    "pm2 = results_df['pm_phi2']\n",
    "\n",
    "pm_mask = (between(pm1, pm1_min, pm1_max) & \n",
    "           between(pm2, pm2_min, pm2_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the sum of a Boolean series is the number of `True` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1049"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pm_mask.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use this mask to select rows from `results_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1049"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_df = results_df[pm_mask]\n",
    "len(selected_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the stars we think are likely to be in GD-1.  Let's see what they look like, plotting their coordinates (not their proper motion)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20180 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "phi1 = selected_df['phi1']\n",
    "phi2 = selected_df['phi2']\n",
    "\n",
    "plt.plot(phi1, phi2, 'ko', markersize=0.5, alpha=0.5)\n",
    "\n",
    "plt.xlabel(r'$\\phi_1$ (degree GD1)')\n",
    "plt.ylabel(r'$\\phi_2$ (degree GD1)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that's starting to look like a tidal stream!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the DataFrame\n",
    "\n",
    "At this point we have run a successful query and cleaned up the results; this is a good time to save the data.\n",
    "\n",
    "To save a Pandas `DataFrame`, one option is to convert it to an Astropy `Table`, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "astropy.table.table.Table"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_table = Table.from_pandas(selected_df)\n",
    "type(selected_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we could write the `Table` to a FITS file, as we did in the previous lesson.  \n",
    "\n",
    "But Pandas provides functions to write DataFrames in other formats; to see what they are [find the functions here that begin with `to_`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).\n",
    "\n",
    "One of the best options is HDF5, which is Version 5 of [Hierarchical Data Format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format).\n",
    "\n",
    "HDF5 is a binary format, so files are small and fast to read and write (like FITS, but unlike XML).\n",
    "\n",
    "An HDF5 file is similar to an SQL database in the sense that it can contain more than one table, although in HDF5 vocabulary, a table is called a Dataset.  ([Multi-extension FITS files](https://www.stsci.edu/itt/review/dhb_2011/Intro/intro_ch23.html) can also contain more than one table.)\n",
    "\n",
    "And HDF5 stores the metadata associated with the table, including column names, row labels, and data types (like FITS).\n",
    "\n",
    "Finally, HDF5 is a cross-language standard, so if you write an HDF5 file with Pandas, you can read it back with many other software tools (more than FITS)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can write a Pandas `DataFrame` to an HDF5 file like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'gd1_dataframe.hdf5'\n",
    "\n",
    "results_df.to_hdf(filename, 'results_df')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because an HDF5 file can contain more than one Dataset, we have to provide a name, or \"key\", that identifies the Dataset in the file.\n",
    "\n",
    "We could use any string as the key, but in this example I use the variable name `results_df`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise \n",
    "\n",
    "We're going to need `centerline_df` and `selected_df` later as well.  Write a line or two of code to add it as a second Dataset in the HDF5 file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "\n",
    "centerline_df.to_hdf(filename, 'centerline_df')\n",
    "selected_df.to_hdf(filename, 'selected_df')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Detail:** Reading and writing HDF5 tables requires a library called `PyTables` that is not always installed with Pandas.  You can install it with pip like this:\n",
    "\n",
    "```\n",
    "pip install tables\n",
    "```\n",
    "\n",
    "If you install it using Conda, the name of the package is `pytables`.\n",
    "\n",
    "```\n",
    "conda install pytables\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `ls` to confirm that the file exists and check the size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 downey downey 20M Dec 29 11:48 gd1_dataframe.hdf5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lh gd1_dataframe.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using Windows, `ls` might not work; in that case, try:\n",
    "\n",
    "```\n",
    "!dir gd1_dataframe.hdf5\n",
    "```\n",
    "\n",
    "We can read the file back like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140340, 12)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "read_back_df = pd.read_hdf(filename, 'results_df')\n",
    "read_back_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can write a variety of other formats, [which you can read about here](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this lesson, we re-loaded the Gaia data we saved from a previous query.\n",
    "\n",
    "We transformed the coordinates and proper motion from ICRS to a frame aligned with the orbit of GD-1, and stored the results in a Pandas `DataFrame`.\n",
    "\n",
    "Then we replicated the selection process from the Price-Whelan and Bonaca paper:\n",
    "\n",
    "* We selected stars near the centerline of GD-1 and made a scatter plot of their proper motion.\n",
    "\n",
    "* We identified a region of proper motion that contains stars likely to be in GD-1.\n",
    "\n",
    "* We used a Boolean `Series` as a mask to select stars whose proper motion is in that region.\n",
    "\n",
    "So far, we have used data from a relatively small region of the sky.  In the next lesson, we'll write a query that selects stars based on proper motion, which will allow us to explore a larger region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best practices\n",
    "\n",
    "* When you make a scatter plot, adjust the size of the markers and their transparency so the figure is not overplotted; otherwise it can misrepresent the data badly.\n",
    "\n",
    "* For simple scatter plots in Matplotlib, `plot` is faster than `scatter`.\n",
    "\n",
    "* An Astropy `Table` and a Pandas `DataFrame` are similar in many ways and they provide many of the same functions.  They have pros and cons, but for many projects, either one would be a reasonable choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
