{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "---\n",
    "title: \"Transform and Select\"\n",
    "teaching: 3000\n",
    "exercises: 0\n",
    "questions:\n",
    "\n",
    "- \"How do we move the computation to the data?\"\n",
    "\n",
    "objectives:\n",
    "\n",
    "- \"Transform proper motions from one frame to another.\"\n",
    "\n",
    "- \"Compute the convex hull of a set of points.\"\n",
    "\n",
    "- \"Write an ADQL query that selects based on proper motion.\"\n",
    "\n",
    "- \"Save data in CSV format.\"\n",
    "\n",
    "keypoints:\n",
    "\n",
    "- \"When possible, 'move the computation to the data'; that is, do as much of the work as possible on the database server before downloading the data.\"\n",
    "\n",
    "- \"For most applications, saving data in FITS or HDF5 is better than CSV.  FITS and HDF5 are binary formats, so the files are usually smaller, and they store metadata, so you don't lose anything when you read the file back.\"\n",
    "\n",
    "- \"On the other hand, CSV is a 'least common denominator' format; that is, it can be read by practically any application that works with data.\"\n",
    "\n",
    "---\n",
    "\n",
    "{% include links.md %}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformation and Selection\n",
    "\n",
    "This is the fourth in a series of notebooks related to astronomy data.\n",
    "\n",
    "As a running example, we are replicating parts of the analysis in a recent paper, \"[Off the beaten path: Gaia reveals GD-1 stars outside of the main stream](https://arxiv.org/abs/1805.00425)\" by Adrian M. Price-Whelan and Ana Bonaca.\n",
    "\n",
    "In the first lesson, we wrote ADQL queries and used them to select and download data from the Gaia server.\n",
    "\n",
    "In the second lesson, we write a query to select stars from the region of the sky where we expect GD-1 to be, and save the results in a FITS file.\n",
    "\n",
    "In the third lesson, we read that data back and identified stars with the proper motion we expect for GD-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "Here are the steps in this lesson:\n",
    "\n",
    "1. Using data from the previous lesson, we'll identify the values of proper motion for stars likely to be in GD-1.\n",
    "\n",
    "2. Then we'll compose an ADQL query that selects stars based on proper motion, so we can download only the data we need.\n",
    "\n",
    "3. We'll also see how to write the results to a CSV file.\n",
    "\n",
    "That will make it possible to search a bigger region of the sky in a single query.\n",
    "\n",
    "After completing this lesson, you should be able to\n",
    "\n",
    "* Transform proper motions from one frame to another.\n",
    "\n",
    "* Compute the convex hull of a set of points.\n",
    "\n",
    "* Write an ADQL query that selects based on proper motion.\n",
    "\n",
    "* Save data in CSV format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "## Installing libraries\n",
    "\n",
    "If you are running this notebook on Colab, you can run the following cell to install Astroquery and the other libraries we'll use.\n",
    "\n",
    "If you are running this notebook on your own computer, you might have to install these libraries yourself.  See the instructions in the preface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# If we're running on Colab, install libraries\n",
    "\n",
    "import sys\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if IN_COLAB:\n",
    "    !pip install astroquery astro-gala wget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reload the data\n",
    "\n",
    "The following cells download the data from the previous lesson, if necessary, and load it into a Pandas `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from wget import download\n",
    "\n",
    "filename = 'gd1_dataframe.hdf5'\n",
    "path = 'https://github.com/AllenDowney/AstronomicalData/raw/main/data/'\n",
    "\n",
    "if not os.path.exists(filename):\n",
    "    print(download(path+filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "centerline_df = pd.read_hdf(filename, 'centerline_df')\n",
    "selected_df = pd.read_hdf(filename, 'selected_df')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection by proper motion\n",
    "\n",
    "Let's review how we got to this point.\n",
    "\n",
    "1. We made an ADQL query to the Gaia server to get data for stars in the vicinity of GD-1.\n",
    "\n",
    "2. We transformed the coordinates to the `GD1Koposov10` frame so we could select stars along the centerline of GD-1.\n",
    "\n",
    "3. We plotted the proper motion of the centerline stars to identify the bounds of the overdense region.\n",
    "\n",
    "4. We made a mask that selects stars whose proper motion is in the overdense region.\n",
    "\n",
    "At this point we have downloaded data for a relatively large number of stars (more than 100,000) and selected a relatively small number (around 1000).\n",
    "\n",
    "It would be more efficient to use ADQL to select only the stars we need.  That would also make it possible to download data covering a larger region of the sky.\n",
    "\n",
    "However, the selection we did was based on proper motion in the `GD1Koposov10` frame.  In order to do the same selection in ADQL, we have to work with proper motions in ICRS.\n",
    "\n",
    "As a reminder, here's the rectangle we selected based on proper motion in the `GD1Koposov10` frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm1_min = -8.9\n",
    "pm1_max = -6.9\n",
    "pm2_min = -2.2\n",
    "pm2_max =  1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_rectangle(x1, x2, y1, y2):\n",
    "    \"\"\"Return the corners of a rectangle.\"\"\"\n",
    "    xs = [x1, x1, x2, x2, x1]\n",
    "    ys = [y1, y2, y2, y1, y1]\n",
    "    return xs, ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm1_rect, pm2_rect = make_rectangle(\n",
    "    pm1_min, pm1_max, pm2_min, pm2_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figure shows:\n",
    "\n",
    "* Proper motion for the stars we selected along the center line of GD-1,\n",
    "\n",
    "* The rectangle we selected, and\n",
    "\n",
    "* The stars inside the rectangle highlighted in green."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 111628 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pm1 = centerline_df['pm_phi1']\n",
    "pm2 = centerline_df['pm_phi2']\n",
    "plt.plot(pm1, pm2, 'ko', markersize=0.3, alpha=0.3)\n",
    "\n",
    "pm1 = selected_df['pm_phi1']\n",
    "pm2 = selected_df['pm_phi2']\n",
    "plt.plot(pm1, pm2, 'gx', markersize=0.3, alpha=0.3)\n",
    "\n",
    "plt.plot(pm1_rect, pm2_rect, '-')\n",
    "    \n",
    "plt.xlabel('Proper motion phi1 (GD1 frame)')\n",
    "plt.ylabel('Proper motion phi2 (GD1 frame)')\n",
    "\n",
    "plt.xlim(-12, 8)\n",
    "plt.ylim(-10, 10);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll make the same plot using proper motions in the ICRS frame, which are stored in columns `pmra` and `pmdec`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 109376 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm1 = centerline_df['pmra']\n",
    "pm2 = centerline_df['pmdec']\n",
    "plt.plot(pm1, pm2, 'ko', markersize=0.3, alpha=0.3)\n",
    "\n",
    "pm1 = selected_df['pmra']\n",
    "pm2 = selected_df['pmdec']\n",
    "plt.plot(pm1, pm2, 'gx', markersize=1, alpha=0.3)\n",
    "    \n",
    "plt.xlabel('Proper motion ra (ICRS frame)')\n",
    "plt.ylabel('Proper motion dec (ICRS frame)')\n",
    "\n",
    "plt.xlim([-10, 5])\n",
    "plt.ylim([-20, 5]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The proper motions of the selected stars are more spread out in this frame, which is why it was preferable to do the selection in the GD-1 frame.\n",
    "\n",
    "But now we can define a polygon that encloses the proper motions of these stars in ICRS, and use that polygon as a selection criterion in an ADQL query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convex Hull\n",
    "\n",
    "SciPy provides a function that computes the [convex hull](https://en.wikipedia.org/wiki/Convex_hull) of a set of points, which is the smallest convex polygon that contains all of the points.\n",
    "\n",
    "To use it, we'll select columns `pmra` and `pmdec` and convert them to a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1049, 2)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "points = selected_df[['pmra','pmdec']].to_numpy()\n",
    "points.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: If you are using an older version of Pandas, you might not have `to_numpy()`; you can use `values` instead, like this:\n",
    "\n",
    "```\n",
    "points = selected_df[['pmra','pmdec']].values\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll pass the points to `ConvexHull`, which returns an object that contains the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<scipy.spatial.qhull.ConvexHull at 0x7fa7c4c03a90>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.spatial import ConvexHull\n",
    "\n",
    "hull = ConvexHull(points)\n",
    "hull"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`hull.vertices` contains the indices of the points that fall on the perimeter of the hull."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 692,  873,  141,  303,   42,  622,   45,   83,  127,  182, 1006,\n",
       "        971,  967, 1001,  969,  940], dtype=int32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hull.vertices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use them as an index into the original array to select the corresponding rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -4.05037121, -14.75623261],\n",
       "       [ -3.41981085, -14.72365546],\n",
       "       [ -3.03521988, -14.44357135],\n",
       "       [ -2.26847919, -13.7140236 ],\n",
       "       [ -2.61172203, -13.24797471],\n",
       "       [ -2.73471401, -13.09054471],\n",
       "       [ -3.19923146, -12.5942653 ],\n",
       "       [ -3.34082546, -12.47611926],\n",
       "       [ -5.67489413, -11.16083338],\n",
       "       [ -5.95159272, -11.10547884],\n",
       "       [ -6.42394023, -11.05981295],\n",
       "       [ -7.09631023, -11.95187806],\n",
       "       [ -7.30641519, -12.24559977],\n",
       "       [ -7.04016696, -12.88580702],\n",
       "       [ -6.00347705, -13.75912098],\n",
       "       [ -4.42442296, -14.74641176]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pm_vertices = points[hull.vertices]\n",
    "pm_vertices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To plot the resulting polygon, we have to pull out the x and y coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "pmra_poly, pmdec_poly = np.transpose(pm_vertices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This use of `transpose` is a bit of a NumPy trick.  Because `pm_vertices` has two columns, its transpose has two rows, which are assigned to the two variables `pmra_poly` and `pmdec_poly`.\n",
    "\n",
    "The following figure shows proper motion in ICRS again, along with the convex hull we just computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 107596 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm1 = centerline_df['pmra']\n",
    "pm2 = centerline_df['pmdec']\n",
    "plt.plot(pm1, pm2, 'ko', markersize=0.3, alpha=0.3)\n",
    "\n",
    "pm1 = selected_df['pmra']\n",
    "pm2 = selected_df['pmdec']\n",
    "plt.plot(pm1, pm2, 'gx', markersize=0.3, alpha=0.3)\n",
    "\n",
    "plt.plot(pmra_poly, pmdec_poly)\n",
    "    \n",
    "plt.xlabel('Proper motion phi1 (ICRS frame)')\n",
    "plt.ylabel('Proper motion phi2 (ICRS frame)')\n",
    "\n",
    "plt.xlim([-10, 5])\n",
    "plt.ylim([-20, 5]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So `pm_vertices` represents the polygon we want to select.\n",
    "The next step is to use it as part of an ADQL query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assembling the query\n",
    "\n",
    "Here's the base string we used for the query in the previous lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_base = \"\"\"SELECT \n",
    "{columns}\n",
    "FROM gaiadr2.gaia_source\n",
    "WHERE parallax < 1\n",
    "  AND bp_rp BETWEEN -0.75 AND 2 \n",
    "  AND 1 = CONTAINS(POINT(ra, dec), \n",
    "                   POLYGON({point_list}))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are the changes we'll make in this lesson:\n",
    "\n",
    "1. We will add another clause to select stars whose proper motion is in the polygon we just computed, `pm_vertices`.\n",
    "\n",
    "2. We will select stars with coordinates in a larger region.\n",
    "\n",
    "To use `pm_vertices` as part of an ADQL query, we have to convert it to a string.\n",
    "\n",
    "We'll use `flatten` to convert from a 2-D array to a 1-D array, and `str` to convert each element to a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['-4.050371212154984',\n",
       " '-14.75623260987968',\n",
       " '-3.4198108491382455',\n",
       " '-14.723655456335619',\n",
       " '-3.035219883740934',\n",
       " '-14.443571352854612',\n",
       " '-2.268479190206636',\n",
       " '-13.714023598831554',\n",
       " '-2.611722027231764',\n",
       " '-13.247974712069263',\n",
       " '-2.7347140078529106',\n",
       " '-13.090544709622938',\n",
       " '-3.199231461993783',\n",
       " '-12.594265302440828',\n",
       " '-3.34082545787549',\n",
       " '-12.476119260818695',\n",
       " '-5.674894125178565',\n",
       " '-11.160833381392624',\n",
       " '-5.95159272432137',\n",
       " '-11.105478836426514',\n",
       " '-6.423940229776128',\n",
       " '-11.05981294804957',\n",
       " '-7.096310230579248',\n",
       " '-11.951878058650085',\n",
       " '-7.306415190921692',\n",
       " '-12.245599765990594',\n",
       " '-7.040166963232815',\n",
       " '-12.885807024935527',\n",
       " '-6.0034770546523735',\n",
       " '-13.759120984106968',\n",
       " '-4.42442296194263',\n",
       " '-14.7464117578883']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = [str(x) for x in pm_vertices.flatten()]\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `t` is a list of strings; we can use `join` to make a single string with commas between the elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'-4.050371212154984, -14.75623260987968, -3.4198108491382455, -14.723655456335619, -3.035219883740934, -14.443571352854612, -2.268479190206636, -13.714023598831554, -2.611722027231764, -13.247974712069263, -2.7347140078529106, -13.090544709622938, -3.199231461993783, -12.594265302440828, -3.34082545787549, -12.476119260818695, -5.674894125178565, -11.160833381392624, -5.95159272432137, -11.105478836426514, -6.423940229776128, -11.05981294804957, -7.096310230579248, -11.951878058650085, -7.306415190921692, -12.245599765990594, -7.040166963232815, -12.885807024935527, -6.0034770546523735, -13.759120984106968, -4.42442296194263, -14.7464117578883'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pm_point_list = ', '.join(t)\n",
    "pm_point_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll add this string to the query soon, but first let's compute the other polygon, the one that specifies the region of the sky we want.\n",
    "\n",
    "Here are the coordinates of the rectangle we'll select, in the GD-1 frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import astropy.units as u\n",
    "\n",
    "phi1_min = -70 * u.degree\n",
    "phi1_max = -20 * u.degree\n",
    "phi2_min = -5 * u.degree\n",
    "phi2_max = 5 * u.degree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "phi1_rect, phi2_rect = make_rectangle(\n",
    "    phi1_min, phi1_max, phi2_min, phi2_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we transform it to ICRS, as we saw in the previous lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gala.coordinates import GD1Koposov10\n",
    "from astropy.coordinates import SkyCoord\n",
    "\n",
    "corners = SkyCoord(phi1=phi1_rect, phi2=phi2_rect, frame=GD1Koposov10)\n",
    "corners_icrs = corners.transform_to('icrs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use `corners_icrs` as part of an ADQL query, we have to convert it to a string.  Here's how we do that, as we saw in the previous lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'135.30559858565638, 8.398623940157561, 126.50951508623503, 13.44494195652069, 163.0173655836748, 54.24242734020255, 172.9328536286811, 46.47260492416258, 135.30559858565638, 8.398623940157561'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "point_base = \"{point.ra.value}, {point.dec.value}\"\n",
    "\n",
    "t = [point_base.format(point=point)\n",
    "     for point in corners_icrs]\n",
    "\n",
    "point_list = ', '.join(t)\n",
    "point_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have everything we need to assemble the query.\n",
    "Here's the base query from the previous lesson again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_base = \"\"\"SELECT \n",
    "{columns}\n",
    "FROM gaiadr2.gaia_source\n",
    "WHERE parallax < 1\n",
    "  AND bp_rp BETWEEN -0.75 AND 2 \n",
    "  AND 1 = CONTAINS(POINT(ra, dec), \n",
    "                   POLYGON({point_list}))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Modify `query_base` by adding a new clause to select stars whose coordinates of proper motion, `pmra` and `pmdec`, fall within the polygon defined by `pm_point_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "\n",
    "query_base = \"\"\"SELECT \n",
    "{columns}\n",
    "FROM gaiadr2.gaia_source\n",
    "WHERE parallax < 1\n",
    "  AND bp_rp BETWEEN -0.75 AND 2 \n",
    "  AND 1 = CONTAINS(POINT(ra, dec), \n",
    "                   POLYGON({point_list}))\n",
    "  AND 1 = CONTAINS(POINT(pmra, pmdec),\n",
    "                   POLYGON({pm_point_list}))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here again are the columns we want to select."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = 'source_id, ra, dec, pmra, pmdec, parallax, radial_velocity'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Use `format` to format `query_base` and define `query`, filling in the values of `columns`, `point_list`, and `pm_point_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SELECT \n",
      "source_id, ra, dec, pmra, pmdec, parallax, radial_velocity\n",
      "FROM gaiadr2.gaia_source\n",
      "WHERE parallax < 1\n",
      "  AND bp_rp BETWEEN -0.75 AND 2 \n",
      "  AND 1 = CONTAINS(POINT(ra, dec), \n",
      "                   POLYGON(135.30559858565638, 8.398623940157561, 126.50951508623503, 13.44494195652069, 163.0173655836748, 54.24242734020255, 172.9328536286811, 46.47260492416258, 135.30559858565638, 8.398623940157561))\n",
      "  AND 1 = CONTAINS(POINT(pmra, pmdec),\n",
      "                   POLYGON(-4.050371212154984, -14.75623260987968, -3.4198108491382455, -14.723655456335619, -3.035219883740934, -14.443571352854612, -2.268479190206636, -13.714023598831554, -2.611722027231764, -13.247974712069263, -2.7347140078529106, -13.090544709622938, -3.199231461993783, -12.594265302440828, -3.34082545787549, -12.476119260818695, -5.674894125178565, -11.160833381392624, -5.95159272432137, -11.105478836426514, -6.423940229776128, -11.05981294804957, -7.096310230579248, -11.951878058650085, -7.306415190921692, -12.245599765990594, -7.040166963232815, -12.885807024935527, -6.0034770546523735, -13.759120984106968, -4.42442296194263, -14.7464117578883))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "query = query_base.format(columns=columns, \n",
    "                            point_list=point_list,\n",
    "                            pm_point_list=pm_point_list)\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run the query like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created TAP+ (v1.2.1) - Connection:\n",
      "\tHost: gea.esac.esa.int\n",
      "\tUse HTTPS: True\n",
      "\tPort: 443\n",
      "\tSSL Port: 443\n",
      "Created TAP+ (v1.2.1) - Connection:\n",
      "\tHost: geadata.esac.esa.int\n",
      "\tUse HTTPS: True\n",
      "\tPort: 443\n",
      "\tSSL Port: 443\n",
      "INFO: Query finished. [astroquery.utils.tap.core]\n",
      "<Table length=7346>\n",
      "      name       dtype    unit                              description                             n_bad\n",
      "--------------- ------- -------- ------------------------------------------------------------------ -----\n",
      "      source_id   int64          Unique source identifier (unique within a particular Data Release)     0\n",
      "             ra float64      deg                                                    Right ascension     0\n",
      "            dec float64      deg                                                        Declination     0\n",
      "           pmra float64 mas / yr                         Proper motion in right ascension direction     0\n",
      "          pmdec float64 mas / yr                             Proper motion in declination direction     0\n",
      "       parallax float64      mas                                                           Parallax     0\n",
      "radial_velocity float64   km / s                                                    Radial velocity  7295\n",
      "Jobid: 1609260557475O\n",
      "Phase: COMPLETED\n",
      "Owner: None\n",
      "Output file: async_20201229114917.vot\n",
      "Results: None\n"
     ]
    }
   ],
   "source": [
    "from astroquery.gaia import Gaia\n",
    "\n",
    "job = Gaia.launch_job_async(query)\n",
    "print(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And get the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7346"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "candidate_table = job.get_results()\n",
    "len(candidate_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting one more time\n",
    "\n",
    "Let's see what the results look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 48944 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = candidate_table['ra']\n",
    "y = candidate_table['dec']\n",
    "plt.plot(x, y, 'ko', markersize=0.3, alpha=0.3)\n",
    "\n",
    "plt.xlabel('ra (degree ICRS)')\n",
    "plt.ylabel('dec (degree ICRS)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see why it was useful to transform these coordinates.  In ICRS, it is more difficult to identity the stars near the centerline of GD-1.\n",
    "\n",
    "So, before we move on to the next step, let's collect the code we used to transform the coordinates and make a Pandas `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gala.coordinates import reflex_correct\n",
    "\n",
    "def make_dataframe(table):\n",
    "    \"\"\"Transform coordinates from ICRS to GD-1 frame.\n",
    "    \n",
    "    table: Astropy Table\n",
    "    \n",
    "    returns: Pandas DataFrame\n",
    "    \"\"\"\n",
    "    skycoord = SkyCoord(\n",
    "               ra=table['ra'], \n",
    "               dec=table['dec'],\n",
    "               pm_ra_cosdec=table['pmra'],\n",
    "               pm_dec=table['pmdec'], \n",
    "               distance=8*u.kpc, \n",
    "               radial_velocity=0*u.km/u.s)\n",
    "\n",
    "    transformed = skycoord.transform_to(GD1Koposov10)\n",
    "    gd1_coord = reflex_correct(transformed)\n",
    "\n",
    "    df = table.to_pandas()\n",
    "    df['phi1'] = gd1_coord.phi1\n",
    "    df['phi2'] = gd1_coord.phi2\n",
    "    df['pm_phi1'] = gd1_coord.pm_phi1_cosphi2\n",
    "    df['pm_phi2'] = gd1_coord.pm_phi2\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we can use this function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_df = make_dataframe(candidate_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's see the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56532 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = candidate_df['phi1']\n",
    "y = candidate_df['phi2']\n",
    "\n",
    "plt.plot(x, y, 'ko', markersize=0.5, alpha=0.5)\n",
    "\n",
    "plt.xlabel('ra (degree GD1)')\n",
    "plt.ylabel('dec (degree GD1)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're starting to see GD-1 more clearly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare this figure with one of these panels in Figure 1 from the original paper:\n",
    "\n",
    "<img height=\"150\" src=\"https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-2.png\">\n",
    "\n",
    "<img height=\"150\" src=\"https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-4.png\">\n",
    "\n",
    "The top panel shows stars selected based on proper motion only, so it is comparable to our figure (although notice that it covers a wider region)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next lesson, we will use photometry data from Pan-STARRS to do a second round of filtering, and see if we can replicate the bottom panel.\n",
    "\n",
    "We'll also learn how to add annotations like the ones in the figure from the paper, and customize the style of the figure to present the results clearly and compellingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the DataFrame\n",
    "\n",
    "Let's save this `DataFrame` so we can pick up where we left off without running this query again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'gd1_candidates.hdf5'\n",
    "\n",
    "candidate_df.to_hdf(filename, 'candidate_df', mode='w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `ls` to confirm that the file exists and check the size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 downey downey 698K Dec 29 11:50 gd1_candidates.hdf5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lh gd1_candidates.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using Windows, `ls` might not work; in that case, try:\n",
    "\n",
    "```\n",
    "!dir gd1_candidates.hdf5\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV\n",
    "\n",
    "Pandas can write a variety of other formats, [which you can read about here](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html).\n",
    "\n",
    "We won't cover all of them, but one other important one is [CSV](https://en.wikipedia.org/wiki/Comma-separated_values), which stands for \"comma-separated values\".\n",
    "\n",
    "CSV is a plain-text format with minimal formatting requirements, so it can be read and written by pretty much any tool that works with data.  In that sense, it is the \"least common denominator\" of data formats.\n",
    "\n",
    "However, it has an important limitation: some information about the data gets lost in translation, notably the data types.  If you read a CSV file from someone else, you might need some additional information to make sure you are getting it right.\n",
    "\n",
    "Also, CSV files tend to be big, and slow to read and write.\n",
    "\n",
    "With those caveats, here's how to write one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_df.to_csv('gd1_candidates.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the file size like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 downey downey 1.4M Dec 29 11:50 gd1_candidates.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lh gd1_candidates.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CSV file about 2 times bigger than the HDF5 file (so that's not that bad, really).\n",
    "\n",
    "We can see the first few lines like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ",source_id,ra,dec,pmra,pmdec,parallax,radial_velocity,phi1,phi2,pm_phi1,pm_phi2\r\n",
      "0,635559124339440000,137.58671691646745,19.1965441084838,-3.770521900009566,-12.490481778113859,0.7913934419894347,,-59.63048941944402,-1.2164852515042963,-7.361362712597496,-0.592632882064492\r\n",
      "1,635860218726658176,138.5187065217173,19.09233926905897,-5.941679495793577,-11.346409129876392,0.30745551377348623,,-59.247329893833296,-2.016078400820631,-7.527126084640531,1.7487794924176672\r\n"
     ]
    }
   ],
   "source": [
    "!head -3 gd1_candidates.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CSV file contains the names of the columns, but not the data types.\n",
    "\n",
    "We can read the CSV file back like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_back_csv = pd.read_csv('gd1_candidates.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the first few rows of `candidate_df` and `read_back_csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_id</th>\n",
       "      <th>ra</th>\n",
       "      <th>dec</th>\n",
       "      <th>pmra</th>\n",
       "      <th>pmdec</th>\n",
       "      <th>parallax</th>\n",
       "      <th>radial_velocity</th>\n",
       "      <th>phi1</th>\n",
       "      <th>phi2</th>\n",
       "      <th>pm_phi1</th>\n",
       "      <th>pm_phi2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>635559124339440000</td>\n",
       "      <td>137.586717</td>\n",
       "      <td>19.196544</td>\n",
       "      <td>-3.770522</td>\n",
       "      <td>-12.490482</td>\n",
       "      <td>0.791393</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.630489</td>\n",
       "      <td>-1.216485</td>\n",
       "      <td>-7.361363</td>\n",
       "      <td>-0.592633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>635860218726658176</td>\n",
       "      <td>138.518707</td>\n",
       "      <td>19.092339</td>\n",
       "      <td>-5.941679</td>\n",
       "      <td>-11.346409</td>\n",
       "      <td>0.307456</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.247330</td>\n",
       "      <td>-2.016078</td>\n",
       "      <td>-7.527126</td>\n",
       "      <td>1.748779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>635674126383965568</td>\n",
       "      <td>138.842874</td>\n",
       "      <td>19.031798</td>\n",
       "      <td>-3.897001</td>\n",
       "      <td>-12.702780</td>\n",
       "      <td>0.779463</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.133391</td>\n",
       "      <td>-2.306901</td>\n",
       "      <td>-7.560608</td>\n",
       "      <td>-0.741800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            source_id          ra        dec      pmra      pmdec  parallax  \\\n",
       "0  635559124339440000  137.586717  19.196544 -3.770522 -12.490482  0.791393   \n",
       "1  635860218726658176  138.518707  19.092339 -5.941679 -11.346409  0.307456   \n",
       "2  635674126383965568  138.842874  19.031798 -3.897001 -12.702780  0.779463   \n",
       "\n",
       "   radial_velocity       phi1      phi2   pm_phi1   pm_phi2  \n",
       "0              NaN -59.630489 -1.216485 -7.361363 -0.592633  \n",
       "1              NaN -59.247330 -2.016078 -7.527126  1.748779  \n",
       "2              NaN -59.133391 -2.306901 -7.560608 -0.741800  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "candidate_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>source_id</th>\n",
       "      <th>ra</th>\n",
       "      <th>dec</th>\n",
       "      <th>pmra</th>\n",
       "      <th>pmdec</th>\n",
       "      <th>parallax</th>\n",
       "      <th>radial_velocity</th>\n",
       "      <th>phi1</th>\n",
       "      <th>phi2</th>\n",
       "      <th>pm_phi1</th>\n",
       "      <th>pm_phi2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>635559124339440000</td>\n",
       "      <td>137.586717</td>\n",
       "      <td>19.196544</td>\n",
       "      <td>-3.770522</td>\n",
       "      <td>-12.490482</td>\n",
       "      <td>0.791393</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.630489</td>\n",
       "      <td>-1.216485</td>\n",
       "      <td>-7.361363</td>\n",
       "      <td>-0.592633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>635860218726658176</td>\n",
       "      <td>138.518707</td>\n",
       "      <td>19.092339</td>\n",
       "      <td>-5.941679</td>\n",
       "      <td>-11.346409</td>\n",
       "      <td>0.307456</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.247330</td>\n",
       "      <td>-2.016078</td>\n",
       "      <td>-7.527126</td>\n",
       "      <td>1.748779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>635674126383965568</td>\n",
       "      <td>138.842874</td>\n",
       "      <td>19.031798</td>\n",
       "      <td>-3.897001</td>\n",
       "      <td>-12.702780</td>\n",
       "      <td>0.779463</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-59.133391</td>\n",
       "      <td>-2.306901</td>\n",
       "      <td>-7.560608</td>\n",
       "      <td>-0.741800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0           source_id          ra        dec      pmra      pmdec  \\\n",
       "0           0  635559124339440000  137.586717  19.196544 -3.770522 -12.490482   \n",
       "1           1  635860218726658176  138.518707  19.092339 -5.941679 -11.346409   \n",
       "2           2  635674126383965568  138.842874  19.031798 -3.897001 -12.702780   \n",
       "\n",
       "   parallax  radial_velocity       phi1      phi2   pm_phi1   pm_phi2  \n",
       "0  0.791393              NaN -59.630489 -1.216485 -7.361363 -0.592633  \n",
       "1  0.307456              NaN -59.247330 -2.016078 -7.527126  1.748779  \n",
       "2  0.779463              NaN -59.133391 -2.306901 -7.560608 -0.741800  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "read_back_csv.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the index in `candidate_df` has become an unnamed column in `read_back_csv`.  The Pandas functions for writing and reading CSV files provide options to avoid that problem, but this is an example of the kind of thing that can go wrong with CSV files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In the previous lesson we downloaded data for a large number of stars and then selected a small fraction of them based on proper motion.\n",
    "\n",
    "In this lesson, we improved this process by writing a more complex query that uses the database to select stars based on proper motion.  This process requires more computation on the Gaia server, but then we're able to either:\n",
    "\n",
    "1. Search the same region and download less data, or\n",
    "\n",
    "2. Search a larger region while still downloading a manageable amount of data.\n",
    "\n",
    "In the next lesson, we'll learn about the databased `JOIN` operation and use it to download photometry data from Pan-STARRS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best practices\n",
    "\n",
    "* When possible, \"move the computation to the data\"; that is, do as much of the work as possible on the database server before downloading the data.\n",
    "\n",
    "* For most applications, saving data in FITS or HDF5 is better than CSV.  FITS and HDF5 are binary formats, so the files are usually smaller, and they store metadata, so you don't lose anything when you read the file back.\n",
    "\n",
    "* On the other hand, CSV is a \"least common denominator\" format; that is, it can be read by practically any application that works with data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
