gitea-mirror/AstronomicalData
1
0
Fork 0
mirror of https://github.com/AllenDowney/AstronomicalData.git synced 2025-12-22 07:10:38 -08:00
Code Issues Packages Projects Releases Wiki Activity
Files
2f2d0fe7eec8d44dfc263ef5f08fc6f757f01cba
AstronomicalData/05_join.html
Allen Downey 2f2d0fe7ee Update documentation
2021-03-17 11:26:14 -04:00

1812 lines
111 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>5. Joining Tables &#8212; Astronomical Data in Python</title>
<link rel="stylesheet" href="_static/css/index.d431a4ee1c1efae0e38bdfebc22debff.css">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet"
href="_static/vendor/open-sans_all/1.44.1/index.css">
<link rel="stylesheet"
href="_static/vendor/lato_latin-ext/1.44.1/index.css">
<link rel="stylesheet" href="_static/sphinx-book-theme.bfb7730f9caf2ec0b46a44615585038c.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.30270b6e4c972e43c488.js">
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/language_data.js"></script>
<script src="_static/togglebutton.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script >var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.be0a4a0c39cd630af62a2fcf693f3f06.js"></script>
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["\\(", "\\)"]], "displayMath": [["\\[", "\\]"]], "processRefs": false, "processEnvironments": false}})</script>
<script async="async" src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
<script >
const thebe_selector = ".thebe"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="6. Photometry" href="06_photo.html" />
<link rel="prev" title="4. Transformation and Selection" href="04_select.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<h1 class="site-logo" id="site-title">Astronomical Data in Python</h1>
</a>
</div>
<form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form>
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<ul class="nav sidenav_l1">
<li class="toctree-l1">
<a class="reference internal" href="README.html">
Astronomical Data in Python
</a>
</li>
</ul>
<ul class="current nav sidenav_l1">
<li class="toctree-l1">
<a class="reference internal" href="01_query.html">
1. Queries
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="02_coords.html">
2. Coordinates and Units
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="03_motion.html">
3. Proper Motion
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="04_select.html">
4. Transformation and Selection
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
5. Joining Tables
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="06_photo.html">
6. Photometry
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="07_plot.html">
Visualization
</a>
</li>
</ul>
</nav>
<!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="row topbar fixed-top container-xl">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show">
</div>
<div class="col pl-2 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/05_join.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onClick="window.print()" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn"
aria-label="Connect with source repository"><i class="fab fa-github"></i></button>
<div class="dropdown-buttons sourcebuttons">
<a class="repository-button"
href="https://github.com/AllenDowney/AstronomicalData"><button type="button" class="btn btn-secondary topbarbtn"
data-toggle="tooltip" data-placement="left" title="Source repository"><i
class="fab fa-github"></i>repository</button></a>
</div>
</div>
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn"
aria-label="Launch interactive content"><i class="fas fa-rocket"></i></button>
<div class="dropdown-buttons">
<a class="binder-button" href="https://mybinder.org/v2/gh/AllenDowney/AstronomicalData/master?urlpath=tree/05_join.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Launch Binder" data-toggle="tooltip"
data-placement="left"><img class="binder-button-logo"
src="_static/images/logo_binder.svg"
alt="Interact on binder">Binder</button></a>
<a class="colab-button" href="https://colab.research.google.com/github/AllenDowney/AstronomicalData/blob/master/05_join.ipynb"><button type="button" class="btn btn-secondary topbarbtn"
title="Launch Colab" data-toggle="tooltip" data-placement="left"><img class="colab-button-logo"
src="_static/images/logo_colab.png"
alt="Interact on Colab">Colab</button></a>
</div>
</div>
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav">
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#outline">
Outline
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#getting-photometry-data">
Getting photometry data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-best-neighbor-table">
The best neighbor table
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-pan-starrs-table">
The Pan-STARRS table
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
Joining tables
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adding-the-best-neighbor-table">
Adding the best neighbor table
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adding-the-pan-starrs-table">
Adding the Pan-STARRS table
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise">
Exercise
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#selecting-by-coordinates-and-proper-motion">
Selecting by coordinates and proper motion
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#id2">
Exercise
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#checking-the-match">
Checking the match
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#transforming-coordinates">
Transforming coordinates
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#saving-the-dataframe">
Saving the DataFrame
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#summary">
Summary
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#csv">
CSV
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#best-practices">
Best practices
</a>
</li>
</ul>
</nav>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<div>
<div class="section" id="joining-tables">
<h1>5. Joining Tables<a class="headerlink" href="#joining-tables" title="Permalink to this headline"></a></h1>
<p>This is the fifth in a series of notebooks related to astronomy data.</p>
<p>As a continuing example, we will replicate part of the analysis in a recent paper, “<a class="reference external" href="https://arxiv.org/abs/1805.00425">Off the beaten path: Gaia reveals GD-1 stars outside of the main stream</a>” by Adrian M. Price-Whelan and Ana Bonaca.</p>
<p>Picking up where we left off, the next step in the analysis is to select candidate stars based on photometry data.
The following figure from the paper is a color-magnitude diagram for the stars selected based on proper motion:</p>
<a class="reference internal image-reference" href="https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-3.png"><img alt="https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-3.png" src="https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/gd1-3.png" style="width: 300px;" /></a>
<p>In red is a <a class="reference external" href="https://en.wikipedia.org/wiki/Stellar_isochrone">stellar isochrone</a>, showing where we expect the stars in GD-1 to fall based on the metallicity and age of their original globular cluster.</p>
<p>By selecting stars in the shaded area, we can further distinguish the main sequence of GD-1 from younger background stars.</p>
<div class="section" id="outline">
<h2>Outline<a class="headerlink" href="#outline" title="Permalink to this headline"></a></h2>
<p>Here are the steps in this notebook:</p>
<ol class="simple">
<li><p>We’ll reload the candidate stars we identified in the previous notebook.</p></li>
<li><p>Then we’ll run a query on the Gaia server that uploads the table of candidates and uses a <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation to select photometry data for the candidate stars.</p></li>
<li><p>We’ll write the results to a file for use in the next notebook.</p></li>
</ol>
<p>After completing this lesson, you should be able to</p>
<ul class="simple">
<li><p>Upload a table to the Gaia server.</p></li>
<li><p>Write ADQL queries involving <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operations.</p></li>
</ul>
</div>
<div class="section" id="getting-photometry-data">
<h2>Getting photometry data<a class="headerlink" href="#getting-photometry-data" title="Permalink to this headline"></a></h2>
<p>The Gaia dataset contains some photometry data, including the variable <code class="docutils literal notranslate"><span class="pre">bp_rp</span></code>, which contains BP-RP color (the difference in mean flux between the BP and RP bands).
We use this variable to select stars with <code class="docutils literal notranslate"><span class="pre">bp_rp</span></code> between -0.75 and 2, which excludes many class M dwarf stars.</p>
<p>Now, to select stars with the age and metal richness we expect in GD-1, we will use <code class="docutils literal notranslate"><span class="pre">g-i</span></code> color and apparent <code class="docutils literal notranslate"><span class="pre">g</span></code>-band magnitude, which are available from the Pan-STARRS survey.</p>
<p>Conveniently, the Gaia server provides data from Pan-STARRS as a table in the same database we have been using, so we can access it by making ADQL queries.</p>
<p>In general, choosing a star from the Gaia catalog and finding the corresponding star in the Pan-STARRS catalog is not easy. This kind of cross matching is not always possible, because a star might appear in one catalog and not the other. And even when both stars are present, there might not be a clear one-to-one relationship between stars in the two catalogs.</p>
<p>Fortunately, smart people have worked on this problem, and the Gaia database includes cross-matching tables that suggest a best neighbor in the Pan-STARRS catalog for many stars in the Gaia catalog.</p>
<p><a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Catalogue_consolidation/chap_cu9val_cu9val/ssec_cu9xma/sssec_cu9xma_extcat.html">This document describes the cross matching process</a>. Briefly, it uses a cone search to find possible matches in approximately the right position, then uses attributes like color and magnitude to choose pairs of observations most likely to be the same star.</p>
</div>
<div class="section" id="the-best-neighbor-table">
<h2>The best neighbor table<a class="headerlink" href="#the-best-neighbor-table" title="Permalink to this headline"></a></h2>
<p>So the hard part of cross-matching has been done for us. Using the results is a little tricky, but it gives us a chance to learn about one of the most important tools for working with databases: “joining” tables.</p>
<p>In general, a “join” is an operation where you match up records from one table with records from another table using as a “key” a piece of information that is common to both tables, usually some kind of ID code.</p>
<p>In this example:</p>
<ul class="simple">
<li><p>Stars in the Gaia dataset are identified by <code class="docutils literal notranslate"><span class="pre">source_id</span></code>.</p></li>
<li><p>Stars in the Pan-STARRS dataset are identified by <code class="docutils literal notranslate"><span class="pre">obj_id</span></code>.</p></li>
</ul>
<p>For each candidate star we have selected so far, we have the <code class="docutils literal notranslate"><span class="pre">source_id</span></code>; the goal is to find the <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> for the same star (we hope) in the Pan-STARRS catalog.</p>
<p>To do that we will:</p>
<ol class="simple">
<li><p>Use the <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operator to look up each <code class="docutils literal notranslate"><span class="pre">source_id</span></code> in the <code class="docutils literal notranslate"><span class="pre">panstarrs1_best_neighbour</span></code> table, which contains the <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> of the best match for each star in the Gaia catalog; then</p></li>
<li><p>Use the <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operator again to look up each <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> in the <code class="docutils literal notranslate"><span class="pre">panstarrs1_original_valid</span></code> table, which contains the Pan-STARRS photometry data we want.</p></li>
</ol>
<p>Before we get to the <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation, let’s explore these tables.
Here’s the metadata for <code class="docutils literal notranslate"><span class="pre">panstarrs1_best_neighbour</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">astroquery.gaia</span> <span class="kn">import</span> <span class="n">Gaia</span>
<span class="n">meta</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_table</span><span class="p">(</span><span class="s1">&#39;gaiadr2.panstarrs1_best_neighbour&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Created TAP+ (v1.2.1) - Connection:
Host: gea.esac.esa.int
Use HTTPS: True
Port: 443
SSL Port: 443
Created TAP+ (v1.2.1) - Connection:
Host: geadata.esac.esa.int
Use HTTPS: True
Port: 443
SSL Port: 443
Retrieving table &#39;gaiadr2.panstarrs1_best_neighbour&#39;
Parsing table &#39;gaiadr2.panstarrs1_best_neighbour&#39;...
Done.
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">meta</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>TAP Table name: gaiadr2.gaiadr2.panstarrs1_best_neighbour
Description: Pan-STARRS1 BestNeighbour table lists each matched Gaia object with its
best neighbour in the external catalogue.
There are 1 327 157 objects in the filtered version of Pan-STARRS1 used
to compute this cross-match that have too early epochMean.
Num. columns: 7
</pre></div>
</div>
</div>
</div>
<p>And here are the columns.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">meta</span><span class="o">.</span><span class="n">columns</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">column</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>source_id
original_ext_source_id
angular_distance
number_of_neighbours
number_of_mates
best_neighbour_multiplicity
gaia_astrometric_params
</pre></div>
</div>
</div>
</div>
<p>Here’s the <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_crossmatches/ssec_dm_panstarrs1_best_neighbour.html">documentation for these variables</a> .</p>
<p>The ones we’ll use are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">source_id</span></code>, which we will match up with <code class="docutils literal notranslate"><span class="pre">source_id</span></code> in the Gaia table.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">number_of_neighbours</span></code>, which indicates how many sources in Pan-STARRS are matched with this source in Gaia.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">number_of_mates</span></code>, which indicates the number of <em>other</em> sources in Gaia that are matched with the same source in Pan-STARRS.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">original_ext_source_id</span></code>, which we will match up with <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> in the Pan-STARRS table.</p></li>
</ul>
<p>Ideally, <code class="docutils literal notranslate"><span class="pre">number_of_neighbours</span></code> should be 1 and <code class="docutils literal notranslate"><span class="pre">number_of_mates</span></code> should be 0; in that case, there is a one-to-one match between the source in Gaia and the corresponding source in Pan-STARRS.</p>
<p>Here’s a query that selects these columns and returns the first 5 rows.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="s2">TOP 5</span>
<span class="s2">source_id, number_of_neighbours, number_of_mates, original_ext_source_id</span>
<span class="s2">FROM gaiadr2.panstarrs1_best_neighbour</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=5</i>
<table id="table140104392580448" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>number_of_neighbours</th><th>number_of_mates</th><th>original_ext_source_id</th></tr></thead>
<thead><tr><th>int64</th><th>int32</th><th>int16</th><th>int64</th></tr></thead>
<tr><td>6745938972433480704</td><td>1</td><td>0</td><td>69742925668851205</td></tr>
<tr><td>6030466788955954048</td><td>1</td><td>0</td><td>69742509325691172</td></tr>
<tr><td>6756488099308169600</td><td>1</td><td>0</td><td>69742879438541228</td></tr>
<tr><td>6700154994715046016</td><td>1</td><td>0</td><td>69743055581721207</td></tr>
<tr><td>6757061941303252736</td><td>1</td><td>0</td><td>69742856540241198</td></tr>
</table></div></div>
</div>
</div>
<div class="section" id="the-pan-starrs-table">
<h2>The Pan-STARRS table<a class="headerlink" href="#the-pan-starrs-table" title="Permalink to this headline"></a></h2>
<p>Here’s the metadata for the table that contains the Pan-STARRS data.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">meta</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_table</span><span class="p">(</span><span class="s1">&#39;gaiadr2.panstarrs1_original_valid&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Retrieving table &#39;gaiadr2.panstarrs1_original_valid&#39;
Parsing table &#39;gaiadr2.panstarrs1_original_valid&#39;...
Done.
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">meta</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>TAP Table name: gaiadr2.gaiadr2.panstarrs1_original_valid
Description: The Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) is
a system for wide-field astronomical imaging developed and operated by
the Institute for Astronomy at the University of Hawaii. Pan-STARRS1
(PS1) is the first part of Pan-STARRS to be completed and is the basis
for Data Release 1 (DR1). The PS1 survey used a 1.8 meter telescope and
its 1.4 Gigapixel camera to image the sky in five broadband filters (g,
r, i, z, y).
The current table contains a filtered subsample of the 10 723 304 629
entries listed in the original ObjectThin table.
We used only ObjectThin and MeanObject tables to extract
panstarrs1OriginalValid table, this means that objects detected only in
stack images are not included here. The main reason for us to avoid the
use of objects detected in stack images is that their astrometry is not
as good as the mean objects astrometry: “The stack positions (raStack,
decStack) have considerably larger systematic astrometric errors than
the mean epoch positions (raMean, decMean).” The astrometry for the
MeanObject positions uses Gaia DR1 as a reference catalog, while the
stack positions use 2MASS as a reference catalog.
In details, we filtered out all objects where:
- nDetections = 1
- no good quality data in Pan-STARRS, objInfoFlag 33554432 not set
- mean astrometry could not be measured, objInfoFlag 524288 set
- stack position used for mean astrometry, objInfoFlag 1048576 set
- error on all magnitudes equal to 0 or to -999;
- all magnitudes set to -999;
- error on RA or DEC greater than 1 arcsec.
The number of objects in panstarrs1OriginalValid is 2 264 263 282.
The panstarrs1OriginalValid table contains only a subset of the columns
available in the combined ObjectThin and MeanObject tables. A
description of the original ObjectThin and MeanObjects tables can be
found at:
https://outerspace.stsci.edu/display/PANSTARRS/PS1+Database+object+and+detection+tables
Download:
http://mastweb.stsci.edu/ps1casjobs/home.aspx
Documentation:
https://outerspace.stsci.edu/display/PANSTARRS
http://pswww.ifa.hawaii.edu/pswww/
References:
The Pan-STARRS1 Surveys, Chambers, K.C., et al. 2016, arXiv:1612.05560
Pan-STARRS Data Processing System, Magnier, E. A., et al. 2016,
arXiv:1612.05240
Pan-STARRS Pixel Processing: Detrending, Warping, Stacking, Waters, C.
Z., et al. 2016, arXiv:1612.05245
Pan-STARRS Pixel Analysis: Source Detection and Characterization,
Magnier, E. A., et al. 2016, arXiv:1612.05244
Pan-STARRS Photometric and Astrometric Calibration, Magnier, E. A., et
al. 2016, arXiv:1612.05242
The Pan-STARRS1 Database and Data Products, Flewelling, H. A., et al.
2016, arXiv:1612.05243
Catalogue curator:
SSDC - ASI Space Science Data Center
https://www.ssdc.asi.it/
Num. columns: 26
</pre></div>
</div>
</div>
</div>
<p>And here are the columns.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">meta</span><span class="o">.</span><span class="n">columns</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">column</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>obj_name
obj_id
ra
dec
ra_error
dec_error
epoch_mean
g_mean_psf_mag
g_mean_psf_mag_error
g_flags
r_mean_psf_mag
r_mean_psf_mag_error
r_flags
i_mean_psf_mag
i_mean_psf_mag_error
i_flags
z_mean_psf_mag
z_mean_psf_mag_error
z_flags
y_mean_psf_mag
y_mean_psf_mag_error
y_flags
n_detections
zone_id
obj_info_flag
quality_flag
</pre></div>
</div>
</div>
</div>
<p>Here’s the <span class="xref myst">documentation for these variables</span> .</p>
<p>The ones we’ll use are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">obj_id</span></code>, which we will match up with <code class="docutils literal notranslate"><span class="pre">original_ext_source_id</span></code> in the best neighbor table.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">g_mean_psf_mag</span></code>, which contains mean magnitude from the <code class="docutils literal notranslate"><span class="pre">i</span></code> filter.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">i_mean_psf_mag</span></code>, which contains mean magnitude from the <code class="docutils literal notranslate"><span class="pre">i</span></code> filter.</p></li>
</ul>
<p>Here’s a query that selects these variables and returns the first 5 rows.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="s2">TOP 5</span>
<span class="s2">obj_id, g_mean_psf_mag, i_mean_psf_mag </span>
<span class="s2">FROM gaiadr2.panstarrs1_original_valid</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=5</i>
<table id="table140104844641184" class="table-striped table-bordered table-condensed">
<thead><tr><th>obj_id</th><th>g_mean_psf_mag</th><th>i_mean_psf_mag</th></tr></thead>
<thead><tr><th></th><th></th><th>mag</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>67130655389101425</td><td>--</td><td>20.3516006469727</td></tr>
<tr><td>67553305590067819</td><td>--</td><td>19.779899597168</td></tr>
<tr><td>67551423248967849</td><td>--</td><td>19.8889007568359</td></tr>
<tr><td>67132026238911331</td><td>--</td><td>20.9062995910645</td></tr>
<tr><td>67553513677687787</td><td>--</td><td>21.2831001281738</td></tr>
</table></div></div>
</div>
<p>The following figure shows how these tables are related.</p>
<ul class="simple">
<li><p>The orange circles and arrows represent the first <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation, which takes each <code class="docutils literal notranslate"><span class="pre">source_id</span></code> in the Gaia table and finds the same value of <code class="docutils literal notranslate"><span class="pre">source_id</span></code> in the best neighbor table.</p></li>
<li><p>The blue circles and arrows represent the second <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation, which takes each <code class="docutils literal notranslate"><span class="pre">original_ext_source_id</span></code> in the Gaia table and finds the same value of <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> in the best neighbor table.</p></li>
</ul>
<p>There’s no guarantee that the corresponding rows of these tables are in the same order, so the <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation involves some searching.
However, ADQL/SQL databases are implemented in a way that makes this kind of source efficient.
If you are curious, you can <a class="reference external" href="https://chartio.com/learn/databases/how-does-indexing-work/">read more about it</a>.</p>
<img alt="https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/join.png" src="https://github.com/datacarpentry/astronomy-python/raw/gh-pages/fig/join.png" />
</div>
<div class="section" id="id1">
<h2>Joining tables<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<p>Now let’s get to the details of performing a <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operation.
As a starting place, let’s go all the way back to the cone search from Lesson 2.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query_cone</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="s2">TOP 10 </span>
<span class="s2">source_id</span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE 1=CONTAINS(</span>
<span class="s2"> POINT(ra, dec),</span>
<span class="s2"> CIRCLE(88.8, 7.4, 0.08333333))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>And let’s run it, to make sure we have a working query to build on.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">astroquery.gaia</span> <span class="kn">import</span> <span class="n">Gaia</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query_cone</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=10</i>
<table id="table140104380877264" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th></tr></thead>
<thead><tr><th>int64</th></tr></thead>
<tr><td>3322773965056065536</td></tr>
<tr><td>3322773758899157120</td></tr>
<tr><td>3322774068134271104</td></tr>
<tr><td>3322773930696320512</td></tr>
<tr><td>3322774377374425728</td></tr>
<tr><td>3322773724537891456</td></tr>
<tr><td>3322773724537891328</td></tr>
<tr><td>3322773930696321792</td></tr>
<tr><td>3322773724537890944</td></tr>
<tr><td>3322773930696322176</td></tr>
</table></div></div>
</div>
<p>Now we can start adding features.
First, let’s replace <code class="docutils literal notranslate"><span class="pre">source_id</span></code> with a format specifier, <code class="docutils literal notranslate"><span class="pre">columns</span></code>:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE 1=CONTAINS(</span>
<span class="s2"> POINT(ra, dec),</span>
<span class="s2"> CIRCLE(88.8, 7.4, 0.08333333))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>Here are the columns we want from the Gaia table, again.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;source_id, ra, dec, pmra, pmdec&#39;</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">query_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
source_id, ra, dec, pmra, pmdec
FROM gaiadr2.gaia_source
WHERE 1=CONTAINS(
POINT(ra, dec),
CIRCLE(88.8, 7.4, 0.08333333))
</pre></div>
</div>
</div>
</div>
<p>And let’s run the query again.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=594</i>
<table id="table140104380909264" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>3322773965056065536</td><td>88.78178020183375</td><td>7.334936530583141</td><td>0.2980633722108194</td><td>-2.5057036964736907</td></tr>
<tr><td>3322773758899157120</td><td>88.83227057144585</td><td>7.325577341429926</td><td>--</td><td>--</td></tr>
<tr><td>3322774068134271104</td><td>88.8206092188033</td><td>7.353158142762173</td><td>-1.1065462654445488</td><td>-1.5260889445858044</td></tr>
<tr><td>3322773930696320512</td><td>88.80843339290348</td><td>7.334853162299928</td><td>2.6074384482375215</td><td>-0.9292104395445717</td></tr>
<tr><td>3322774377374425728</td><td>88.86806108182265</td><td>7.371287731275939</td><td>3.9555477866915383</td><td>-3.8676624830902435</td></tr>
<tr><td>3322773724537891456</td><td>88.81308602813434</td><td>7.32488574492059</td><td>51.34995462741039</td><td>-33.078133430952086</td></tr>
<tr><td>3322773724537891328</td><td>88.81570329208743</td><td>7.3223019772324855</td><td>1.9389988498951845</td><td>0.3110526931576576</td></tr>
<tr><td>3322773930696321792</td><td>88.8050736770331</td><td>7.332371472206583</td><td>2.264014834476311</td><td>1.0772755505138008</td></tr>
<tr><td>3322773724537890944</td><td>88.81241651540533</td><td>7.327864052479726</td><td>-0.36003627434304625</td><td>-6.393939291541333</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>3322962118983356032</td><td>88.76109637722949</td><td>7.380564308268047</td><td>--</td><td>--</td></tr>
<tr><td>3322963527732585984</td><td>88.78813701704823</td><td>7.456696889759524</td><td>1.1363354614104264</td><td>-2.46251296961979</td></tr>
<tr><td>3322961775385969024</td><td>88.79723215862369</td><td>7.359756552906535</td><td>2.121021366548921</td><td>-6.605711792572964</td></tr>
<tr><td>3322962084625312512</td><td>88.78286756313868</td><td>7.384598632215225</td><td>-0.09350717810996487</td><td>1.3495903680571226</td></tr>
<tr><td>3322962939322692608</td><td>88.73289357818679</td><td>7.407688975612043</td><td>-0.11002934783569704</td><td>1.002126813991455</td></tr>
<tr><td>3322963768250760576</td><td>88.7592444035961</td><td>7.469624531882018</td><td>--</td><td>--</td></tr>
<tr><td>3322963459013111808</td><td>88.80348931842845</td><td>7.438699901204871</td><td>0.800833828337078</td><td>-3.3780655466364626</td></tr>
<tr><td>3322963355935626368</td><td>88.75528507586058</td><td>7.427795463027667</td><td>--</td><td>--</td></tr>
<tr><td>3322963287216149888</td><td>88.7658164932195</td><td>7.415726370886557</td><td>2.3743092647634034</td><td>-0.5046963243400879</td></tr>
<tr><td>3322962015904143872</td><td>88.74740822271643</td><td>7.387057037713974</td><td>-0.7201178533250112</td><td>0.5565841272341593</td></tr>
</table></div></div>
</div>
</div>
<div class="section" id="adding-the-best-neighbor-table">
<h2>Adding the best neighbor table<a class="headerlink" href="#adding-the-best-neighbor-table" title="Permalink to this headline"></a></h2>
<p>Now we’re ready for the first join.
The join operation requires two clauses:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">JOIN</span></code> specifies the name of the table we want to join with, and</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ON</span></code> specifies how we’ll match up rows between the tables.</p></li>
</ul>
<p>In this example, we join with <code class="docutils literal notranslate"><span class="pre">gaiadr2.panstarrs1_best_neighbour</span> <span class="pre">AS</span> <span class="pre">best</span></code>, which means we can refer to the best neighbor table with the abbreviated name <code class="docutils literal notranslate"><span class="pre">best</span></code>.</p>
<p>And the <code class="docutils literal notranslate"><span class="pre">ON</span></code> clause indicates that we’ll match up the <code class="docutils literal notranslate"><span class="pre">source_id</span></code> column from the Gaia table with the <code class="docutils literal notranslate"><span class="pre">source_id</span></code> column from the best neighbor table.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source AS gaia</span>
<span class="s2">JOIN gaiadr2.panstarrs1_best_neighbour AS best</span>
<span class="s2"> ON gaia.source_id = best.source_id</span>
<span class="s2">WHERE 1=CONTAINS(</span>
<span class="s2"> POINT(gaia.ra, gaia.dec),</span>
<span class="s2"> CIRCLE(88.8, 7.4, 0.08333333))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p><strong>SQL detail:</strong> In this example, the <code class="docutils literal notranslate"><span class="pre">ON</span></code> column has the same name in both tables, so we could replace the <code class="docutils literal notranslate"><span class="pre">ON</span></code> clause with a simpler <a class="reference external" href="https://docs.oracle.com/javadb/10.8.3.0/ref/rrefsqljusing.html"><code class="docutils literal notranslate"><span class="pre">USING</span></code> clause</a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">USING</span><span class="p">(</span><span class="n">source_id</span><span class="p">)</span>
</pre></div>
</div>
<p>Now that there’s more than one table involved, we can’t use simple column names any more; we have to use <strong>qualified column names</strong>.
In other words, we have to specify which table each column is in.
Here’s the complete query, including the columns we want from the Gaia and best neighbor tables.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">column_list</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;gaia.source_id&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.ra&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.dec&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.pmra&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.pmdec&#39;</span><span class="p">,</span>
<span class="s1">&#39;best.best_neighbour_multiplicity&#39;</span><span class="p">,</span>
<span class="s1">&#39;best.number_of_mates&#39;</span><span class="p">,</span>
<span class="p">]</span>
<span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">column_list</span><span class="p">)</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">query_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
gaia.source_id, gaia.ra, gaia.dec, gaia.pmra, gaia.pmdec, best.best_neighbour_multiplicity, best.number_of_mates
FROM gaiadr2.gaia_source AS gaia
JOIN gaiadr2.panstarrs1_best_neighbour AS best
ON gaia.source_id = best.source_id
WHERE 1=CONTAINS(
POINT(gaia.ra, gaia.dec),
CIRCLE(88.8, 7.4, 0.08333333))
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=490</i>
<table id="table140104380908352" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>best_neighbour_multiplicity</th><th>number_of_mates</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th></th><th></th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int16</th><th>int16</th></tr></thead>
<tr><td>3322773965056065536</td><td>88.78178020183375</td><td>7.334936530583141</td><td>0.2980633722108194</td><td>-2.5057036964736907</td><td>1</td><td>0</td></tr>
<tr><td>3322774068134271104</td><td>88.8206092188033</td><td>7.353158142762173</td><td>-1.1065462654445488</td><td>-1.5260889445858044</td><td>1</td><td>0</td></tr>
<tr><td>3322773930696320512</td><td>88.80843339290348</td><td>7.334853162299928</td><td>2.6074384482375215</td><td>-0.9292104395445717</td><td>1</td><td>0</td></tr>
<tr><td>3322774377374425728</td><td>88.86806108182265</td><td>7.371287731275939</td><td>3.9555477866915383</td><td>-3.8676624830902435</td><td>1</td><td>0</td></tr>
<tr><td>3322773724537891456</td><td>88.81308602813434</td><td>7.32488574492059</td><td>51.34995462741039</td><td>-33.078133430952086</td><td>1</td><td>0</td></tr>
<tr><td>3322773724537891328</td><td>88.81570329208743</td><td>7.3223019772324855</td><td>1.9389988498951845</td><td>0.3110526931576576</td><td>1</td><td>0</td></tr>
<tr><td>3322773930696321792</td><td>88.8050736770331</td><td>7.332371472206583</td><td>2.264014834476311</td><td>1.0772755505138008</td><td>1</td><td>0</td></tr>
<tr><td>3322773724537890944</td><td>88.81241651540533</td><td>7.327864052479726</td><td>-0.36003627434304625</td><td>-6.393939291541333</td><td>1</td><td>0</td></tr>
<tr><td>3322773930696322176</td><td>88.80128682574824</td><td>7.334292036448643</td><td>--</td><td>--</td><td>1</td><td>0</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>3322962359501481088</td><td>88.85037722908271</td><td>7.402162717053584</td><td>2.058216493648542</td><td>-2.249255322558584</td><td>1</td><td>0</td></tr>
<tr><td>3322962393861228544</td><td>88.82108234976155</td><td>7.4044425496203</td><td>-0.916760881643629</td><td>-1.1113319053861441</td><td>1</td><td>0</td></tr>
<tr><td>3322955831151254912</td><td>88.74620347799508</td><td>7.342728619145855</td><td>0.1559833902071379</td><td>-1.750598455959734</td><td>1</td><td>0</td></tr>
<tr><td>3322962118983356032</td><td>88.76109637722949</td><td>7.380564308268047</td><td>--</td><td>--</td><td>1</td><td>0</td></tr>
<tr><td>3322963527732585984</td><td>88.78813701704823</td><td>7.456696889759524</td><td>1.1363354614104264</td><td>-2.46251296961979</td><td>1</td><td>0</td></tr>
<tr><td>3322961775385969024</td><td>88.79723215862369</td><td>7.359756552906535</td><td>2.121021366548921</td><td>-6.605711792572964</td><td>1</td><td>0</td></tr>
<tr><td>3322962084625312512</td><td>88.78286756313868</td><td>7.384598632215225</td><td>-0.09350717810996487</td><td>1.3495903680571226</td><td>1</td><td>0</td></tr>
<tr><td>3322962939322692608</td><td>88.73289357818679</td><td>7.407688975612043</td><td>-0.11002934783569704</td><td>1.002126813991455</td><td>1</td><td>0</td></tr>
<tr><td>3322963459013111808</td><td>88.80348931842845</td><td>7.438699901204871</td><td>0.800833828337078</td><td>-3.3780655466364626</td><td>1</td><td>0</td></tr>
<tr><td>3322962015904143872</td><td>88.74740822271643</td><td>7.387057037713974</td><td>-0.7201178533250112</td><td>0.5565841272341593</td><td>1</td><td>0</td></tr>
</table></div></div>
</div>
<p>Notice that this result has fewer rows than the previous result.
That’s because there are sources in the Gaia table with no corresponding source in the Pan-STARRS table.</p>
<p>By default, the result of the join only includes rows where the same <code class="docutils literal notranslate"><span class="pre">source_id</span></code> appears in both tables.
This default is called an “inner” join because the results include only the intersection of the two tables.
<a class="reference external" href="https://www.geeksforgeeks.org/sql-join-set-1-inner-left-right-and-full-joins/">You can read about the other kinds of join here</a>.</p>
</div>
<div class="section" id="adding-the-pan-starrs-table">
<h2>Adding the Pan-STARRS table<a class="headerlink" href="#adding-the-pan-starrs-table" title="Permalink to this headline"></a></h2>
<div class="section" id="exercise">
<h3>Exercise<a class="headerlink" href="#exercise" title="Permalink to this headline"></a></h3>
<p>Now we’re ready to bring in the Pan-STARRS table. Starting with the previous query, add a second <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> clause that joins with <code class="docutils literal notranslate"><span class="pre">gaiadr2.panstarrs1_original_valid</span></code>, gives it the abbreviated name <code class="docutils literal notranslate"><span class="pre">ps</span></code>, and matches <code class="docutils literal notranslate"><span class="pre">original_ext_source_id</span></code> from the best neighbor table with <code class="docutils literal notranslate"><span class="pre">obj_id</span></code> from the Pan-STARRS table.</p>
<p>Add <code class="docutils literal notranslate"><span class="pre">g_mean_psf_mag</span></code> and <code class="docutils literal notranslate"><span class="pre">i_mean_psf_mag</span></code> to the column list, and run the query.
The result should contain 490 rows and 9 columns.</p>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">query_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source as gaia</span>
<span class="s2">JOIN gaiadr2.panstarrs1_best_neighbour as best</span>
<span class="s2"> ON gaia.source_id = best.source_id</span>
<span class="s2">JOIN gaiadr2.panstarrs1_original_valid as ps</span>
<span class="s2"> ON best.original_ext_source_id = ps.obj_id</span>
<span class="s2">WHERE 1=CONTAINS(</span>
<span class="s2"> POINT(gaia.ra, gaia.dec),</span>
<span class="s2"> CIRCLE(88.8, 7.4, 0.08333333))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">column_list</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;gaia.source_id&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.ra&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.dec&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.pmra&#39;</span><span class="p">,</span>
<span class="s1">&#39;gaia.pmdec&#39;</span><span class="p">,</span>
<span class="s1">&#39;best.best_neighbour_multiplicity&#39;</span><span class="p">,</span>
<span class="s1">&#39;best.number_of_mates&#39;</span><span class="p">,</span>
<span class="s1">&#39;ps.g_mean_psf_mag&#39;</span><span class="p">,</span>
<span class="s1">&#39;ps.i_mean_psf_mag&#39;</span><span class="p">]</span>
<span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">column_list</span><span class="p">)</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">query_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
gaia.source_id, gaia.ra, gaia.dec, gaia.pmra, gaia.pmdec, best.best_neighbour_multiplicity, best.number_of_mates, ps.g_mean_psf_mag, ps.i_mean_psf_mag
FROM gaiadr2.gaia_source as gaia
JOIN gaiadr2.panstarrs1_best_neighbour as best
ON gaia.source_id = best.source_id
JOIN gaiadr2.panstarrs1_original_valid as ps
ON best.original_ext_source_id = ps.obj_id
WHERE 1=CONTAINS(
POINT(gaia.ra, gaia.dec),
CIRCLE(88.8, 7.4, 0.08333333))
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=490</i>
<table id="table140104380910896" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>best_neighbour_multiplicity</th><th>number_of_mates</th><th>g_mean_psf_mag</th><th>i_mean_psf_mag</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th></th><th></th><th></th><th>mag</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int16</th><th>int16</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>3322773965056065536</td><td>88.78178020183375</td><td>7.334936530583141</td><td>0.2980633722108194</td><td>-2.5057036964736907</td><td>1</td><td>0</td><td>19.9431991577148</td><td>17.4221992492676</td></tr>
<tr><td>3322774068134271104</td><td>88.8206092188033</td><td>7.353158142762173</td><td>-1.1065462654445488</td><td>-1.5260889445858044</td><td>1</td><td>0</td><td>18.6212005615234</td><td>16.6007995605469</td></tr>
<tr><td>3322773930696320512</td><td>88.80843339290348</td><td>7.334853162299928</td><td>2.6074384482375215</td><td>-0.9292104395445717</td><td>1</td><td>0</td><td>--</td><td>20.2203998565674</td></tr>
<tr><td>3322774377374425728</td><td>88.86806108182265</td><td>7.371287731275939</td><td>3.9555477866915383</td><td>-3.8676624830902435</td><td>1</td><td>0</td><td>18.0676002502441</td><td>16.9762001037598</td></tr>
<tr><td>3322773724537891456</td><td>88.81308602813434</td><td>7.32488574492059</td><td>51.34995462741039</td><td>-33.078133430952086</td><td>1</td><td>0</td><td>20.1907005310059</td><td>17.8700008392334</td></tr>
<tr><td>3322773724537891328</td><td>88.81570329208743</td><td>7.3223019772324855</td><td>1.9389988498951845</td><td>0.3110526931576576</td><td>1</td><td>0</td><td>22.6308002471924</td><td>19.6004009246826</td></tr>
<tr><td>3322773930696321792</td><td>88.8050736770331</td><td>7.332371472206583</td><td>2.264014834476311</td><td>1.0772755505138008</td><td>1</td><td>0</td><td>21.2119998931885</td><td>18.3528003692627</td></tr>
<tr><td>3322773724537890944</td><td>88.81241651540533</td><td>7.327864052479726</td><td>-0.36003627434304625</td><td>-6.393939291541333</td><td>1</td><td>0</td><td>20.8094005584717</td><td>18.1343002319336</td></tr>
<tr><td>3322773930696322176</td><td>88.80128682574824</td><td>7.334292036448643</td><td>--</td><td>--</td><td>1</td><td>0</td><td>19.7306003570557</td><td>--</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>3322962359501481088</td><td>88.85037722908271</td><td>7.402162717053584</td><td>2.058216493648542</td><td>-2.249255322558584</td><td>1</td><td>0</td><td>17.4034996032715</td><td>15.9040002822876</td></tr>
<tr><td>3322962393861228544</td><td>88.82108234976155</td><td>7.4044425496203</td><td>-0.916760881643629</td><td>-1.1113319053861441</td><td>1</td><td>0</td><td>--</td><td>--</td></tr>
<tr><td>3322955831151254912</td><td>88.74620347799508</td><td>7.342728619145855</td><td>0.1559833902071379</td><td>-1.750598455959734</td><td>1</td><td>0</td><td>18.4960994720459</td><td>17.3892993927002</td></tr>
<tr><td>3322962118983356032</td><td>88.76109637722949</td><td>7.380564308268047</td><td>--</td><td>--</td><td>1</td><td>0</td><td>18.0643997192383</td><td>16.7395000457764</td></tr>
<tr><td>3322963527732585984</td><td>88.78813701704823</td><td>7.456696889759524</td><td>1.1363354614104264</td><td>-2.46251296961979</td><td>1</td><td>0</td><td>17.8034992218018</td><td>16.1214008331299</td></tr>
<tr><td>3322961775385969024</td><td>88.79723215862369</td><td>7.359756552906535</td><td>2.121021366548921</td><td>-6.605711792572964</td><td>1</td><td>0</td><td>18.2070007324219</td><td>15.9947996139526</td></tr>
<tr><td>3322962084625312512</td><td>88.78286756313868</td><td>7.384598632215225</td><td>-0.09350717810996487</td><td>1.3495903680571226</td><td>1</td><td>0</td><td>16.7978992462158</td><td>15.1180000305176</td></tr>
<tr><td>3322962939322692608</td><td>88.73289357818679</td><td>7.407688975612043</td><td>-0.11002934783569704</td><td>1.002126813991455</td><td>1</td><td>0</td><td>17.18630027771</td><td>16.3645992279053</td></tr>
<tr><td>3322963459013111808</td><td>88.80348931842845</td><td>7.438699901204871</td><td>0.800833828337078</td><td>-3.3780655466364626</td><td>1</td><td>0</td><td>--</td><td>16.294900894165</td></tr>
<tr><td>3322962015904143872</td><td>88.74740822271643</td><td>7.387057037713974</td><td>-0.7201178533250112</td><td>0.5565841272341593</td><td>1</td><td>0</td><td>18.4706993103027</td><td>16.8038005828857</td></tr>
</table></div></div>
</div>
</div>
</div>
<div class="section" id="selecting-by-coordinates-and-proper-motion">
<h2>Selecting by coordinates and proper motion<a class="headerlink" href="#selecting-by-coordinates-and-proper-motion" title="Permalink to this headline"></a></h2>
<p>Now let’s bring in the <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clause from the previous lesson, which selects sources based on parallax, BP-RP color, sky coordinates, and proper motion.</p>
<p>Here’s <code class="docutils literal notranslate"><span class="pre">query6_base</span></code> from the previous lesson.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query6_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; 1</span>
<span class="s2"> AND bp_rp BETWEEN -0.75 AND 2 </span>
<span class="s2"> AND 1 = CONTAINS(POINT(ra, dec), </span>
<span class="s2"> POLYGON(</span><span class="si">{point_list}</span><span class="s2">))</span>
<span class="s2"> AND 1 = CONTAINS(POINT(pmra, pmdec),</span>
<span class="s2"> POLYGON(</span><span class="si">{pm_point_list}</span><span class="s2">))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>Let’s reload the Pandas <code class="docutils literal notranslate"><span class="pre">Series</span></code> that contains <code class="docutils literal notranslate"><span class="pre">point_list</span></code> and <code class="docutils literal notranslate"><span class="pre">pm_point_list</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">&#39;gd1_data.hdf&#39;</span>
<span class="n">point_series</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_hdf</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;point_series&#39;</span><span class="p">)</span>
<span class="n">point_series</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>point_list 135.306, 8.39862, 126.51, 13.4449, 163.017, 54...
pm_point_list -4.05037121,-14.75623261, -3.41981085,-14.723...
dtype: object
</pre></div>
</div>
</div>
</div>
<p>Now we can assemble the query.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;source_id, ra, dec, pmra, pmdec&#39;</span>
<span class="n">query6</span> <span class="o">=</span> <span class="n">query6_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
<span class="n">point_list</span><span class="o">=</span><span class="n">point_series</span><span class="p">[</span><span class="s1">&#39;point_list&#39;</span><span class="p">],</span>
<span class="n">pm_point_list</span><span class="o">=</span><span class="n">point_series</span><span class="p">[</span><span class="s1">&#39;pm_point_list&#39;</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query6</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
source_id, ra, dec, pmra, pmdec
FROM gaiadr2.gaia_source
WHERE parallax &lt; 1
AND bp_rp BETWEEN -0.75 AND 2
AND 1 = CONTAINS(POINT(ra, dec),
POLYGON(135.306, 8.39862, 126.51, 13.4449, 163.017, 54.2424, 172.933, 46.4726, 135.306, 8.39862))
AND 1 = CONTAINS(POINT(pmra, pmdec),
POLYGON( -4.05037121,-14.75623261, -3.41981085,-14.72365546, -3.03521988,-14.44357135, -2.26847919,-13.7140236 , -2.61172203,-13.24797471, -2.73471401,-13.09054471, -3.19923146,-12.5942653 , -3.34082546,-12.47611926, -5.67489413,-11.16083338, -5.95159272,-11.10547884, -6.42394023,-11.05981295, -7.09631023,-11.95187806, -7.30641519,-12.24559977, -7.04016696,-12.88580702, -6.00347705,-13.75912098, -4.42442296,-14.74641176))
</pre></div>
</div>
</div>
</div>
<p>Again, let’s run it to make sure we are starting with a working query.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query6</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=7345</i>
<table id="table140104380909120" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>635559124339440000</td><td>137.58671691646745</td><td>19.1965441084838</td><td>-3.770521900009566</td><td>-12.490481778113859</td></tr>
<tr><td>635860218726658176</td><td>138.5187065217173</td><td>19.09233926905897</td><td>-5.941679495793577</td><td>-11.346409129876392</td></tr>
<tr><td>635674126383965568</td><td>138.8428741026386</td><td>19.031798198627634</td><td>-3.8970011609340207</td><td>-12.702779525389634</td></tr>
<tr><td>635535454774983040</td><td>137.8377518255436</td><td>18.864006786112604</td><td>-4.335040664412791</td><td>-14.492308604905652</td></tr>
<tr><td>635497276810313600</td><td>138.0445160213759</td><td>19.00947118796605</td><td>-7.1729306406216615</td><td>-12.291499169815987</td></tr>
<tr><td>635614168640132864</td><td>139.59219748145836</td><td>18.807955539071433</td><td>-3.309602916796381</td><td>-13.708904908478631</td></tr>
<tr><td>635821843194387840</td><td>139.88094034815086</td><td>19.62185456718988</td><td>-6.544201177153814</td><td>-12.55978220563274</td></tr>
<tr><td>635551706931167104</td><td>138.04665586038192</td><td>19.248909662830798</td><td>-6.224595114220405</td><td>-12.224246333795001</td></tr>
<tr><td>635518889086133376</td><td>137.2374229207837</td><td>18.7428630711791</td><td>-3.3186800714801046</td><td>-12.710314902969365</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>612282738058264960</td><td>134.0445768189235</td><td>18.11915820167003</td><td>-2.5972485319419127</td><td>-13.651740929272187</td></tr>
<tr><td>612485911486166656</td><td>134.96582769047063</td><td>19.309965857307247</td><td>-4.519325315774155</td><td>-11.998725329569156</td></tr>
<tr><td>612386332668697600</td><td>135.45701048323093</td><td>18.63266345155342</td><td>-5.07684899854408</td><td>-12.436641304786672</td></tr>
<tr><td>612296172717818624</td><td>133.80060286960668</td><td>18.08186533343457</td><td>-6.112792578821885</td><td>-12.50750861370402</td></tr>
<tr><td>612250375480101760</td><td>134.64754712466774</td><td>18.122419425065015</td><td>-2.8969262278467127</td><td>-14.061676353845487</td></tr>
<tr><td>612394926899159168</td><td>135.51997060013844</td><td>18.817675531233004</td><td>-3.9968965218753763</td><td>-13.526821099431533</td></tr>
<tr><td>612288854091187712</td><td>134.07970733489358</td><td>18.15424015818678</td><td>-5.96977151283562</td><td>-11.162471664228455</td></tr>
<tr><td>612428870024913152</td><td>134.8384242853297</td><td>18.758253070693225</td><td>-4.0022333299353825</td><td>-14.247379430659198</td></tr>
<tr><td>612256418500423168</td><td>134.90752972739924</td><td>18.280596648172743</td><td>-6.109836304219565</td><td>-12.145212331165776</td></tr>
<tr><td>612429144902815104</td><td>134.77293979509543</td><td>18.73628415871413</td><td>-5.257085979310591</td><td>-13.962312685889454</td></tr>
</table></div></div>
</div>
<div class="section" id="id2">
<h3>Exercise<a class="headerlink" href="#id2" title="Permalink to this headline"></a></h3>
<p>Create a new query base called <code class="docutils literal notranslate"><span class="pre">query7_base</span></code> that combines the <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clauses from the previous query with the <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> clauses for the best neighbor and Pan-STARRS tables.
Format the query base using the column names in <code class="docutils literal notranslate"><span class="pre">column_list</span></code>, and call the result <code class="docutils literal notranslate"><span class="pre">query7</span></code>.</p>
<p>Hint: Make sure you use qualified column names everywhere!</p>
<p>Run your query and download the results. The table you get should have 3725 rows and 9 columns.</p>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">query7_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="s2">SELECT </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source as gaia</span>
<span class="s2">JOIN gaiadr2.panstarrs1_best_neighbour as best</span>
<span class="s2"> ON gaia.source_id = best.source_id</span>
<span class="s2">JOIN gaiadr2.panstarrs1_original_valid as ps</span>
<span class="s2"> ON best.original_ext_source_id = ps.obj_id</span>
<span class="s2">WHERE parallax &lt; 1</span>
<span class="s2"> AND bp_rp BETWEEN -0.75 AND 2 </span>
<span class="s2"> AND 1 = CONTAINS(POINT(gaia.ra, gaia.dec), </span>
<span class="s2"> POLYGON(</span><span class="si">{point_list}</span><span class="s2">))</span>
<span class="s2"> AND 1 = CONTAINS(POINT(gaia.pmra, gaia.pmdec),</span>
<span class="s2"> POLYGON(</span><span class="si">{pm_point_list}</span><span class="s2">))</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">column_list</span><span class="p">)</span>
<span class="n">query7</span> <span class="o">=</span> <span class="n">query7_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
<span class="n">point_list</span><span class="o">=</span><span class="n">point_series</span><span class="p">[</span><span class="s1">&#39;point_list&#39;</span><span class="p">],</span>
<span class="n">pm_point_list</span><span class="o">=</span><span class="n">point_series</span><span class="p">[</span><span class="s1">&#39;pm_point_list&#39;</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query7</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
gaia.source_id, gaia.ra, gaia.dec, gaia.pmra, gaia.pmdec, best.best_neighbour_multiplicity, best.number_of_mates, ps.g_mean_psf_mag, ps.i_mean_psf_mag
FROM gaiadr2.gaia_source as gaia
JOIN gaiadr2.panstarrs1_best_neighbour as best
ON gaia.source_id = best.source_id
JOIN gaiadr2.panstarrs1_original_valid as ps
ON best.original_ext_source_id = ps.obj_id
WHERE parallax &lt; 1
AND bp_rp BETWEEN -0.75 AND 2
AND 1 = CONTAINS(POINT(gaia.ra, gaia.dec),
POLYGON(135.306, 8.39862, 126.51, 13.4449, 163.017, 54.2424, 172.933, 46.4726, 135.306, 8.39862))
AND 1 = CONTAINS(POINT(gaia.pmra, gaia.pmdec),
POLYGON( -4.05037121,-14.75623261, -3.41981085,-14.72365546, -3.03521988,-14.44357135, -2.26847919,-13.7140236 , -2.61172203,-13.24797471, -2.73471401,-13.09054471, -3.19923146,-12.5942653 , -3.34082546,-12.47611926, -5.67489413,-11.16083338, -5.95159272,-11.10547884, -6.42394023,-11.05981295, -7.09631023,-11.95187806, -7.30641519,-12.24559977, -7.04016696,-12.88580702, -6.00347705,-13.75912098, -4.42442296,-14.74641176))
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">query7</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=3725</i>
<table id="table140104308204640" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>best_neighbour_multiplicity</th><th>number_of_mates</th><th>g_mean_psf_mag</th><th>i_mean_psf_mag</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th></th><th></th><th></th><th>mag</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int16</th><th>int16</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>635860218726658176</td><td>138.5187065217173</td><td>19.09233926905897</td><td>-5.941679495793577</td><td>-11.346409129876392</td><td>1</td><td>0</td><td>17.8978004455566</td><td>17.5174007415771</td></tr>
<tr><td>635674126383965568</td><td>138.8428741026386</td><td>19.031798198627634</td><td>-3.8970011609340207</td><td>-12.702779525389634</td><td>1</td><td>0</td><td>19.2873001098633</td><td>17.6781005859375</td></tr>
<tr><td>635535454774983040</td><td>137.8377518255436</td><td>18.864006786112604</td><td>-4.335040664412791</td><td>-14.492308604905652</td><td>1</td><td>0</td><td>16.9237995147705</td><td>16.478099822998</td></tr>
<tr><td>635497276810313600</td><td>138.0445160213759</td><td>19.00947118796605</td><td>-7.1729306406216615</td><td>-12.291499169815987</td><td>1</td><td>0</td><td>19.9242000579834</td><td>18.3339996337891</td></tr>
<tr><td>635614168640132864</td><td>139.59219748145836</td><td>18.807955539071433</td><td>-3.309602916796381</td><td>-13.708904908478631</td><td>1</td><td>0</td><td>16.1515998840332</td><td>14.6662998199463</td></tr>
<tr><td>635598607974369792</td><td>139.20920023089508</td><td>18.624132868942702</td><td>-6.124445176881091</td><td>-12.833824027100611</td><td>1</td><td>0</td><td>16.5223999023438</td><td>16.1375007629395</td></tr>
<tr><td>635737661835496576</td><td>139.93327552473934</td><td>19.167962454651423</td><td>-7.119403303682826</td><td>-12.687947497633793</td><td>1</td><td>0</td><td>14.5032997131348</td><td>13.9849004745483</td></tr>
<tr><td>635850945892748672</td><td>139.86542888472115</td><td>20.011312663154804</td><td>-3.786655365804428</td><td>-14.28415600718206</td><td>1</td><td>0</td><td>16.5174999237061</td><td>16.0450000762939</td></tr>
<tr><td>635600532119713664</td><td>139.22869949616816</td><td>18.685939084485494</td><td>-3.9742788217925122</td><td>-12.342426623384245</td><td>1</td><td>0</td><td>20.4505996704102</td><td>19.5177001953125</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>612241781249124608</td><td>134.3755835065194</td><td>18.129179169751275</td><td>-2.831807894848964</td><td>-13.902118573613597</td><td>1</td><td>0</td><td>20.2343997955322</td><td>18.6518001556396</td></tr>
<tr><td>612332147361443072</td><td>134.14584721363653</td><td>18.45685585044513</td><td>-6.234287981021865</td><td>-11.500464195695072</td><td>1</td><td>0</td><td>21.3848991394043</td><td>20.3076000213623</td></tr>
<tr><td>612426744016802432</td><td>134.68522805061076</td><td>18.77090626983678</td><td>-3.7691372464459554</td><td>-12.889167493118862</td><td>1</td><td>0</td><td>17.8281002044678</td><td>17.4281005859375</td></tr>
<tr><td>612331739340341760</td><td>134.12176196902254</td><td>18.42768872157865</td><td>-3.9894012386388735</td><td>-12.60504410507441</td><td>1</td><td>0</td><td>21.8656997680664</td><td>19.5223007202148</td></tr>
<tr><td>612282738058264960</td><td>134.0445768189235</td><td>18.11915820167003</td><td>-2.5972485319419127</td><td>-13.651740929272187</td><td>1</td><td>0</td><td>22.5151996612549</td><td>19.9743995666504</td></tr>
<tr><td>612386332668697600</td><td>135.45701048323093</td><td>18.63266345155342</td><td>-5.07684899854408</td><td>-12.436641304786672</td><td>1</td><td>0</td><td>19.3792991638184</td><td>17.9923000335693</td></tr>
<tr><td>612296172717818624</td><td>133.80060286960668</td><td>18.08186533343457</td><td>-6.112792578821885</td><td>-12.50750861370402</td><td>1</td><td>0</td><td>17.4944000244141</td><td>16.926700592041</td></tr>
<tr><td>612250375480101760</td><td>134.64754712466774</td><td>18.122419425065015</td><td>-2.8969262278467127</td><td>-14.061676353845487</td><td>1</td><td>0</td><td>15.3330001831055</td><td>14.6280002593994</td></tr>
<tr><td>612394926899159168</td><td>135.51997060013844</td><td>18.817675531233004</td><td>-3.9968965218753763</td><td>-13.526821099431533</td><td>1</td><td>0</td><td>16.4414005279541</td><td>15.8212003707886</td></tr>
<tr><td>612256418500423168</td><td>134.90752972739924</td><td>18.280596648172743</td><td>-6.109836304219565</td><td>-12.145212331165776</td><td>1</td><td>0</td><td>20.8715991973877</td><td>19.9612007141113</td></tr>
</table></div></div>
</div>
</div>
</div>
<div class="section" id="checking-the-match">
<h2>Checking the match<a class="headerlink" href="#checking-the-match" title="Permalink to this headline"></a></h2>
<p>To get more information about the matching process, we can inspect <code class="docutils literal notranslate"><span class="pre">best_neighbour_multiplicity</span></code>, which indicates for each star in Gaia how many stars in Pan-STARRS are equally likely matches.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span><span class="p">[</span><span class="s1">&#39;best_neighbour_multiplicity&#39;</span><span class="p">]</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html">&lt;MaskedColumn name=&apos;best_neighbour_multiplicity&apos; dtype=&apos;int16&apos; description=&apos;Number of neighbours with same probability as best neighbour&apos; length=3725&gt;
<table>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>...</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
<tr><td>1</td></tr>
</table></div></div>
</div>
<p>It looks like most of the values are <code class="docutils literal notranslate"><span class="pre">1</span></code>, which is good; that means that for each candidate star we have identified exactly one source in Pan-STARRS that is likely to be the same star.</p>
<p>To check whether there are any values other than <code class="docutils literal notranslate"><span class="pre">1</span></code>, we can convert this column to a Pandas <code class="docutils literal notranslate"><span class="pre">Series</span></code> and use <code class="docutils literal notranslate"><span class="pre">describe</span></code>, which we saw in in Lesson 3.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">multiplicity</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">results</span><span class="p">[</span><span class="s1">&#39;best_neighbour_multiplicity&#39;</span><span class="p">])</span>
<span class="n">multiplicity</span><span class="o">.</span><span class="n">describe</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>count 3725.0
mean 1.0
std 0.0
min 1.0
25% 1.0
50% 1.0
75% 1.0
max 1.0
dtype: float64
</pre></div>
</div>
</div>
</div>
<p>In fact, <code class="docutils literal notranslate"><span class="pre">1</span></code> is the only value in the <code class="docutils literal notranslate"><span class="pre">Series</span></code>, so every candidate star has a single best match.</p>
<p>Similarly, <code class="docutils literal notranslate"><span class="pre">number_of_mates</span></code> indicates the number of <em>other</em> stars in Gaia that match with the same star in Pan-STARRS.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">mates</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">results</span><span class="p">[</span><span class="s1">&#39;number_of_mates&#39;</span><span class="p">])</span>
<span class="n">mates</span><span class="o">.</span><span class="n">describe</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>count 3725.0
mean 0.0
std 0.0
min 0.0
25% 0.0
50% 0.0
75% 0.0
max 0.0
dtype: float64
</pre></div>
</div>
</div>
</div>
<p>All values in this column are <code class="docutils literal notranslate"><span class="pre">0</span></code>, which means that for each match we found in Pan-STARRS, there are no other stars in Gaia that also match.</p>
<p><strong>Detail:</strong> The table also contains <code class="docutils literal notranslate"><span class="pre">number_of_neighbors</span></code> which is the number of stars in Pan-STARRS that match in terms of position, before using other criteria to choose the most likely match. But we are more interested in the final match, using both criteria.</p>
</div>
<div class="section" id="transforming-coordinates">
<h2>Transforming coordinates<a class="headerlink" href="#transforming-coordinates" title="Permalink to this headline"></a></h2>
<p>Here’s the function we’ve used to transform the results from ICRS to GD-1 coordinates.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">astropy.units</span> <span class="k">as</span> <span class="nn">u</span>
<span class="kn">from</span> <span class="nn">astropy.coordinates</span> <span class="kn">import</span> <span class="n">SkyCoord</span>
<span class="kn">from</span> <span class="nn">gala.coordinates</span> <span class="kn">import</span> <span class="n">GD1Koposov10</span>
<span class="kn">from</span> <span class="nn">gala.coordinates</span> <span class="kn">import</span> <span class="n">reflex_correct</span>
<span class="k">def</span> <span class="nf">make_dataframe</span><span class="p">(</span><span class="n">table</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;Transform coordinates from ICRS to GD-1 frame.</span>
<span class="sd"> </span>
<span class="sd"> table: Astropy Table</span>
<span class="sd"> </span>
<span class="sd"> returns: Pandas DataFrame</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">skycoord</span> <span class="o">=</span> <span class="n">SkyCoord</span><span class="p">(</span>
<span class="n">ra</span><span class="o">=</span><span class="n">table</span><span class="p">[</span><span class="s1">&#39;ra&#39;</span><span class="p">],</span>
<span class="n">dec</span><span class="o">=</span><span class="n">table</span><span class="p">[</span><span class="s1">&#39;dec&#39;</span><span class="p">],</span>
<span class="n">pm_ra_cosdec</span><span class="o">=</span><span class="n">table</span><span class="p">[</span><span class="s1">&#39;pmra&#39;</span><span class="p">],</span>
<span class="n">pm_dec</span><span class="o">=</span><span class="n">table</span><span class="p">[</span><span class="s1">&#39;pmdec&#39;</span><span class="p">],</span>
<span class="n">distance</span><span class="o">=</span><span class="mi">8</span><span class="o">*</span><span class="n">u</span><span class="o">.</span><span class="n">kpc</span><span class="p">,</span>
<span class="n">radial_velocity</span><span class="o">=</span><span class="mi">0</span><span class="o">*</span><span class="n">u</span><span class="o">.</span><span class="n">km</span><span class="o">/</span><span class="n">u</span><span class="o">.</span><span class="n">s</span><span class="p">)</span>
<span class="n">gd1_frame</span> <span class="o">=</span> <span class="n">GD1Koposov10</span><span class="p">()</span>
<span class="n">transformed</span> <span class="o">=</span> <span class="n">skycoord</span><span class="o">.</span><span class="n">transform_to</span><span class="p">(</span><span class="n">gd1_frame</span><span class="p">)</span>
<span class="n">skycoord_gd1</span> <span class="o">=</span> <span class="n">reflex_correct</span><span class="p">(</span><span class="n">transformed</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
<span class="n">df</span><span class="p">[</span><span class="s1">&#39;phi1&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">skycoord_gd1</span><span class="o">.</span><span class="n">phi1</span>
<span class="n">df</span><span class="p">[</span><span class="s1">&#39;phi2&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">skycoord_gd1</span><span class="o">.</span><span class="n">phi2</span>
<span class="n">df</span><span class="p">[</span><span class="s1">&#39;pm_phi1&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">skycoord_gd1</span><span class="o">.</span><span class="n">pm_phi1_cosphi2</span>
<span class="n">df</span><span class="p">[</span><span class="s1">&#39;pm_phi2&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">skycoord_gd1</span><span class="o">.</span><span class="n">pm_phi2</span>
<span class="k">return</span> <span class="n">df</span>
</pre></div>
</div>
</div>
</div>
<p>Now can transform the result from the last query.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">candidate_df</span> <span class="o">=</span> <span class="n">make_dataframe</span><span class="p">(</span><span class="n">results</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>And see how it looks.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">candidate_df</span><span class="p">[</span><span class="s1">&#39;phi1&#39;</span><span class="p">]</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">candidate_df</span><span class="p">[</span><span class="s1">&#39;phi2&#39;</span><span class="p">]</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s1">&#39;ko&#39;</span><span class="p">,</span> <span class="n">markersize</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;phi1 (degree GD1)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;phi2 (degree GD1)&#39;</span><span class="p">);</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/05_join_78_0.png" src="_images/05_join_78_0.png" />
</div>
</div>
<p>The result is similar to what we saw in the previous lesson, except that have fewer stars now, because we did not find photometry data for all of the candidate sources.</p>
</div>
<div class="section" id="saving-the-dataframe">
<h2>Saving the DataFrame<a class="headerlink" href="#saving-the-dataframe" title="Permalink to this headline"></a></h2>
<p>Let’s save this <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> so we can pick up where we left off without running this query again.
The HDF file should already exist, so we’ll add <code class="docutils literal notranslate"><span class="pre">candidate_df</span></code> to it.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">filename</span> <span class="o">=</span> <span class="s1">&#39;gd1_data.hdf&#39;</span>
<span class="n">candidate_df</span><span class="o">.</span><span class="n">to_hdf</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;candidate_df&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>We can use <code class="docutils literal notranslate"><span class="pre">getsize</span></code> to confirm that the file exists and check the size:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">os.path</span> <span class="kn">import</span> <span class="n">getsize</span>
<span class="n">MB</span> <span class="o">=</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span>
<span class="n">getsize</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="o">/</span> <span class="n">MB</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>5.574869155883789
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="summary">
<h2>Summary<a class="headerlink" href="#summary" title="Permalink to this headline"></a></h2>
<p>In this notebook, we used database <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operations to select photometry data for the stars we’ve identified as candidates to be in GD-1.</p>
<p>In the next notebook, we’ll use this data for a second round of selection, identifying stars that have photometry data consistent with GD-1.</p>
<p>But before you go on, you might be interested in another file format, CSV.</p>
</div>
<div class="section" id="csv">
<h2>CSV<a class="headerlink" href="#csv" title="Permalink to this headline"></a></h2>
<p>Pandas can write a variety of other formats, <a class="reference external" href="https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html">which you can read about here</a>.
We won’t cover all of them, but one other important one is <a class="reference external" href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a>, which stands for “comma-separated values”.</p>
<p>CSV is a plain-text format that can be read and written by pretty much any tool that works with data. In that sense, it is the “least common denominator” of data formats.</p>
<p>However, it has an important limitation: some information about the data gets lost in translation, notably the data types. If you read a CSV file from someone else, you might need some additional information to make sure you are getting it right.</p>
<p>Also, CSV files tend to be big, and slow to read and write.</p>
<p>With those caveats, here’s how to write one:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">candidate_df</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s1">&#39;gd1_data.csv&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>We can check the file size like this:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">getsize</span><span class="p">(</span><span class="s1">&#39;gd1_data.csv&#39;</span><span class="p">)</span> <span class="o">/</span> <span class="n">MB</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.7606849670410156
</pre></div>
</div>
</div>
</div>
<p>We can see the first few lines like this:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">head</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;Print the first `n` lines of a file.&quot;&quot;&quot;</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">next</span><span class="p">(</span><span class="n">fp</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">head</span><span class="p">(</span><span class="s1">&#39;gd1_data.csv&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>,source_id,ra,dec,pmra,pmdec,best_neighbour_multiplicity,number_of_mates,g_mean_psf_mag,i_mean_psf_mag,phi1,phi2,pm_phi1,pm_phi2
0,635860218726658176,138.5187065217173,19.09233926905897,-5.941679495793577,-11.346409129876392,1,0,17.8978004455566,17.5174007415771,-59.247329893833296,-2.016078400820631,-7.527126084640531,1.7487794924176672
1,635674126383965568,138.8428741026386,19.031798198627634,-3.8970011609340207,-12.702779525389634,1,0,19.2873001098633,17.6781005859375,-59.13339098769217,-2.306900745179831,-7.560607655557415,-0.7417999555980248
</pre></div>
</div>
</div>
</div>
<p>The CSV file contains the names of the columns, but not the data types.</p>
<p>We can read the CSV file back like this:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">read_back_csv</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s1">&#39;gd1_data.csv&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Let’s compare the first few rows of <code class="docutils literal notranslate"><span class="pre">candidate_df</span></code> and <code class="docutils literal notranslate"><span class="pre">read_back_csv</span></code></p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">candidate_df</span><span class="o">.</span><span class="n">head</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>source_id</th>
<th>ra</th>
<th>dec</th>
<th>pmra</th>
<th>pmdec</th>
<th>best_neighbour_multiplicity</th>
<th>number_of_mates</th>
<th>g_mean_psf_mag</th>
<th>i_mean_psf_mag</th>
<th>phi1</th>
<th>phi2</th>
<th>pm_phi1</th>
<th>pm_phi2</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>635860218726658176</td>
<td>138.518707</td>
<td>19.092339</td>
<td>-5.941679</td>
<td>-11.346409</td>
<td>1</td>
<td>0</td>
<td>17.8978</td>
<td>17.517401</td>
<td>-59.247330</td>
<td>-2.016078</td>
<td>-7.527126</td>
<td>1.748779</td>
</tr>
<tr>
<th>1</th>
<td>635674126383965568</td>
<td>138.842874</td>
<td>19.031798</td>
<td>-3.897001</td>
<td>-12.702780</td>
<td>1</td>
<td>0</td>
<td>19.2873</td>
<td>17.678101</td>
<td>-59.133391</td>
<td>-2.306901</td>
<td>-7.560608</td>
<td>-0.741800</td>
</tr>
<tr>
<th>2</th>
<td>635535454774983040</td>
<td>137.837752</td>
<td>18.864007</td>
<td>-4.335041</td>
<td>-14.492309</td>
<td>1</td>
<td>0</td>
<td>16.9238</td>
<td>16.478100</td>
<td>-59.785300</td>
<td>-1.594569</td>
<td>-9.357536</td>
<td>-1.218492</td>
</tr>
</tbody>
</table>
</div></div></div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">read_back_csv</span><span class="o">.</span><span class="n">head</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>Unnamed: 0</th>
<th>source_id</th>
<th>ra</th>
<th>dec</th>
<th>pmra</th>
<th>pmdec</th>
<th>best_neighbour_multiplicity</th>
<th>number_of_mates</th>
<th>g_mean_psf_mag</th>
<th>i_mean_psf_mag</th>
<th>phi1</th>
<th>phi2</th>
<th>pm_phi1</th>
<th>pm_phi2</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0</td>
<td>635860218726658176</td>
<td>138.518707</td>
<td>19.092339</td>
<td>-5.941679</td>
<td>-11.346409</td>
<td>1</td>
<td>0</td>
<td>17.8978</td>
<td>17.517401</td>
<td>-59.247330</td>
<td>-2.016078</td>
<td>-7.527126</td>
<td>1.748779</td>
</tr>
<tr>
<th>1</th>
<td>1</td>
<td>635674126383965568</td>
<td>138.842874</td>
<td>19.031798</td>
<td>-3.897001</td>
<td>-12.702780</td>
<td>1</td>
<td>0</td>
<td>19.2873</td>
<td>17.678101</td>
<td>-59.133391</td>
<td>-2.306901</td>
<td>-7.560608</td>
<td>-0.741800</td>
</tr>
<tr>
<th>2</th>
<td>2</td>
<td>635535454774983040</td>
<td>137.837752</td>
<td>18.864007</td>
<td>-4.335041</td>
<td>-14.492309</td>
<td>1</td>
<td>0</td>
<td>16.9238</td>
<td>16.478100</td>
<td>-59.785300</td>
<td>-1.594569</td>
<td>-9.357536</td>
<td>-1.218492</td>
</tr>
</tbody>
</table>
</div></div></div>
</div>
<p>Notice that the index in <code class="docutils literal notranslate"><span class="pre">candidate_df</span></code> has become an unnamed column in <code class="docutils literal notranslate"><span class="pre">read_back_csv</span></code>. The Pandas functions for writing and reading CSV files provide options to avoid that problem, but this is an example of the kind of thing that can go wrong with CSV files.</p>
</div>
<div class="section" id="best-practices">
<h2>Best practices<a class="headerlink" href="#best-practices" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>Use <code class="docutils literal notranslate"><span class="pre">JOIN</span></code> operations to combine data from multiple tables in a databased, using some kind of identifier to match up records from one table with records from another.</p></li>
<li><p>This is another example of a practice we saw in the previous notebook, moving the computation to the data.</p></li>
<li><p>For most applications, saving data in FITS or HDF5 is better than CSV. FITS and HDF5 are binary formats, so the files are usually smaller, and they store metadata, so you don’t lose anything when you read the file back.</p></li>
<li><p>On the other hand, CSV is a “least common denominator” format; that is, it can be read by practically any application that works with data.</p></li>
</ul>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
</div>
</div>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="04_select.html" title="previous page">4. Transformation and Selection</a>
<a class='right-next' id="next-link" href="06_photo.html" title="next page">6. Photometry</a>
</div>
<footer class="footer mt-5 mt-md-0">
<div class="container">
<p>
By Allen B. Downey<br/>
&copy; Copyright 2020.<br/>
</p>
</div>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.30270b6e4c972e43c488.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink