gitea-mirror/AstronomicalData
1
0
Fork 0
mirror of https://github.com/AllenDowney/AstronomicalData.git synced 2025-12-24 12:04:01 -08:00
Code Issues Packages Projects Releases Wiki Activity
Files
f1a11932eaf282be46a5ed7db1ab697a6ddaecfd
AstronomicalData/01_query.html
Allen Downey f4982019a3 Update documentation
2020-11-13 19:31:58 -05:00

1345 lines
77 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Chapter 1 &#8212; Astronomical Data in Python</title>
<link rel="stylesheet" href="_static/css/index.d431a4ee1c1efae0e38bdfebc22debff.css">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet"
href="_static/vendor/open-sans_all/1.44.1/index.css">
<link rel="stylesheet"
href="_static/vendor/lato_latin-ext/1.44.1/index.css">
<link rel="stylesheet" href="_static/sphinx-book-theme.bfb7730f9caf2ec0b46a44615585038c.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.30270b6e4c972e43c488.js">
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/language_data.js"></script>
<script src="_static/togglebutton.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script >var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.be0a4a0c39cd630af62a2fcf693f3f06.js"></script>
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["\\(", "\\)"]], "displayMath": [["\\[", "\\]"]], "processRefs": false, "processEnvironments": false}})</script>
<script async="async" src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
<script >
const thebe_selector = ".thebe"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Chapter 2" href="02_coords.html" />
<link rel="prev" title="Astronomical Data in Python" href="README.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<h1 class="site-logo" id="site-title">Astronomical Data in Python</h1>
</a>
</div>
<form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form>
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<ul class="nav sidenav_l1">
<li class="toctree-l1">
<a class="reference internal" href="README.html">
Astronomical Data in Python
</a>
</li>
</ul>
<ul class="current nav sidenav_l1">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Chapter 1
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="02_coords.html">
Chapter 2
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="03_motion.html">
Chapter 3
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="04_select.html">
Chapter 4
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="05_join.html">
Chapter 5
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="06_photo.html">
Chapter 6
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="07_plot.html">
Chapter 7
</a>
</li>
</ul>
</nav>
<!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="row topbar fixed-top container-xl">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show">
</div>
<div class="col pl-2 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/01_query.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onClick="window.print()" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn"
aria-label="Connect with source repository"><i class="fab fa-github"></i></button>
<div class="dropdown-buttons sourcebuttons">
<a class="repository-button"
href="https://github.com/AllenDowney/AstronomyDataPython"><button type="button" class="btn btn-secondary topbarbtn"
data-toggle="tooltip" data-placement="left" title="Source repository"><i
class="fab fa-github"></i>repository</button></a>
</div>
</div>
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn"
aria-label="Launch interactive content"><i class="fas fa-rocket"></i></button>
<div class="dropdown-buttons">
<a class="binder-button" href="https://mybinder.org/v2/gh/AllenDowney/AstronomyDataPython/master?urlpath=tree/01_query.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Launch Binder" data-toggle="tooltip"
data-placement="left"><img class="binder-button-logo"
src="_static/images/logo_binder.svg"
alt="Interact on binder">Binder</button></a>
<a class="colab-button" href="https://colab.research.google.com/github/AllenDowney/AstronomyDataPython/blob/master/01_query.ipynb"><button type="button" class="btn btn-secondary topbarbtn"
title="Launch Colab" data-toggle="tooltip" data-placement="left"><img class="colab-button-logo"
src="_static/images/logo_colab.png"
alt="Interact on Colab">Colab</button></a>
</div>
</div>
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav">
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#data">
Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#prerequisites">
Prerequisites
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#outline">
Outline
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#query-language">
Query Language
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#connecting-to-gaia">
Connecting to Gaia
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#databases-and-tables">
Databases and Tables
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#columns">
Columns
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#writing-queries">
Writing queries
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#asynchronous-queries">
Asynchronous queries
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#operators">
Operators
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cleaning-up">
Cleaning up
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#formatting-queries">
Formatting queries
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#summary">
Summary
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#best-practices">
Best practices
</a>
</li>
</ul>
</nav>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<div>
<div class="section" id="chapter-1">
<h1>Chapter 1<a class="headerlink" href="#chapter-1" title="Permalink to this headline"></a></h1>
<p><em>Astronomical Data in Python</em> is an introduction to tools and practices for working with astronomical data. Topics covered include:</p>
<ul class="simple">
<li><p>Writing queries that select and download data from a database.</p></li>
<li><p>Using data stored in an Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> or Pandas <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code>.</p></li>
<li><p>Working with coordinates and other quantities with units.</p></li>
<li><p>Storing data in various formats.</p></li>
<li><p>Performing database join operations that combine data from multiple tables.</p></li>
<li><p>Visualizing data and preparing publication-quality figures.</p></li>
</ul>
<p>As a running example, we will replicate part of the analysis in a recent paper, “<a class="reference external" href="https://arxiv.org/abs/1805.00425">Off the beaten path: Gaia reveals GD-1 stars outside of the main stream</a>” by Adrian M. Price-Whelan and Ana Bonaca.</p>
<p>As the abstract explains, “Using data from the Gaia second data release combined with Pan-STARRS photometry, we present a sample of highly-probable members of the longest cold stream in the Milky Way, GD-1.”</p>
<p>GD-1 is a <a class="reference external" href="https://en.wikipedia.org/wiki/List_of_stellar_streams">stellar stream</a>, which is “an association of stars orbiting a galaxy that was once a globular cluster or dwarf galaxy that has now been torn apart and stretched out along its orbit by tidal forces.”</p>
<p><a class="reference external" href="https://www.sciencemag.org/news/2018/10/streams-stars-reveal-galaxy-s-violent-history-and-perhaps-its-unseen-dark-matter">This article in <em>Science</em> magazine</a> explains some of the background, including the process that led to the paper and a discussion of the scientific implications:</p>
<ul class="simple">
<li><p>“The streams are particularly useful for … galactic archaeology — rewinding the cosmic clock to reconstruct the assembly of the Milky Way.”</p></li>
<li><p>“They also are being used as exquisitely sensitive scales to measure the galaxy’s mass.”</p></li>
<li><p>“… the streams are well-positioned to reveal the presence of dark matter … because the streams are so fragile, theorists say, collisions with marauding clumps of dark matter could leave telltale scars, potential clues to its nature.”</p></li>
</ul>
<div class="section" id="data">
<h2>Data<a class="headerlink" href="#data" title="Permalink to this headline"></a></h2>
<p>The datasets we will work with are:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://en.wikipedia.org/wiki/Gaia_(spacecraft)">Gaia</a>, which is “a space observatory of the European Space Agency (ESA), launched in 2013 … designed for astrometry: measuring the positions, distances and motions of stars with unprecedented precision”, and</p></li>
<li><p><a class="reference external" href="https://en.wikipedia.org/wiki/Pan-STARRS">Pan-STARRS</a>, The Panoramic Survey Telescope and Rapid Response System, which is a survey designed to monitor the sky for transient objects, producing a catalog with accurate astronometry and photometry of detected sources.</p></li>
</ul>
<p>Both of these datasets are very large, which can make them challenging to work with. It might not be possible, or practical, to download the entire dataset.
One of the goals of this workshop is to provide tools for working with large datasets.</p>
</div>
<div class="section" id="prerequisites">
<h2>Prerequisites<a class="headerlink" href="#prerequisites" title="Permalink to this headline"></a></h2>
<p>These notebooks are meant for people who are familiar with basic Python, but not necessarily the libraries we will use, like Astropy or Pandas. If you are familiar with Python lists and dictionaries, and you know how to write a function that takes parameters and returns a value, you know enough Python to get started.</p>
<p>We assume that you have some familiarity with operating systems, like the ability to use a command-line interface. But we don’t assume you have any prior experience with databases.</p>
<p>We assume that you are familiar with astronomy at the undergraduate level, but we will not assume specialized knowledge of the datasets or analysis methods we’ll use.</p>
</div>
<div class="section" id="outline">
<h2>Outline<a class="headerlink" href="#outline" title="Permalink to this headline"></a></h2>
<p>The first lesson demonstrates the steps for selecting and downloading data from the Gaia Database:</p>
<ol class="simple">
<li><p>First we’ll make a connection to the Gaia server,</p></li>
<li><p>We will explore information about the database and the tables it contains,</p></li>
<li><p>We will write a query and send it to the server, and finally</p></li>
<li><p>We will download the response from the server.</p></li>
</ol>
<p>After completing this lesson, you should be able to</p>
<ul class="simple">
<li><p>Compose a basic query in ADQL.</p></li>
<li><p>Use queries to explore a database and its tables.</p></li>
<li><p>Use queries to download data.</p></li>
<li><p>Develop, test, and debug a query incrementally.</p></li>
</ul>
</div>
<div class="section" id="query-language">
<h2>Query Language<a class="headerlink" href="#query-language" title="Permalink to this headline"></a></h2>
<p>In order to select data from a database, you have to compose a query, which is a program written in a “query language”.
The query language we’ll use is ADQL, which stands for “Astronomical Data Query Language”.</p>
<p>ADQL is a dialect of <a class="reference external" href="https://en.wikipedia.org/wiki/SQL">SQL</a> (Structured Query Language), which is by far the most commonly used query language. Almost everything you will learn about ADQL also works in SQL.</p>
<p><a class="reference external" href="http://www.ivoa.net/documents/ADQL/20180112/PR-ADQL-2.1-20180112.html">The reference manual for ADQL is here</a>.
But you might find it easier to learn from <a class="reference external" href="https://www.gaia.ac.uk/data/gaia-data-release-1/adql-cookbook">this ADQL Cookbook</a>.</p>
</div>
<div class="section" id="connecting-to-gaia">
<h2>Connecting to Gaia<a class="headerlink" href="#connecting-to-gaia" title="Permalink to this headline"></a></h2>
<p>The library we’ll use to get Gaia data is <a class="reference external" href="https://astroquery.readthedocs.io/en/latest/">Astroquery</a>.</p>
<p>Astroquery provides <code class="docutils literal notranslate"><span class="pre">Gaia</span></code>, which is an <a class="reference external" href="https://astroquery.readthedocs.io/en/latest/gaia/gaia.html">object that represents a connection to the Gaia database</a>.</p>
<p>We can connect to the Gaia database like this:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">astroquery.gaia</span> <span class="kn">import</span> <span class="n">Gaia</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Created TAP+ (v1.2.1) - Connection:
Host: gea.esac.esa.int
Use HTTPS: True
Port: 443
SSL Port: 443
Created TAP+ (v1.2.1) - Connection:
Host: geadata.esac.esa.int
Use HTTPS: True
Port: 443
SSL Port: 443
</pre></div>
</div>
</div>
</div>
<p>Running this import statement has the effect of creating a <a class="reference external" href="http://www.ivoa.net/documents/TAP/">TAP+</a> connection; TAP stands for “Table Access Protocol”. It is a network protocol for sending queries to the database and getting back the results. We’re not sure why it seems to create two connections.</p>
</div>
<div class="section" id="databases-and-tables">
<h2>Databases and Tables<a class="headerlink" href="#databases-and-tables" title="Permalink to this headline"></a></h2>
<p>What is a database, anyway? Most generally, it can be any collection of data, but when we are talking about ADQL or SQL:</p>
<ul class="simple">
<li><p>A database is a collection of one or more named tables.</p></li>
<li><p>Each table is a 2-D array with one or more named columns of data.</p></li>
</ul>
<p>We can use <code class="docutils literal notranslate"><span class="pre">Gaia.load_tables</span></code> to get the names of the tables in the Gaia database. With the option <code class="docutils literal notranslate"><span class="pre">only_names=True</span></code>, it loads information about the tables, called the “metadata”, not the data itself.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">tables</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_tables</span><span class="p">(</span><span class="n">only_names</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Retrieving tables... [astroquery.utils.tap.core]
INFO: Parsing tables... [astroquery.utils.tap.core]
INFO: Done. [astroquery.utils.tap.core]
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-output docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">table</span> <span class="ow">in</span> <span class="p">(</span><span class="n">tables</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">get_qualified_name</span><span class="p">())</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>external.external.apassdr9
external.external.gaiadr2_geometric_distance
external.external.galex_ais
external.external.ravedr5_com
external.external.ravedr5_dr5
external.external.ravedr5_gra
external.external.ravedr5_on
external.external.sdssdr13_photoprimary
external.external.skymapperdr1_master
external.external.tmass_xsc
public.public.hipparcos
public.public.hipparcos_newreduction
public.public.hubble_sc
public.public.igsl_source
public.public.igsl_source_catalog_ids
public.public.tycho2
public.public.dual
tap_config.tap_config.coord_sys
tap_config.tap_config.properties
tap_schema.tap_schema.columns
tap_schema.tap_schema.key_columns
tap_schema.tap_schema.keys
tap_schema.tap_schema.schemas
tap_schema.tap_schema.tables
gaiadr1.gaiadr1.aux_qso_icrf2_match
gaiadr1.gaiadr1.ext_phot_zero_point
gaiadr1.gaiadr1.allwise_best_neighbour
gaiadr1.gaiadr1.allwise_neighbourhood
gaiadr1.gaiadr1.gsc23_best_neighbour
gaiadr1.gaiadr1.gsc23_neighbourhood
gaiadr1.gaiadr1.ppmxl_best_neighbour
gaiadr1.gaiadr1.ppmxl_neighbourhood
gaiadr1.gaiadr1.sdss_dr9_best_neighbour
gaiadr1.gaiadr1.sdss_dr9_neighbourhood
gaiadr1.gaiadr1.tmass_best_neighbour
gaiadr1.gaiadr1.tmass_neighbourhood
gaiadr1.gaiadr1.ucac4_best_neighbour
gaiadr1.gaiadr1.ucac4_neighbourhood
gaiadr1.gaiadr1.urat1_best_neighbour
gaiadr1.gaiadr1.urat1_neighbourhood
gaiadr1.gaiadr1.cepheid
gaiadr1.gaiadr1.phot_variable_time_series_gfov
gaiadr1.gaiadr1.phot_variable_time_series_gfov_statistical_parameters
gaiadr1.gaiadr1.rrlyrae
gaiadr1.gaiadr1.variable_summary
gaiadr1.gaiadr1.allwise_original_valid
gaiadr1.gaiadr1.gsc23_original_valid
gaiadr1.gaiadr1.ppmxl_original_valid
gaiadr1.gaiadr1.sdssdr9_original_valid
gaiadr1.gaiadr1.tmass_original_valid
gaiadr1.gaiadr1.ucac4_original_valid
gaiadr1.gaiadr1.urat1_original_valid
gaiadr1.gaiadr1.gaia_source
gaiadr1.gaiadr1.tgas_source
gaiadr2.gaiadr2.aux_allwise_agn_gdr2_cross_id
gaiadr2.gaiadr2.aux_iers_gdr2_cross_id
gaiadr2.gaiadr2.aux_sso_orbit_residuals
gaiadr2.gaiadr2.aux_sso_orbits
gaiadr2.gaiadr2.dr1_neighbourhood
gaiadr2.gaiadr2.allwise_best_neighbour
gaiadr2.gaiadr2.allwise_neighbourhood
gaiadr2.gaiadr2.apassdr9_best_neighbour
gaiadr2.gaiadr2.apassdr9_neighbourhood
gaiadr2.gaiadr2.gsc23_best_neighbour
gaiadr2.gaiadr2.gsc23_neighbourhood
gaiadr2.gaiadr2.hipparcos2_best_neighbour
gaiadr2.gaiadr2.hipparcos2_neighbourhood
gaiadr2.gaiadr2.panstarrs1_best_neighbour
gaiadr2.gaiadr2.panstarrs1_neighbourhood
gaiadr2.gaiadr2.ppmxl_best_neighbour
gaiadr2.gaiadr2.ppmxl_neighbourhood
gaiadr2.gaiadr2.ravedr5_best_neighbour
gaiadr2.gaiadr2.ravedr5_neighbourhood
gaiadr2.gaiadr2.sdssdr9_best_neighbour
gaiadr2.gaiadr2.sdssdr9_neighbourhood
gaiadr2.gaiadr2.tmass_best_neighbour
gaiadr2.gaiadr2.tmass_neighbourhood
gaiadr2.gaiadr2.tycho2_best_neighbour
gaiadr2.gaiadr2.tycho2_neighbourhood
gaiadr2.gaiadr2.urat1_best_neighbour
gaiadr2.gaiadr2.urat1_neighbourhood
gaiadr2.gaiadr2.sso_observation
gaiadr2.gaiadr2.sso_source
gaiadr2.gaiadr2.vari_cepheid
gaiadr2.gaiadr2.vari_classifier_class_definition
gaiadr2.gaiadr2.vari_classifier_definition
gaiadr2.gaiadr2.vari_classifier_result
gaiadr2.gaiadr2.vari_long_period_variable
gaiadr2.gaiadr2.vari_rotation_modulation
gaiadr2.gaiadr2.vari_rrlyrae
gaiadr2.gaiadr2.vari_short_timescale
gaiadr2.gaiadr2.vari_time_series_statistics
gaiadr2.gaiadr2.panstarrs1_original_valid
gaiadr2.gaiadr2.gaia_source
gaiadr2.gaiadr2.ruwe
</pre></div>
</div>
</div>
</div>
<p>So that’s a lot of tables. The ones we’ll use are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">gaiadr2.gaia_source</span></code>, which contains Gaia data from <a class="reference external" href="https://www.cosmos.esa.int/web/gaia/data-release-2">data release 2</a>,</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">gaiadr2.panstarrs1_original_valid</span></code>, which contains the photometry data we’ll use from PanSTARRS, and</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">gaiadr2.panstarrs1_best_neighbour</span></code>, which we’ll use to cross-match each star observed by Gaia with the same star observed by PanSTARRS.</p></li>
</ul>
<p>We can use <code class="docutils literal notranslate"><span class="pre">load_table</span></code> (not <code class="docutils literal notranslate"><span class="pre">load_tables</span></code>) to get the metadata for a single table. The name of this function is misleading, because it only downloads metadata.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">meta</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_table</span><span class="p">(</span><span class="s1">&#39;gaiadr2.gaia_source&#39;</span><span class="p">)</span>
<span class="n">meta</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Retrieving table &#39;gaiadr2.gaia_source&#39;
Parsing table &#39;gaiadr2.gaia_source&#39;...
Done.
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;astroquery.utils.tap.model.taptable.TapTableMeta at 0x7fb565288520&gt;
</pre></div>
</div>
</div>
</div>
<p>Jupyter shows that the result is an object of type <code class="docutils literal notranslate"><span class="pre">TapTableMeta</span></code>, but it does not display the contents.</p>
<p>To see the metadata, we have to print the object.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">meta</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>TAP Table name: gaiadr2.gaiadr2.gaia_source
Description: This table has an entry for every Gaia observed source as listed in the
Main Database accumulating catalogue version from which the catalogue
release has been generated. It contains the basic source parameters,
that is only final data (no epoch data) and no spectra (neither final
nor epoch).
Num. columns: 96
</pre></div>
</div>
</div>
</div>
<p>Notice one gotcha: in the list of table names, this table appears as <code class="docutils literal notranslate"><span class="pre">gaiadr2.gaiadr2.gaia_source</span></code>, but when we load the metadata, we refer to it as <code class="docutils literal notranslate"><span class="pre">gaiadr2.gaia_source</span></code>.</p>
<p><strong>Exercise:</strong> Go back and try</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">meta</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_table</span><span class="p">(</span><span class="s1">&#39;gaiadr2.gaiadr2.gaia_source&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>What happens? Is the error message helpful? If you had not made this error deliberately, would you have been able to figure it out?</p>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="c1"># The error message, last time we tried, was</span>
<span class="c1"># Retrieving table &#39;gaiadr2.gaiadr2.gaia_source&#39;</span>
<span class="c1"># 500 Error 500:</span>
<span class="c1"># esavo.tap.TAPException: esavo.tap.TAPException: Schema cannot be null</span>
<span class="c1"># Which is not remotely helpful.</span>
<span class="c1"># The point of this exercise is to alert the participants to the difficulty</span>
<span class="c1"># of debugging queries with VERY limited feedback. So developing and testing</span>
<span class="c1"># incrementally is very important.</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="columns">
<h2>Columns<a class="headerlink" href="#columns" title="Permalink to this headline"></a></h2>
<p>The following loop prints the names of the columns in the table.</p>
<div class="cell tag_hide-output docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">meta</span><span class="o">.</span><span class="n">columns</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">column</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>solution_id
designation
source_id
random_index
ref_epoch
ra
ra_error
dec
dec_error
parallax
parallax_error
parallax_over_error
pmra
pmra_error
pmdec
pmdec_error
ra_dec_corr
ra_parallax_corr
ra_pmra_corr
ra_pmdec_corr
dec_parallax_corr
dec_pmra_corr
dec_pmdec_corr
parallax_pmra_corr
parallax_pmdec_corr
pmra_pmdec_corr
astrometric_n_obs_al
astrometric_n_obs_ac
astrometric_n_good_obs_al
astrometric_n_bad_obs_al
astrometric_gof_al
astrometric_chi2_al
astrometric_excess_noise
astrometric_excess_noise_sig
astrometric_params_solved
astrometric_primary_flag
astrometric_weight_al
astrometric_pseudo_colour
astrometric_pseudo_colour_error
mean_varpi_factor_al
astrometric_matched_observations
visibility_periods_used
astrometric_sigma5d_max
frame_rotator_object_type
matched_observations
duplicated_source
phot_g_n_obs
phot_g_mean_flux
phot_g_mean_flux_error
phot_g_mean_flux_over_error
phot_g_mean_mag
phot_bp_n_obs
phot_bp_mean_flux
phot_bp_mean_flux_error
phot_bp_mean_flux_over_error
phot_bp_mean_mag
phot_rp_n_obs
phot_rp_mean_flux
phot_rp_mean_flux_error
phot_rp_mean_flux_over_error
phot_rp_mean_mag
phot_bp_rp_excess_factor
phot_proc_mode
bp_rp
bp_g
g_rp
radial_velocity
radial_velocity_error
rv_nb_transits
rv_template_teff
rv_template_logg
rv_template_fe_h
phot_variable_flag
l
b
ecl_lon
ecl_lat
priam_flags
teff_val
teff_percentile_lower
teff_percentile_upper
a_g_val
a_g_percentile_lower
a_g_percentile_upper
e_bp_min_rp_val
e_bp_min_rp_percentile_lower
e_bp_min_rp_percentile_upper
flame_flags
radius_val
radius_percentile_lower
radius_percentile_upper
lum_val
lum_percentile_lower
lum_percentile_upper
datalink_url
epoch_photometry_url
</pre></div>
</div>
</div>
</div>
<p>You can probably guess what many of these columns are by looking at the names, but you should resist the temptation to guess.
To find out what the columns mean, <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">read the documentation</a>.</p>
<p>If you want to know what can go wrong when you don’t read the documentation, <a class="reference external" href="https://www.vox.com/future-perfect/2019/6/4/18650969/married-women-miserable-fake-paul-dolan-happiness">you might like this article</a>.</p>
<p><strong>Exercise:</strong> One of the other tables we’ll use is <code class="docutils literal notranslate"><span class="pre">gaiadr2.gaiadr2.panstarrs1_original_valid</span></code>. Use <code class="docutils literal notranslate"><span class="pre">load_table</span></code> to get the metadata for this table. How many columns are there and what are their names?</p>
<p>Hint: Remember the gotcha we mentioned earlier.</p>
<div class="cell tag_hide-cell tag_remove-output docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">meta2</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">load_table</span><span class="p">(</span><span class="s1">&#39;gaiadr2.panstarrs1_original_valid&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">meta2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">meta2</span><span class="o">.</span><span class="n">columns</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">column</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>obj_name
obj_id
ra
dec
ra_error
dec_error
epoch_mean
g_mean_psf_mag
g_mean_psf_mag_error
g_flags
r_mean_psf_mag
r_mean_psf_mag_error
r_flags
i_mean_psf_mag
i_mean_psf_mag_error
i_flags
z_mean_psf_mag
z_mean_psf_mag_error
z_flags
y_mean_psf_mag
y_mean_psf_mag_error
y_flags
n_detections
zone_id
obj_info_flag
quality_flag
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="writing-queries">
<h2>Writing queries<a class="headerlink" href="#writing-queries" title="Permalink to this headline"></a></h2>
<p>By now you might be wondering how we actually download the data. With tables this big, you generally don’t. Instead, you use queries to select only the data you want.</p>
<p>A query is a string written in a query language like SQL; for the Gaia database, the query language is a dialect of SQL called ADQL.</p>
<p>Here’s an example of an ADQL query.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query1</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT </span>
<span class="s2">TOP 10</span>
<span class="s2">source_id, ref_epoch, ra, dec, parallax </span>
<span class="s2">FROM gaiadr2.gaia_source&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p><strong>Python note:</strong> We use a <a class="reference external" href="https://docs.python.org/3/tutorial/introduction.html#strings">triple-quoted string</a> here so we can include line breaks in the query, which makes it easier to read.</p>
<p>The words in uppercase are ADQL keywords:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">SELECT</span></code> indicates that we are selecting data (as opposed to adding or modifying data).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">TOP</span></code> indicates that we only want the first 10 rows of the table, which is useful for testing a query before asking for all of the data.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">FROM</span></code> specifies which table we want data from.</p></li>
</ul>
<p>The third line is a list of column names, indicating which columns we want.</p>
<p>In this example, the keywords are capitalized and the column names are lowercase. This is a common style, but it is not required. ADQL and SQL are not case-sensitive.</p>
<p>To run this query, we use the <code class="docutils literal notranslate"><span class="pre">Gaia</span></code> object, which represents our connection to the Gaia database, and invoke <code class="docutils literal notranslate"><span class="pre">launch_job</span></code>:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job1</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query1</span><span class="p">)</span>
<span class="n">job1</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;astroquery.utils.tap.model.job.Job at 0x7f9222e9cb20&gt;
</pre></div>
</div>
</div>
</div>
<p>The result is an object that represents the job running on a Gaia server.</p>
<p>If you print it, it displays metadata for the forthcoming table.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">job1</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;Table length=10&gt;
name dtype unit description
--------- ------- ---- ------------------------------------------------------------------
source_id int64 Unique source identifier (unique within a particular Data Release)
ref_epoch float64 yr Reference epoch
ra float64 deg Right ascension
dec float64 deg Declination
parallax float64 mas Parallax
Jobid: None
Phase: COMPLETED
Owner: None
Output file: sync_20201005090721.xml.gz
Results: None
</pre></div>
</div>
</div>
</div>
<p>Don’t worry about <code class="docutils literal notranslate"><span class="pre">Results:</span> <span class="pre">None</span></code>. That does not actually mean there are no results.</p>
<p>However, <code class="docutils literal notranslate"><span class="pre">Phase:</span> <span class="pre">COMPLETED</span></code> indicates that the job is complete, so we can get the results like this:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results1</span> <span class="o">=</span> <span class="n">job1</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="nb">type</span><span class="p">(</span><span class="n">results1</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>astropy.table.table.Table
</pre></div>
</div>
</div>
</div>
<p><strong>Optional detail:</strong> Why is <code class="docutils literal notranslate"><span class="pre">table</span></code> repeated three times? The first is the name of the module, the second is the name of the submodule, and the third is the name of the class. Most of the time we only care about the last one. It’s like the Linnean name for gorilla, which is <em>Gorilla Gorilla Gorilla</em>.</p>
<p>The result is an <a class="reference external" href="https://docs.astropy.org/en/stable/table/">Astropy Table</a>, which is similar to a table in an SQL database except:</p>
<ul class="simple">
<li><p>SQL databases are stored on disk drives, so they are persistent; that is, they “survive” even if you turn off the computer. An Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> is stored in memory; it disappears when you turn off the computer (or shut down this Jupyter notebook).</p></li>
<li><p>SQL databases are designed to process queries. An Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> can perform some query-like operations, like selecting columns and rows. But these operations use Python syntax, not SQL.</p></li>
</ul>
<p>Jupyter knows how to display the contents of a <code class="docutils literal notranslate"><span class="pre">Table</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results1</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=10</i>
<table id="table140265627585264" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ref_epoch</th><th>ra</th><th>dec</th><th>parallax</th></tr></thead>
<thead><tr><th></th><th>yr</th><th>deg</th><th>deg</th><th>mas</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>4530738361793769600</td><td>2015.5</td><td>281.56725362448725</td><td>20.40682117430378</td><td>0.9785380604519425</td></tr>
<tr><td>4530752651135081216</td><td>2015.5</td><td>281.0861565355257</td><td>20.523350496351846</td><td>0.2674800612552977</td></tr>
<tr><td>4530743343951405568</td><td>2015.5</td><td>281.37114418299177</td><td>20.474147574053124</td><td>-0.43911323550176806</td></tr>
<tr><td>4530755060627162368</td><td>2015.5</td><td>281.2676236268299</td><td>20.558523922346158</td><td>1.1422630184554958</td></tr>
<tr><td>4530746844341315968</td><td>2015.5</td><td>281.1370431749541</td><td>20.377852388898184</td><td>1.0092247424630945</td></tr>
<tr><td>4530768456615026432</td><td>2015.5</td><td>281.8720921436347</td><td>20.31829694530366</td><td>-0.06900136127674149</td></tr>
<tr><td>4530763513119137280</td><td>2015.5</td><td>281.9211808864116</td><td>20.20956829578524</td><td>0.1266016679823622</td></tr>
<tr><td>4530736364618539264</td><td>2015.5</td><td>281.4913475613274</td><td>20.346579041327693</td><td>0.3894019486060072</td></tr>
<tr><td>4530735952305177728</td><td>2015.5</td><td>281.4085549165704</td><td>20.311030903719928</td><td>0.2041189982608354</td></tr>
<tr><td>4530751281056022656</td><td>2015.5</td><td>281.0585328377638</td><td>20.460309556214753</td><td>0.10294642821734962</td></tr>
</table></div></div>
</div>
<p>Each column has a name, units, and a data type.</p>
<p>For example, the units of <code class="docutils literal notranslate"><span class="pre">ra</span></code> and <code class="docutils literal notranslate"><span class="pre">dec</span></code> are degrees, and their data type is <code class="docutils literal notranslate"><span class="pre">float64</span></code>, which is a 64-bit floating-point number, used to store measurements with a fraction part.</p>
<p>This information comes from the Gaia database, and has been stored in the Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> by Astroquery.</p>
<p><strong>Exercise:</strong> Read <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">the documentation of this table</a> and choose a column that looks interesting to you. Add the column name to the query and run it again. What are the units of the column you selected? What is its data type?</p>
</div>
<div class="section" id="asynchronous-queries">
<h2>Asynchronous queries<a class="headerlink" href="#asynchronous-queries" title="Permalink to this headline"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">launch_job</span></code> asks the server to run the job “synchronously”, which normally means it runs immediately. But synchronous jobs are limited to 2000 rows. For queries that return more rows, you should run “asynchronously”, which mean they might take longer to get started.</p>
<p>If you are not sure how many rows a query will return, you can use the SQL command <code class="docutils literal notranslate"><span class="pre">COUNT</span></code> to find out how many rows are in the result without actually returning them. We’ll see an example of this later.</p>
<p>The results of an asynchronous query are stored in a file on the server, so you can start a query and come back later to get the results.</p>
<p>For anonymous users, files are kept for three days.</p>
<p>As an example, let’s try a query that’s similar to <code class="docutils literal notranslate"><span class="pre">query1</span></code>, with two changes:</p>
<ul class="simple">
<li><p>It selects the first 3000 rows, so it is bigger than we should run synchronously.</p></li>
<li><p>It uses a new keyword, <code class="docutils literal notranslate"><span class="pre">WHERE</span></code>.</p></li>
</ul>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query2</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT TOP 3000</span>
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; 1</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>A <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clause indicates which rows we want; in this case, the query selects only rows “where” <code class="docutils literal notranslate"><span class="pre">parallax</span></code> is less than 1. This has the effect of selecting stars with relatively low parallax, which are farther away. We’ll use this clause to exclude nearby stars that are unlikely to be part of GD-1.</p>
<p><code class="docutils literal notranslate"><span class="pre">WHERE</span></code> is one of the most common clauses in ADQL/SQL, and one of the most useful, because it allows us to select only the rows we need from the database.</p>
<p>We use <code class="docutils literal notranslate"><span class="pre">launch_job_async</span></code> to submit an asynchronous query.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job2</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query2</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">job2</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
&lt;Table length=3000&gt;
name dtype unit description
--------- ------- ---- ------------------------------------------------------------------
source_id int64 Unique source identifier (unique within a particular Data Release)
ref_epoch float64 yr Reference epoch
ra float64 deg Right ascension
dec float64 deg Declination
parallax float64 mas Parallax
Jobid: 1601903242219O
Phase: COMPLETED
Owner: None
Output file: async_20201005090722.vot
Results: None
</pre></div>
</div>
</div>
</div>
<p>And here are the results.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results2</span> <span class="o">=</span> <span class="n">job2</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results2</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=3000</i>
<table id="table140265625141056" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ref_epoch</th><th>ra</th><th>dec</th><th>parallax</th></tr></thead>
<thead><tr><th></th><th>yr</th><th>deg</th><th>deg</th><th>mas</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>4530738361793769600</td><td>2015.5</td><td>281.56725362448725</td><td>20.40682117430378</td><td>0.9785380604519425</td></tr>
<tr><td>4530752651135081216</td><td>2015.5</td><td>281.0861565355257</td><td>20.523350496351846</td><td>0.2674800612552977</td></tr>
<tr><td>4530743343951405568</td><td>2015.5</td><td>281.37114418299177</td><td>20.474147574053124</td><td>-0.43911323550176806</td></tr>
<tr><td>4530768456615026432</td><td>2015.5</td><td>281.8720921436347</td><td>20.31829694530366</td><td>-0.06900136127674149</td></tr>
<tr><td>4530763513119137280</td><td>2015.5</td><td>281.9211808864116</td><td>20.20956829578524</td><td>0.1266016679823622</td></tr>
<tr><td>4530736364618539264</td><td>2015.5</td><td>281.4913475613274</td><td>20.346579041327693</td><td>0.3894019486060072</td></tr>
<tr><td>4530735952305177728</td><td>2015.5</td><td>281.4085549165704</td><td>20.311030903719928</td><td>0.2041189982608354</td></tr>
<tr><td>4530751281056022656</td><td>2015.5</td><td>281.0585328377638</td><td>20.460309556214753</td><td>0.10294642821734962</td></tr>
<tr><td>4530740938774409344</td><td>2015.5</td><td>281.3762569536416</td><td>20.436140058941206</td><td>0.9242670062090182</td></tr>
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
<tr><td>4467710915011802624</td><td>2015.5</td><td>269.9680969307347</td><td>1.1429085038160882</td><td>0.42361471245557913</td></tr>
<tr><td>4467706551328679552</td><td>2015.5</td><td>270.033164589881</td><td>1.0565747323689927</td><td>0.922888231734588</td></tr>
<tr><td>4467712255037300096</td><td>2015.5</td><td>270.7724717923047</td><td>0.6581664892880896</td><td>-2.669179465293931</td></tr>
<tr><td>4467735001181761792</td><td>2015.5</td><td>270.3628606248308</td><td>0.8947079323599124</td><td>0.6117399163086398</td></tr>
<tr><td>4467737101421916672</td><td>2015.5</td><td>270.5110834661444</td><td>0.9806225910160181</td><td>-0.39818224846127004</td></tr>
<tr><td>4467707547757327488</td><td>2015.5</td><td>269.88746280594927</td><td>1.0212759940136962</td><td>0.7741412301054209</td></tr>
<tr><td>4467732772094573056</td><td>2015.5</td><td>270.55997182760126</td><td>0.9037072088489417</td><td>-1.7920417800164183</td></tr>
<tr><td>4467732355491087744</td><td>2015.5</td><td>270.6730790702491</td><td>0.9197224705139885</td><td>-0.3464446494840354</td></tr>
<tr><td>4467717099766944512</td><td>2015.5</td><td>270.57667173120825</td><td>0.726277659009568</td><td>0.05443955111134051</td></tr>
<tr><td>4467719058265781248</td><td>2015.5</td><td>270.7248052971514</td><td>0.8205551921782785</td><td>0.3733943917490343</td></tr>
</table></div></div>
</div>
<p>You might notice that some values of <code class="docutils literal notranslate"><span class="pre">parallax</span></code> are negative. As <a class="reference external" href="https://www.cosmos.esa.int/web/gaia/archive-tips#negative%20parallax">this FAQ explains</a>, “Negative parallaxes are caused by errors in the observations.” Negative parallaxes have “no physical meaning,” but they can be a “useful diagnostic on the quality of the astrometric solution.”</p>
<p>Later we will see an example where we use <code class="docutils literal notranslate"><span class="pre">parallax</span></code> and <code class="docutils literal notranslate"><span class="pre">parallax_error</span></code> to identify stars where the distance estimate is likely to be inaccurate.</p>
<p><strong>Exercise:</strong> The clauses in a query have to be in the right order. Go back and change the order of the clauses in <code class="docutils literal notranslate"><span class="pre">query2</span></code> and run it again.</p>
<p>The query should fail, but notice that you don’t get much useful debugging information.</p>
<p>For this reason, developing and debugging ADQL queries can be really hard. A few suggestions that might help:</p>
<ul class="simple">
<li><p>Whenever possible, start with a working query, either an example you find online or a query you have used in the past.</p></li>
<li><p>Make small changes and test each change before you continue.</p></li>
<li><p>While you are debugging, use <code class="docutils literal notranslate"><span class="pre">TOP</span></code> to limit the number of rows in the result. That will make each attempt run faster, which reduces your testing time.</p></li>
<li><p>Launching test queries synchronously might make them start faster, too.</p></li>
</ul>
</div>
<div class="section" id="operators">
<h2>Operators<a class="headerlink" href="#operators" title="Permalink to this headline"></a></h2>
<p>In a <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clause, you can use any of the <a class="reference external" href="https://www.w3schools.com/sql/sql_operators.asp">SQL comparison operators</a>; here are the most common ones:</p>
<table class="colwidths-auto table">
<thead>
<tr class="row-odd"><th class="head"><p>Symbol</p></th>
<th class="text-align:left head"><p>Operation</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">&gt;</span></code></p></td>
<td class="text-align:left"><p>greater than</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">&lt;</span></code></p></td>
<td class="text-align:left"><p>less than</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">&gt;=</span></code></p></td>
<td class="text-align:left"><p>greater than or equal</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">&lt;=</span></code></p></td>
<td class="text-align:left"><p>less than or equal</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">=</span></code></p></td>
<td class="text-align:left"><p>equal</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">!=</span></code> or <code class="docutils literal notranslate"><span class="pre">&lt;&gt;</span></code></p></td>
<td class="text-align:left"><p>not equal</p></td>
</tr>
</tbody>
</table>
<p>Most of these are the same as Python, but some are not. In particular, notice that the equality operator is <code class="docutils literal notranslate"><span class="pre">=</span></code>, not <code class="docutils literal notranslate"><span class="pre">==</span></code>.
Be careful to keep your Python out of your ADQL!</p>
<p>You can combine comparisons using the logical operators:</p>
<ul class="simple">
<li><p>AND: true if both comparisons are true</p></li>
<li><p>OR: true if either or both comparisons are true</p></li>
</ul>
<p>Finally, you can use <code class="docutils literal notranslate"><span class="pre">NOT</span></code> to invert the result of a comparison.</p>
<p><strong>Exercise:</strong> <a class="reference external" href="https://www.w3schools.com/sql/sql_operators.asp">Read about SQL operators here</a> and then modify the previous query to select rows where <code class="docutils literal notranslate"><span class="pre">bp_rp</span></code> is between <code class="docutils literal notranslate"><span class="pre">-0.75</span></code> and <code class="docutils literal notranslate"><span class="pre">2</span></code>.</p>
<p>You can <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">read about this variable here</a>.</p>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="c1"># This is what most people will probably do</span>
<span class="n">query</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT TOP 10</span>
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; 1 </span>
<span class="s2"> AND bp_rp &gt; -0.75 AND bp_rp &lt; 2</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="c1"># But if someone notices the BETWEEN operator, </span>
<span class="c1"># they might do this</span>
<span class="n">query</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT TOP 10</span>
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; 1 </span>
<span class="s2"> AND bp_rp BETWEEN -0.75 AND 2</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>This <a class="reference external" href="https://sci.esa.int/web/gaia/-/60198-gaia-hertzsprung-russell-diagram">Hertzsprung-Russell diagram</a> shows the BP-RP color and luminosity of stars in the Gaia catalog.</p>
<p>Selecting stars with <code class="docutils literal notranslate"><span class="pre">bp-rp</span></code> less than 2 excludes many <a class="reference external" href="https://xkcd.com/2360/">class M dwarf stars</a>, which are low temperature, low luminosity. A star like that at GD-1’s distance would be hard to detect, so if it is detected, it it more likely to be in the foreground.</p>
</div>
<div class="section" id="cleaning-up">
<h2>Cleaning up<a class="headerlink" href="#cleaning-up" title="Permalink to this headline"></a></h2>
<p>Asynchronous jobs have a <code class="docutils literal notranslate"><span class="pre">jobid</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job1</span><span class="o">.</span><span class="n">jobid</span><span class="p">,</span> <span class="n">job2</span><span class="o">.</span><span class="n">jobid</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(None, &#39;1601903242219O&#39;)
</pre></div>
</div>
</div>
</div>
<p>Which you can use to remove the job from the server.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">Gaia</span><span class="o">.</span><span class="n">remove_jobs</span><span class="p">([</span><span class="n">job2</span><span class="o">.</span><span class="n">jobid</span><span class="p">])</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Removed jobs: &#39;[&#39;1601903242219O&#39;]&#39;.
</pre></div>
</div>
</div>
</div>
<p>If you don’t remove it job from the server, it will be removed eventually, so don’t feel too bad if you don’t clean up after yourself.</p>
</div>
<div class="section" id="formatting-queries">
<h2>Formatting queries<a class="headerlink" href="#formatting-queries" title="Permalink to this headline"></a></h2>
<p>So far the queries have been string “literals”, meaning that the entire string is part of the program.
But writing queries yourself can be slow, repetitive, and error-prone.</p>
<p>It is often a good idea to write Python code that assembles a query for you. One useful tool for that is the <a class="reference external" href="https://www.w3schools.com/python/ref_string_format.asp">string <code class="docutils literal notranslate"><span class="pre">format</span></code> method</a>.</p>
<p>As an example, we’ll divide the previous query into two parts; a list of column names and a “base” for the query that contains everything except the column names.</p>
<p>Here’s the list of columns we’ll select.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">columns</span> <span class="o">=</span> <span class="s1">&#39;source_id, ra, dec, pmra, pmdec, parallax, parallax_error, radial_velocity&#39;</span>
</pre></div>
</div>
</div>
</div>
<p>And here’s the base; it’s a string that contains at least one format specifier in curly brackets (braces).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query3_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT TOP 10 </span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; 1</span>
<span class="s2"> AND bp_rp BETWEEN -0.75 AND 2</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<p>This base query contains one format specifier, <code class="docutils literal notranslate"><span class="pre">{columns}</span></code>, which is a placeholder for the list of column names we will provide.</p>
<p>To assemble the query, we invoke <code class="docutils literal notranslate"><span class="pre">format</span></code> on the base string and provide a keyword argument that assigns a value to <code class="docutils literal notranslate"><span class="pre">columns</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query3</span> <span class="o">=</span> <span class="n">query3_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>The result is a string with line breaks. If you display it, the line breaks appear as <code class="docutils literal notranslate"><span class="pre">\n</span></code>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query3</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&#39;SELECT TOP 10 \nsource_id, ra, dec, pmra, pmdec, parallax, parallax_error, radial_velocity\nFROM gaiadr2.gaia_source\nWHERE parallax &lt; 1\n AND bp_rp BETWEEN -0.75 AND 2\n&#39;
</pre></div>
</div>
</div>
</div>
<p>But if you print it, the line breaks appear as… line breaks.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">query3</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT TOP 10
source_id, ra, dec, pmra, pmdec, parallax, parallax_error, radial_velocity
FROM gaiadr2.gaia_source
WHERE parallax &lt; 1
AND bp_rp BETWEEN -0.75 AND 2
</pre></div>
</div>
</div>
</div>
<p>Notice that the format specifier has been replaced with the value of <code class="docutils literal notranslate"><span class="pre">columns</span></code>.</p>
<p>Let’s run it and see if it works:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job3</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query3</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">job3</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;Table length=10&gt;
name dtype unit description n_bad
--------------- ------- -------- ------------------------------------------------------------------ -----
source_id int64 Unique source identifier (unique within a particular Data Release) 0
ra float64 deg Right ascension 0
dec float64 deg Declination 0
pmra float64 mas / yr Proper motion in right ascension direction 0
pmdec float64 mas / yr Proper motion in declination direction 0
parallax float64 mas Parallax 0
parallax_error float64 mas Standard error of parallax 0
radial_velocity float64 km / s Radial velocity 10
Jobid: None
Phase: COMPLETED
Owner: None
Output file: sync_20201005090726.xml.gz
Results: None
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results3</span> <span class="o">=</span> <span class="n">job3</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
<span class="n">results3</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><i>Table length=10</i>
<table id="table140265627700432" class="table-striped table-bordered table-condensed">
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>parallax</th><th>parallax_error</th><th>radial_velocity</th></tr></thead>
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th>mas</th><th>mas</th><th>km / s</th></tr></thead>
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
<tr><td>4467710915011802624</td><td>269.9680969307347</td><td>1.1429085038160882</td><td>2.0233280236600626</td><td>-2.5692427875510266</td><td>0.42361471245557913</td><td>0.470352406647465</td><td>--</td></tr>
<tr><td>4467706551328679552</td><td>270.033164589881</td><td>1.0565747323689927</td><td>-3.414829591355289</td><td>-3.8437215857495737</td><td>0.922888231734588</td><td>0.927008559859825</td><td>--</td></tr>
<tr><td>4467712255037300096</td><td>270.7724717923047</td><td>0.6581664892880896</td><td>-3.5620173752896025</td><td>-6.595792323153987</td><td>-2.669179465293931</td><td>0.9719742773203504</td><td>--</td></tr>
<tr><td>4467735001181761792</td><td>270.3628606248308</td><td>0.8947079323599124</td><td>2.1307079926489205</td><td>0.8826727710910712</td><td>0.6117399163086398</td><td>0.509812721702093</td><td>--</td></tr>
<tr><td>4467737101421916672</td><td>270.5110834661444</td><td>0.9806225910160181</td><td>0.17532366511560785</td><td>-5.113270239706202</td><td>-0.39818224846127004</td><td>0.7549581886719651</td><td>--</td></tr>
<tr><td>4467707547757327488</td><td>269.88746280594927</td><td>1.0212759940136962</td><td>-2.6382230817672987</td><td>-3.707776532049287</td><td>0.7741412301054209</td><td>0.3022057897812064</td><td>--</td></tr>
<tr><td>4467732355491087744</td><td>270.6730790702491</td><td>0.9197224705139885</td><td>-2.2735991502653037</td><td>-11.864952855984358</td><td>-0.3464446494840354</td><td>0.4937921513912002</td><td>--</td></tr>
<tr><td>4467717099766944512</td><td>270.57667173120825</td><td>0.726277659009568</td><td>-3.4598362614808367</td><td>-4.601426893365921</td><td>0.05443955111134051</td><td>0.8867339293525688</td><td>--</td></tr>
<tr><td>4467719058265781248</td><td>270.7248052971514</td><td>0.8205551921782785</td><td>-3.255079498426542</td><td>-9.249285069111085</td><td>0.3733943917490343</td><td>0.390952370410666</td><td>--</td></tr>
<tr><td>4467722326741572352</td><td>270.87431291888504</td><td>0.8595565975869158</td><td>0.10696398351859826</td><td>1.2035993780158853</td><td>-0.11850943432864373</td><td>0.1660452431882023</td><td>--</td></tr>
</table></div></div>
</div>
<p>Good so far.</p>
<p><strong>Exercise:</strong> This query always selects sources with <code class="docutils literal notranslate"><span class="pre">parallax</span></code> less than 1. But suppose you want to take that upper bound as an input.</p>
<p>Modify <code class="docutils literal notranslate"><span class="pre">query3_base</span></code> to replace <code class="docutils literal notranslate"><span class="pre">1</span></code> with a format specifier like <code class="docutils literal notranslate"><span class="pre">{max_parallax}</span></code>. Now, when you call <code class="docutils literal notranslate"><span class="pre">format</span></code>, add a keyword argument that assigns a value to <code class="docutils literal notranslate"><span class="pre">max_parallax</span></code>, and confirm that the format specifier gets replaced with the value you provide.</p>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">query4_base</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;SELECT TOP 10</span>
<span class="si">{columns}</span><span class="s2"></span>
<span class="s2">FROM gaiadr2.gaia_source</span>
<span class="s2">WHERE parallax &lt; </span><span class="si">{max_parallax}</span><span class="s2"> AND </span>
<span class="s2">bp_rp BETWEEN -0.75 AND 2</span>
<span class="s2">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
</div>
<div class="cell tag_hide-cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
<span class="n">query4</span> <span class="o">=</span> <span class="n">query4_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
<span class="n">max_parallax</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT TOP 10
source_id, ra, dec, pmra, pmdec, parallax, parallax_error, radial_velocity
FROM gaiadr2.gaia_source
WHERE parallax &lt; 0.5 AND
bp_rp BETWEEN -0.75 AND 2
</pre></div>
</div>
</div>
</div>
<p><strong>Style note:</strong> You might notice that the variable names in this notebook are numbered, like <code class="docutils literal notranslate"><span class="pre">query1</span></code>, <code class="docutils literal notranslate"><span class="pre">query2</span></code>, etc.</p>
<p>The advantage of this style is that it isolates each section of the notebook from the others, so if you go back and run the cells out of order, it’s less likely that you will get unexpected interactions.</p>
<p>A drawback of this style is that it can be a nuisance to update the notebook if you add, remove, or reorder a section.</p>
<p>What do you think of this choice? Are there alternatives you prefer?</p>
</div>
<div class="section" id="summary">
<h2>Summary<a class="headerlink" href="#summary" title="Permalink to this headline"></a></h2>
<p>This notebook demonstrates the following steps:</p>
<ol class="simple">
<li><p>Making a connection to the Gaia server,</p></li>
<li><p>Exploring information about the database and the tables it contains,</p></li>
<li><p>Writing a query and sending it to the server, and finally</p></li>
<li><p>Downloading the response from the server as an Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code>.</p></li>
</ol>
</div>
<div class="section" id="best-practices">
<h2>Best practices<a class="headerlink" href="#best-practices" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>If you can’t download an entire dataset (or it’s not practical) use queries to select the data you need.</p></li>
<li><p>Read the metadata and the documentation to make sure you understand the tables, their columns, and what they mean.</p></li>
<li><p>Develop queries incrementally: start with something simple, test it, and add a little bit at a time.</p></li>
<li><p>Use ADQL features like <code class="docutils literal notranslate"><span class="pre">TOP</span></code> and <code class="docutils literal notranslate"><span class="pre">COUNT</span></code> to test before you run a query that might return a lot of data.</p></li>
<li><p>If you know your query will return fewer than 3000 rows, you can run it synchronously, which might complete faster (but it doesn’t seem to make much difference). If it might return more than 3000 rows, you should run it asynchronously.</p></li>
<li><p>ADQL and SQL are not case-sensitive, so you don’t have to capitalize the keywords, but you should.</p></li>
<li><p>ADQL and SQL don’t require you to break a query into multiple lines, but you should.</p></li>
</ul>
<p>Jupyter notebooks can be good for developing and testing code, but they have some drawbacks. In particular, if you run the cells out of order, you might find that variables don’t have the values you expect.</p>
<p>There are a few things you can do to mitigate these problems:</p>
<ul class="simple">
<li><p>Make each section of the notebook self-contained. Try not to use the same variable name in more than one section.</p></li>
<li><p>Keep notebooks short. Look for places where you can break your analysis into phases with one notebook per phase.</p></li>
</ul>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
</div>
</div>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="README.html" title="previous page">Astronomical Data in Python</a>
<a class='right-next' id="next-link" href="02_coords.html" title="next page">Chapter 2</a>
</div>
<footer class="footer mt-5 mt-md-0">
<div class="container">
<p>
By Allen B. Downey<br/>
&copy; Copyright 2020.<br/>
</p>
</div>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.30270b6e4c972e43c488.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink