mirror of
https://github.com/AllenDowney/AstronomicalData.git
synced 2025-12-31 14:55:31 -08:00
Update documentation
This commit is contained in:
Allen Downey
366
01_query.html
366
01_query.html
@@ -5,7 +5,7 @@
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Queries — Astronomical Data in Python</title>
|
||||
<title>1. Queries — Astronomical Data in Python</title>
|
||||
|
||||
<link rel="stylesheet" href="_static/css/index.d431a4ee1c1efae0e38bdfebc22debff.css">
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
<script async="async" src="_static/sphinx-thebe.js"></script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="Coordinates and units" href="02_coords.html" />
|
||||
<link rel="next" title="2. Coordinates and Units" href="02_coords.html" />
|
||||
<link rel="prev" title="Astronomical Data in Python" href="README.html" />
|
||||
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
@@ -99,22 +99,22 @@
|
||||
<ul class="current nav sidenav_l1">
|
||||
<li class="toctree-l1 current active">
|
||||
<a class="current reference internal" href="#">
|
||||
Queries
|
||||
1. Queries
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="02_coords.html">
|
||||
Coordinates and units
|
||||
2. Coordinates and Units
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="03_motion.html">
|
||||
Proper Motion
|
||||
3. Proper Motion
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="04_select.html">
|
||||
Transformation and Selection
|
||||
4. Transformation and Selection
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
@@ -311,11 +311,6 @@
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#cleaning-up">
|
||||
Cleaning up
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#formatting-queries">
|
||||
Formatting queries
|
||||
@@ -350,7 +345,9 @@
|
||||
<div>
|
||||
|
||||
<div class="section" id="queries">
|
||||
<h1>Queries<a class="headerlink" href="#queries" title="Permalink to this headline">¶</a></h1>
|
||||
<h1>1. Queries<a class="headerlink" href="#queries" title="Permalink to this headline">¶</a></h1>
|
||||
<p>This is the first in a series of lessons about working with astronomical data.</p>
|
||||
<p>As a running example, we will replicate parts of the analysis in a recent paper, “<a class="reference external" href="https://arxiv.org/abs/1805.00425">Off the beaten path: Gaia reveals GD-1 stars outside of the main stream</a>” by Adrian Price-Whelan and Ana Bonaca.</p>
|
||||
<div class="section" id="outline">
|
||||
<h2>Outline<a class="headerlink" href="#outline" title="Permalink to this headline">¶</a></h2>
|
||||
<p>This lesson demonstrates the steps for selecting and downloading data from the Gaia Database:</p>
|
||||
@@ -378,19 +375,19 @@ But you might find it easier to learn from <a class="reference external" href="h
|
||||
<li><p>“Jupyter Lab” is a newer environment with more features.</p></li>
|
||||
</ul>
|
||||
<p>For these lessons, you can use either one.</p>
|
||||
<p>If you are too impatient for the tutorials, here’s are the most important things to know:</p>
|
||||
<p>If you are too impatient for the tutorials, here are the most important things to know:</p>
|
||||
<ol class="simple">
|
||||
<li><p>Notebooks are made up of code cells and text cells (and a few other less common kinds). Code cells contain code; text cells, like this one, contain explanatory text written in <a class="reference external" href="https://www.markdownguide.org/">Markdown</a>.</p></li>
|
||||
<li><p>To run a code cell, click the cell to select it and press Shift-Enter. The output of the code should appear below the cell.</p></li>
|
||||
<li><p>In general, notebooks only run correctly if you run every code cell in order from top to bottom. If you run cells out of order, you are likely to get errors.</p></li>
|
||||
<li><p>You can modify existing cells, but then you have to run them again to see the effect.</p></li>
|
||||
<li><p>You can add new cells, but again, you might have to be careful about the order you run them in.</p></li>
|
||||
<li><p>You can add new cells, but again, you have to be careful about the order you run them in.</p></li>
|
||||
<li><p>If you have added or modified cells and the behavior of the notebook seems strange, you can restart the “kernel”, which clears all of the variables and functions you have defined, and run the cells again from the beginning.</p></li>
|
||||
</ol>
|
||||
<ul class="simple">
|
||||
<li><p>If you are using Jupyter notebook, open the Kernel menu and select “Restart and Run All”.</p></li>
|
||||
<li><p>In Jupyter Lab…</p></li>
|
||||
<li><p>In Colab, open the Runtime menu and select “Restart and run all”</p></li>
|
||||
<li><p>If you are using Jupyter notebook, open the <code class="docutils literal notranslate"><span class="pre">Kernel</span></code> menu and select “Restart and Run All”.</p></li>
|
||||
<li><p>In Jupyter Lab, open the <code class="docutils literal notranslate"><span class="pre">Kernel</span></code> menu and select “Restart Kernel and Run All Cells”</p></li>
|
||||
<li><p>In Colab, open the <code class="docutils literal notranslate"><span class="pre">Runtime</span></code> menu and select “Restart and run all”</p></li>
|
||||
</ul>
|
||||
<p>Before you go on, you might want to explore the other menus and the toolbar to see what else you can do.</p>
|
||||
</div>
|
||||
@@ -420,7 +417,7 @@ Created TAP+ (v1.2.1) - Connection:
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Running this import statement has the effect of creating a <a class="reference external" href="http://www.ivoa.net/documents/TAP/">TAP+</a> connection; TAP stands for “Table Access Protocol”, which is a network protocol for sending queries to the database and getting back the results.</p>
|
||||
<p>This import statement creates a <a class="reference external" href="http://www.ivoa.net/documents/TAP/">TAP+</a> connection; TAP stands for “Table Access Protocol”, which is a network protocol for sending queries to the database and getting back the results.</p>
|
||||
</div>
|
||||
<div class="section" id="databases-and-tables">
|
||||
<h2>Databases and Tables<a class="headerlink" href="#databases-and-tables" title="Permalink to this headline">¶</a></h2>
|
||||
@@ -444,6 +441,7 @@ INFO: Done. [astroquery.utils.tap.core]
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>The following <code class="docutils literal notranslate"><span class="pre">for</span></code> loop prints the names of the tables.</p>
|
||||
<div class="cell tag_hide-output tag_truncate-output docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">table</span> <span class="ow">in</span> <span class="n">tables</span><span class="p">:</span>
|
||||
@@ -454,6 +452,7 @@ INFO: Done. [astroquery.utils.tap.core]
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>external.apassdr9
|
||||
external.gaiadr2_geometric_distance
|
||||
external.gaiaedr3_distance
|
||||
external.galex_ais
|
||||
external.ravedr5_com
|
||||
external.ravedr5_dr5
|
||||
@@ -482,17 +481,31 @@ gaiaedr3.agn_cross_id
|
||||
gaiaedr3.commanded_scan_law
|
||||
gaiaedr3.dr2_neighbourhood
|
||||
gaiaedr3.frame_rotator_source
|
||||
gaiaedr3.allwise_best_neighbour
|
||||
gaiaedr3.allwise_neighbourhood
|
||||
gaiaedr3.apassdr9_best_neighbour
|
||||
gaiaedr3.apassdr9_join
|
||||
gaiaedr3.apassdr9_neighbourhood
|
||||
gaiaedr3.gsc23_best_neighbour
|
||||
gaiaedr3.gsc23_join
|
||||
gaiaedr3.gsc23_neighbourhood
|
||||
gaiaedr3.hipparcos2_best_neighbour
|
||||
gaiaedr3.hipparcos2_neighbourhood
|
||||
gaiaedr3.panstarrs1_best_neighbour
|
||||
gaiaedr3.panstarrs1_join
|
||||
gaiaedr3.panstarrs1_neighbourhood
|
||||
gaiaedr3.ravedr5_best_neighbour
|
||||
gaiaedr3.ravedr5_join
|
||||
gaiaedr3.ravedr5_neighbourhood
|
||||
gaiaedr3.sdssdr13_best_neighbour
|
||||
gaiaedr3.sdssdr13_join
|
||||
gaiaedr3.sdssdr13_neighbourhood
|
||||
gaiaedr3.skymapperdr2_best_neighbour
|
||||
gaiaedr3.skymapperdr2_join
|
||||
gaiaedr3.skymapperdr2_neighbourhood
|
||||
gaiaedr3.tmass_psc_xsc_best_neighbour
|
||||
gaiaedr3.tmass_psc_xsc_join
|
||||
gaiaedr3.tmass_psc_xsc_neighbourhood
|
||||
gaiaedr3.tycho2tdsc_merge_best_neighbour
|
||||
gaiaedr3.tycho2tdsc_merge_neighbourhood
|
||||
gaiaedr3.urat1_best_neighbour
|
||||
@@ -595,7 +608,7 @@ Parsing table 'gaiadr2.gaia_source'...
|
||||
Done.
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><astroquery.utils.tap.model.taptable.TapTableMeta at 0x7f2e23f089d0>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><astroquery.utils.tap.model.taptable.TapTableMeta at 0x7f50edd2aeb0>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -732,7 +745,7 @@ epoch_photometry_url
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>You can probably guess what many of these columns are by looking at the names, but you should resist the temptation to guess.
|
||||
<p>You can probably infer what many of these columns are by looking at the names, but you should resist the temptation to guess.
|
||||
To find out what the columns mean, <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">read the documentation</a>.</p>
|
||||
<p>If you want to know what can go wrong when you don’t read the documentation, <a class="reference external" href="https://www.vox.com/future-perfect/2019/6/4/18650969/married-women-miserable-fake-paul-dolan-happiness">you might like this article</a>.</p>
|
||||
<div class="section" id="exercise">
|
||||
@@ -855,15 +868,16 @@ quality_flag
|
||||
</div>
|
||||
<div class="section" id="writing-queries">
|
||||
<h2>Writing queries<a class="headerlink" href="#writing-queries" title="Permalink to this headline">¶</a></h2>
|
||||
<p>By now you might be wondering how we download the actual data. With tables this big, you generally don’t. Instead, you use queries to select only the data you want.</p>
|
||||
<p>By now you might be wondering how we download these tables. With tables this big, you generally don’t. Instead, you use queries to select only the data you want.</p>
|
||||
<p>A query is a string written in a query language like SQL; for the Gaia database, the query language is a dialect of SQL called ADQL.</p>
|
||||
<p>Here’s an example of an ADQL query.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query1</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10</span>
|
||||
<span class="s2">source_id, ref_epoch, ra, dec, parallax </span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source"""</span>
|
||||
<span class="s2">source_id, ra, dec, parallax </span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">"""</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -877,25 +891,26 @@ quality_flag
|
||||
</ul>
|
||||
<p>The third line is a list of column names, indicating which columns we want.</p>
|
||||
<p>In this example, the keywords are capitalized and the column names are lowercase. This is a common style, but it is not required. ADQL and SQL are not case-sensitive.</p>
|
||||
<p>Also, the query is broken into multiple lines to make it more readable. This is a common style, but not required. Line breaks don’t affect the behavior of the query.</p>
|
||||
<p>To run this query, we use the <code class="docutils literal notranslate"><span class="pre">Gaia</span></code> object, which represents our connection to the Gaia database, and invoke <code class="docutils literal notranslate"><span class="pre">launch_job</span></code>:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job1</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query1</span><span class="p">)</span>
|
||||
<span class="n">job1</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query1</span><span class="p">)</span>
|
||||
<span class="n">job</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><astroquery.utils.tap.model.job.Job at 0x7f2e23f2afa0>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><astroquery.utils.tap.model.job.Job at 0x7f50edd2adc0>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>The result is an object that represents the job running on a Gaia server.</p>
|
||||
<p>If you print it, it displays metadata for the forthcoming table.</p>
|
||||
<p>If you print it, it displays metadata for the forthcoming results.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">job1</span><span class="p">)</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">job</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -904,14 +919,13 @@ quality_flag
|
||||
name dtype unit description n_bad
|
||||
--------- ------- ---- ------------------------------------------------------------------ -----
|
||||
source_id int64 Unique source identifier (unique within a particular Data Release) 0
|
||||
ref_epoch float64 yr Reference epoch 0
|
||||
ra float64 deg Right ascension 0
|
||||
dec float64 deg Declination 0
|
||||
parallax float64 mas Parallax 2
|
||||
Jobid: None
|
||||
Phase: COMPLETED
|
||||
Owner: None
|
||||
Output file: sync_20201229114647.xml.gz
|
||||
Output file: sync_20210315090602.xml.gz
|
||||
Results: None
|
||||
</pre></div>
|
||||
</div>
|
||||
@@ -921,8 +935,8 @@ Results: None
|
||||
<p>However, <code class="docutils literal notranslate"><span class="pre">Phase:</span> <span class="pre">COMPLETED</span></code> indicates that the job is complete, so we can get the results like this:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results1</span> <span class="o">=</span> <span class="n">job1</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="nb">type</span><span class="p">(</span><span class="n">results1</span><span class="p">)</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="nb">type</span><span class="p">(</span><span class="n">results</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -932,8 +946,9 @@ Results: None
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>The <code class="docutils literal notranslate"><span class="pre">type</span></code> function indicates that the result is an <a class="reference external" href="https://docs.astropy.org/en/stable/table/">Astropy Table</a>.</p>
|
||||
<p><strong>Optional detail:</strong> Why is <code class="docutils literal notranslate"><span class="pre">table</span></code> repeated three times? The first is the name of the module, the second is the name of the submodule, and the third is the name of the class. Most of the time we only care about the last one. It’s like the Linnean name for gorilla, which is <em>Gorilla gorilla gorilla</em>.</p>
|
||||
<p>The result is an <a class="reference external" href="https://docs.astropy.org/en/stable/table/">Astropy Table</a>, which is similar to a table in an SQL database except:</p>
|
||||
<p>An Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> is similar to a table in an SQL database except:</p>
|
||||
<ul class="simple">
|
||||
<li><p>SQL databases are stored on disk drives, so they are persistent; that is, they “survive” even if you turn off the computer. An Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> is stored in memory; it disappears when you turn off the computer (or shut down this Jupyter notebook).</p></li>
|
||||
<li><p>SQL databases are designed to process queries. An Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> can perform some query-like operations, like selecting columns and rows. But these operations use Python syntax, not SQL.</p></li>
|
||||
@@ -941,37 +956,53 @@ Results: None
|
||||
<p>Jupyter knows how to display the contents of a <code class="docutils literal notranslate"><span class="pre">Table</span></code>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results1</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_html"><i>Table length=10</i>
|
||||
<table id="table139836148327136" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ref_epoch</th><th>ra</th><th>dec</th><th>parallax</th></tr></thead>
|
||||
<thead><tr><th></th><th>yr</th><th>deg</th><th>deg</th><th>mas</th></tr></thead>
|
||||
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
|
||||
<tr><td>6758509757594141440</td><td>2015.5</td><td>290.4899010727352</td><td>-30.34317218420783</td><td>0.48023816159705535</td></tr>
|
||||
<tr><td>6758508692437976960</td><td>2015.5</td><td>290.64176055082714</td><td>-30.28972013142225</td><td>2.2625971293368154</td></tr>
|
||||
<tr><td>6758527036250849280</td><td>2015.5</td><td>290.35703708993327</td><td>-30.193657826181596</td><td>-0.2763960334229464</td></tr>
|
||||
<tr><td>6758564458298242944</td><td>2015.5</td><td>290.77376294142846</td><td>-29.765368439225238</td><td>0.5907906528352993</td></tr>
|
||||
<tr><td>6758558612842155776</td><td>2015.5</td><td>290.67550630386245</td><td>-29.92113396078169</td><td>0.2858563565989917</td></tr>
|
||||
<tr><td>6758556379459921920</td><td>2015.5</td><td>290.5454006212404</td><td>-29.900709054816964</td><td>-1.0012355835832834</td></tr>
|
||||
<tr><td>6758537520260385152</td><td>2015.5</td><td>290.7341683169994</td><td>-30.158181298418626</td><td>--</td></tr>
|
||||
<tr><td>6758506355980010624</td><td>2015.5</td><td>290.586950869878</td><td>-30.37137642167569</td><td>0.3769870991981157</td></tr>
|
||||
<tr><td>6758561606433968128</td><td>2015.5</td><td>290.82697136098506</td><td>-29.75247697212053</td><td>--</td></tr>
|
||||
<tr><td>6758564183414408448</td><td>2015.5</td><td>290.7133096669958</td><td>-29.781743673679607</td><td>0.9376387942856869</td></tr>
|
||||
<table id="table139985564118224" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>parallax</th></tr></thead>
|
||||
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas</th></tr></thead>
|
||||
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
|
||||
<tr><td>5887983246081387776</td><td>227.978818386372</td><td>-53.64996962450103</td><td>1.0493172163332998</td></tr>
|
||||
<tr><td>5887971250213117952</td><td>228.32280834041364</td><td>-53.66270726203726</td><td>0.29455652682279093</td></tr>
|
||||
<tr><td>5887991866047288704</td><td>228.1582047014091</td><td>-53.454724911639794</td><td>-0.5789179941669236</td></tr>
|
||||
<tr><td>5887968673232040832</td><td>228.07420888099884</td><td>-53.8064612895961</td><td>0.41030970779603076</td></tr>
|
||||
<tr><td>5887979844465854720</td><td>228.42547805195946</td><td>-53.48882284470035</td><td>-0.23379683441525864</td></tr>
|
||||
<tr><td>5887978607515442688</td><td>228.23831627636855</td><td>-53.56328249482688</td><td>-0.9252161956789068</td></tr>
|
||||
<tr><td>5887978298278520704</td><td>228.26015640396173</td><td>-53.607284412896476</td><td>--</td></tr>
|
||||
<tr><td>5887995581231772928</td><td>228.12871598211902</td><td>-53.373625663608316</td><td>-0.3325818206439385</td></tr>
|
||||
<tr><td>5887982043490374016</td><td>227.985260087594</td><td>-53.683444499055575</td><td>0.02878111976456593</td></tr>
|
||||
<tr><td>5887982971205433856</td><td>227.89884570686218</td><td>-53.67430215342567</td><td>--</td></tr>
|
||||
</table></div></div>
|
||||
</div>
|
||||
<p>Each column has a name, units, and a data type.</p>
|
||||
<p>For example, the units of <code class="docutils literal notranslate"><span class="pre">ra</span></code> and <code class="docutils literal notranslate"><span class="pre">dec</span></code> are degrees, and their data type is <code class="docutils literal notranslate"><span class="pre">float64</span></code>, which is a 64-bit floating-point number, used to store measurements with a fraction part.</p>
|
||||
<p>For example, the units of <code class="docutils literal notranslate"><span class="pre">ra</span></code> and <code class="docutils literal notranslate"><span class="pre">dec</span></code> are degrees, and their data type is <code class="docutils literal notranslate"><span class="pre">float64</span></code>, which is a 64-bit <a class="reference external" href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">floating-point number</a>, used to store measurements with a fraction part.</p>
|
||||
<p>This information comes from the Gaia database, and has been stored in the Astropy <code class="docutils literal notranslate"><span class="pre">Table</span></code> by Astroquery.</p>
|
||||
<div class="section" id="id1">
|
||||
<h3>Exercise<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Read <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">the documentation of this table</a> and choose a column that looks interesting to you. Add the column name to the query and run it again. What are the units of the column you selected? What is its data type?</p>
|
||||
<p>Read <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">the documentation</a> of this table and choose a column that looks interesting to you. Add the column name to the query and run it again. What are the units of the column you selected? What is its data type?</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
|
||||
|
||||
<span class="c1"># Let's add</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># radial_velocity : Radial velocity (double, Velocity[km/s] )</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># Spectroscopic radial velocity in the solar barycentric </span>
|
||||
<span class="c1"># reference frame.</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># The radial velocity provided is the median value of the </span>
|
||||
<span class="c1"># radial velocity measurements at all epochs.</span>
|
||||
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10</span>
|
||||
<span class="s2">source_id, ra, dec, parallax, radial_velocity</span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">"""</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -981,18 +1012,20 @@ Results: None
|
||||
<div class="section" id="asynchronous-queries">
|
||||
<h2>Asynchronous queries<a class="headerlink" href="#asynchronous-queries" title="Permalink to this headline">¶</a></h2>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">launch_job</span></code> asks the server to run the job “synchronously”, which normally means it runs immediately. But synchronous jobs are limited to 2000 rows. For queries that return more rows, you should run “asynchronously”, which mean they might take longer to get started.</p>
|
||||
<p>If you are not sure how many rows a query will return, you can use the SQL command <code class="docutils literal notranslate"><span class="pre">COUNT</span></code> to find out how many rows are in the result without actually returning them. We’ll see an example of this later.</p>
|
||||
<p>The results of an asynchronous query are stored in a file on the server, so you can start a query and come back later to get the results.</p>
|
||||
<p>For anonymous users, files are kept for three days.</p>
|
||||
<p>As an example, let’s try a query that’s similar to <code class="docutils literal notranslate"><span class="pre">query1</span></code>, with two changes:</p>
|
||||
<p>If you are not sure how many rows a query will return, you can use the SQL command <code class="docutils literal notranslate"><span class="pre">COUNT</span></code> to find out how many rows are in the result without actually returning them. We’ll see an example in the next lesson.</p>
|
||||
<p>The results of an asynchronous query are stored in a file on the server, so you can start a query and come back later to get the results.
|
||||
For anonymous users, files are kept for three days.</p>
|
||||
<p>As an example, let’s try a query that’s similar to <code class="docutils literal notranslate"><span class="pre">query1</span></code>, with these changes:</p>
|
||||
<ul class="simple">
|
||||
<li><p>It selects the first 3000 rows, so it is bigger than we should run synchronously.</p></li>
|
||||
<li><p>It selects two additional columns, <code class="docutils literal notranslate"><span class="pre">pmra</span></code> and <code class="docutils literal notranslate"><span class="pre">pmdec</span></code>, which are proper motions along the axes of <code class="docutils literal notranslate"><span class="pre">ra</span></code> and <code class="docutils literal notranslate"><span class="pre">dec</span></code>.</p></li>
|
||||
<li><p>It uses a new keyword, <code class="docutils literal notranslate"><span class="pre">WHERE</span></code>.</p></li>
|
||||
</ul>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query2</span> <span class="o">=</span> <span class="s2">"""SELECT TOP 3000</span>
|
||||
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query2</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 3000</span>
|
||||
<span class="s2">source_id, ra, dec, pmra, pmdec, parallax</span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">WHERE parallax < 1</span>
|
||||
<span class="s2">"""</span>
|
||||
@@ -1000,31 +1033,22 @@ Results: None
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>A <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clause indicates which rows we want; in this case, the query selects only rows “where” <code class="docutils literal notranslate"><span class="pre">parallax</span></code> is less than 1. This has the effect of selecting stars with relatively low parallax, which are farther away. We’ll use this clause to exclude nearby stars that are unlikely to be part of GD-1.</p>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">WHERE</span></code> is one of the most common clauses in ADQL/SQL, and one of the most useful, because it allows us to select only the rows we need from the database.</p>
|
||||
<p>A <code class="docutils literal notranslate"><span class="pre">WHERE</span></code> clause indicates which rows we want; in this case, the query selects only rows “where” <code class="docutils literal notranslate"><span class="pre">parallax</span></code> is less than 1. This has the effect of selecting stars with relatively low parallax, which are farther away.
|
||||
We’ll use this clause to exclude nearby stars that are unlikely to be part of GD-1.</p>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">WHERE</span></code> is one of the most common clauses in ADQL/SQL, and one of the most useful, because it allows us to download only the rows we need from the database.</p>
|
||||
<p>We use <code class="docutils literal notranslate"><span class="pre">launch_job_async</span></code> to submit an asynchronous query.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job2</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query2</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">job2</span><span class="p">)</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job_async</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
|
||||
<span class="n">job</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>INFO: Query finished. [astroquery.utils.tap.core]
|
||||
<Table length=3000>
|
||||
name dtype unit description
|
||||
--------- ------- ---- ------------------------------------------------------------------
|
||||
source_id int64 Unique source identifier (unique within a particular Data Release)
|
||||
ref_epoch float64 yr Reference epoch
|
||||
ra float64 deg Right ascension
|
||||
dec float64 deg Declination
|
||||
parallax float64 mas Parallax
|
||||
Jobid: 1609260407863O
|
||||
Phase: COMPLETED
|
||||
Owner: None
|
||||
Output file: async_20201229114648.vot
|
||||
Results: None
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><astroquery.utils.tap.model.job.Job at 0x7f50edd40f40>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1032,44 +1056,34 @@ Results: None
|
||||
<p>And here are the results.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results2</span> <span class="o">=</span> <span class="n">job2</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="n">results2</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="n">results</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_html"><i>Table length=3000</i>
|
||||
<table id="table139836145366016" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ref_epoch</th><th>ra</th><th>dec</th><th>parallax</th></tr></thead>
|
||||
<thead><tr><th></th><th>yr</th><th>deg</th><th>deg</th><th>mas</th></tr></thead>
|
||||
<div class="output text_html"><i>Table length=10</i>
|
||||
<table id="table139986226873968" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>parallax</th><th>radial_velocity</th></tr></thead>
|
||||
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas</th><th>km / s</th></tr></thead>
|
||||
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
|
||||
<tr><td>6758509757594141440</td><td>2015.5</td><td>290.4899010727352</td><td>-30.34317218420783</td><td>0.48023816159705535</td></tr>
|
||||
<tr><td>6758527036250849280</td><td>2015.5</td><td>290.35703708993327</td><td>-30.193657826181596</td><td>-0.2763960334229464</td></tr>
|
||||
<tr><td>6758564458298242944</td><td>2015.5</td><td>290.77376294142846</td><td>-29.765368439225238</td><td>0.5907906528352993</td></tr>
|
||||
<tr><td>6758558612842155776</td><td>2015.5</td><td>290.67550630386245</td><td>-29.92113396078169</td><td>0.2858563565989917</td></tr>
|
||||
<tr><td>6758556379459921920</td><td>2015.5</td><td>290.5454006212404</td><td>-29.900709054816964</td><td>-1.0012355835832834</td></tr>
|
||||
<tr><td>6758506355980010624</td><td>2015.5</td><td>290.586950869878</td><td>-30.37137642167569</td><td>0.3769870991981157</td></tr>
|
||||
<tr><td>6758564183414408448</td><td>2015.5</td><td>290.7133096669958</td><td>-29.781743673679607</td><td>0.9376387942856869</td></tr>
|
||||
<tr><td>6758529789322653184</td><td>2015.5</td><td>290.35449465190385</td><td>-30.10231481465004</td><td>-1.3305133038319952</td></tr>
|
||||
<tr><td>6758507485555336064</td><td>2015.5</td><td>290.6745961953205</td><td>-30.35965148061028</td><td>0.057302711920868686</td></tr>
|
||||
<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>
|
||||
<tr><td>4660464894038295424</td><td>2015.5</td><td>79.74509945051724</td><td>-66.49656012737759</td><td>0.1631257821260955</td></tr>
|
||||
<tr><td>4660463519650480128</td><td>2015.5</td><td>80.06150680482719</td><td>-66.42891601595986</td><td>-0.5822669736733328</td></tr>
|
||||
<tr><td>4660487674554071040</td><td>2015.5</td><td>80.03536662790846</td><td>-66.33136792885134</td><td>-1.4978065466579966</td></tr>
|
||||
<tr><td>4660494718300886784</td><td>2015.5</td><td>79.6181997343967</td><td>-66.17472335318212</td><td>0.1488652133861382</td></tr>
|
||||
<tr><td>4660476473274159360</td><td>2015.5</td><td>79.12508291180023</td><td>-66.35644844757275</td><td>0.10671540523101529</td></tr>
|
||||
<tr><td>4660471010093940224</td><td>2015.5</td><td>79.1908378165463</td><td>-66.48916840294096</td><td>0.8152150079365807</td></tr>
|
||||
<tr><td>4660498394807167744</td><td>2015.5</td><td>80.23431532382547</td><td>-66.08688681815092</td><td>-0.14060596426163252</td></tr>
|
||||
<tr><td>4660498738391722880</td><td>2015.5</td><td>80.20205898946918</td><td>-66.04392313295126</td><td>0.1894642263668475</td></tr>
|
||||
<tr><td>4660474068091096960</td><td>2015.5</td><td>78.93686107060341</td><td>-66.45795576745896</td><td>0.4665513634059366</td></tr>
|
||||
<tr><td>4660437303162697728</td><td>2015.5</td><td>80.63872120225027</td><td>-66.46144560233674</td><td>-1.1319163320535979</td></tr>
|
||||
<tr><td>5895270396817359872</td><td>213.08433715252883</td><td>-56.64104701005694</td><td>2.041947005434917</td><td>--</td></tr>
|
||||
<tr><td>5895272561481374080</td><td>213.2606587905109</td><td>-56.55044401535715</td><td>0.15693467895110133</td><td>--</td></tr>
|
||||
<tr><td>5895247410183786368</td><td>213.38479712976664</td><td>-56.97008551185148</td><td>-0.19017525742552605</td><td>--</td></tr>
|
||||
<tr><td>5895249226912448000</td><td>213.41587389088238</td><td>-56.849596577635786</td><td>--</td><td>--</td></tr>
|
||||
<tr><td>5895261875598904576</td><td>213.5508930114549</td><td>-56.61037780154348</td><td>-0.29471722363529257</td><td>--</td></tr>
|
||||
<tr><td>5895258302187834624</td><td>213.87631129557286</td><td>-56.678537259039906</td><td>0.6468437015289753</td><td>--</td></tr>
|
||||
<tr><td>5895247444506644992</td><td>213.33215109206796</td><td>-56.975347759380995</td><td>0.390215490234287</td><td>--</td></tr>
|
||||
<tr><td>5895259470417635968</td><td>213.78815034206346</td><td>-56.64585047451594</td><td>0.953377710788918</td><td>--</td></tr>
|
||||
<tr><td>5895264899260932352</td><td>213.21521027193236</td><td>-56.78420864489118</td><td>--</td><td>--</td></tr>
|
||||
<tr><td>5895265925746051584</td><td>213.17082359534547</td><td>-56.74540885107754</td><td>0.2986918215101751</td><td>--</td></tr>
|
||||
</table></div></div>
|
||||
</div>
|
||||
<p>You might notice that some values of <code class="docutils literal notranslate"><span class="pre">parallax</span></code> are negative. As <a class="reference external" href="https://www.cosmos.esa.int/web/gaia/archive-tips#negative%20parallax">this FAQ explains</a>, “Negative parallaxes are caused by errors in the observations.” Negative parallaxes have “no physical meaning,” but they can be a “useful diagnostic on the quality of the astrometric solution.”</p>
|
||||
<p>You might notice that some values of <code class="docutils literal notranslate"><span class="pre">parallax</span></code> are negative. As <a class="reference external" href="https://www.cosmos.esa.int/web/gaia/archive-tips#negative%20parallax">this FAQ explains</a>, “Negative parallaxes are caused by errors in the observations.” They have “no physical meaning,” but they can be a “useful diagnostic on the quality of the astrometric solution.”</p>
|
||||
<div class="section" id="id2">
|
||||
<h3>Exercise<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The clauses in a query have to be in the right order. Go back and change the order of the clauses in <code class="docutils literal notranslate"><span class="pre">query2</span></code> and run it again.</p>
|
||||
<p>The query should fail, but notice that you don’t get much useful debugging information.</p>
|
||||
<p>The clauses in a query have to be in the right order. Go back and change the order of the clauses in <code class="docutils literal notranslate"><span class="pre">query2</span></code> and run it again.
|
||||
The modified query should fail, but notice that you don’t get much useful debugging information.</p>
|
||||
<p>For this reason, developing and debugging ADQL queries can be really hard. A few suggestions that might help:</p>
|
||||
<ul class="simple">
|
||||
<li><p>Whenever possible, start with a working query, either an example you find online or a query you have used in the past.</p></li>
|
||||
@@ -1080,6 +1094,15 @@ Results: None
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
|
||||
|
||||
<span class="c1"># In this example, the WHERE clause is in the wrong place</span>
|
||||
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 3000</span>
|
||||
<span class="s2">WHERE parallax < 1</span>
|
||||
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">"""</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1127,14 +1150,14 @@ Be careful to keep your Python out of your ADQL!</p>
|
||||
<div class="section" id="id3">
|
||||
<h3>Exercise<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h3>
|
||||
<p><a class="reference external" href="https://www.w3schools.com/sql/sql_operators.asp">Read about SQL operators here</a> and then modify the previous query to select rows where <code class="docutils literal notranslate"><span class="pre">bp_rp</span></code> is between <code class="docutils literal notranslate"><span class="pre">-0.75</span></code> and <code class="docutils literal notranslate"><span class="pre">2</span></code>.</p>
|
||||
<p>You can <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">read about this variable here</a>.</p>
|
||||
<div class="cell tag_hide-cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
|
||||
|
||||
<span class="c1"># Here's a solution using > and < operators</span>
|
||||
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT TOP 10</span>
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10</span>
|
||||
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">WHERE parallax < 1 </span>
|
||||
@@ -1143,7 +1166,8 @@ Be careful to keep your Python out of your ADQL!</p>
|
||||
|
||||
<span class="c1"># And here's a solution using the BETWEEN operator</span>
|
||||
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT TOP 10</span>
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10</span>
|
||||
<span class="s2">source_id, ref_epoch, ra, dec, parallax</span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">WHERE parallax < 1 </span>
|
||||
@@ -1153,51 +1177,23 @@ Be careful to keep your Python out of your ADQL!</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">bp_rp</span></code> contains BP-RP color, which is the difference between two other columns, <code class="docutils literal notranslate"><span class="pre">phot_bp_mean_mag</span></code> and <code class="docutils literal notranslate"><span class="pre">phot_rp_mean_mag</span></code>.
|
||||
You can <a class="reference external" href="https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html">read about this variable here</a>.</p>
|
||||
<p>This <a class="reference external" href="https://sci.esa.int/web/gaia/-/60198-gaia-hertzsprung-russell-diagram">Hertzsprung-Russell diagram</a> shows the BP-RP color and luminosity of stars in the Gaia catalog (Copyright: ESA/Gaia/DPAC, CC BY-SA 3.0 IGO).</p>
|
||||
<a class="reference internal image-reference" href="https://github.com/AllenDowney/AstronomicalData/raw/main/images/1567214809100-ESA_Gaia_DR2_HRD_Gaia_625.jpg"><img alt="https://github.com/AllenDowney/AstronomicalData/raw/main/images/1567214809100-ESA_Gaia_DR2_HRD_Gaia_625.jpg" src="https://github.com/AllenDowney/AstronomicalData/raw/main/images/1567214809100-ESA_Gaia_DR2_HRD_Gaia_625.jpg" style="width: 300px;" /></a>
|
||||
<p>Selecting stars with <code class="docutils literal notranslate"><span class="pre">bp-rp</span></code> less than 2 excludes many <a class="reference external" href="https://xkcd.com/2360/">class M dwarf stars</a>, which are low temperature, low luminosity. A star like that at GD-1’s distance would be hard to detect, so if it is detected, it it more likely to be in the foreground.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="cleaning-up">
|
||||
<h2>Cleaning up<a class="headerlink" href="#cleaning-up" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Asynchronous jobs have a <code class="docutils literal notranslate"><span class="pre">jobid</span></code>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job1</span><span class="o">.</span><span class="n">jobid</span><span class="p">,</span> <span class="n">job2</span><span class="o">.</span><span class="n">jobid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(None, '1609260407863O')
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Which you can use to remove the job from the server.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">Gaia</span><span class="o">.</span><span class="n">remove_jobs</span><span class="p">([</span><span class="n">job2</span><span class="o">.</span><span class="n">jobid</span><span class="p">])</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Removed jobs: '['1609260407863O']'.
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>If you don’t remove it job from the server, it will be removed eventually, so don’t feel too bad if you don’t clean up after yourself.</p>
|
||||
</div>
|
||||
<div class="section" id="formatting-queries">
|
||||
<h2>Formatting queries<a class="headerlink" href="#formatting-queries" title="Permalink to this headline">¶</a></h2>
|
||||
<p>So far the queries have been string “literals”, meaning that the entire string is part of the program.
|
||||
<p>The queries we have written so far are string “literals”, meaning that the entire string is part of the program.
|
||||
But writing queries yourself can be slow, repetitive, and error-prone.</p>
|
||||
<p>It is often a good idea to write Python code that assembles a query for you. One useful tool for that is the <a class="reference external" href="https://www.w3schools.com/python/ref_string_format.asp">string <code class="docutils literal notranslate"><span class="pre">format</span></code> method</a>.</p>
|
||||
<p>It is often better to write Python code that assembles a query for you. One useful tool for that is the <a class="reference external" href="https://www.w3schools.com/python/ref_string_format.asp">string <code class="docutils literal notranslate"><span class="pre">format</span></code> method</a>.</p>
|
||||
<p>As an example, we’ll divide the previous query into two parts; a list of column names and a “base” for the query that contains everything except the column names.</p>
|
||||
<p>Here’s the list of columns we’ll select.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">columns</span> <span class="o">=</span> <span class="s1">'source_id, ra, dec, pmra, pmdec, parallax, radial_velocity'</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">columns</span> <span class="o">=</span> <span class="s1">'source_id, ra, dec, pmra, pmdec, parallax'</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1205,7 +1201,8 @@ But writing queries yourself can be slow, repetitive, and error-prone.</p>
|
||||
<p>And here’s the base; it’s a string that contains at least one format specifier in curly brackets (braces).</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query3_base</span> <span class="o">=</span> <span class="s2">"""SELECT TOP 10 </span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">query3_base</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10 </span>
|
||||
<span class="si">{columns}</span><span class="s2"></span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">WHERE parallax < 1</span>
|
||||
@@ -1224,6 +1221,8 @@ But writing queries yourself can be slow, repetitive, and error-prone.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>In this example, the variable that contains the column names and the variable in the format specifier have the same name.
|
||||
That’s not required, but it is a common style.</p>
|
||||
<p>The result is a string with line breaks. If you display it, the line breaks appear as <code class="docutils literal notranslate"><span class="pre">\n</span></code>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -1232,7 +1231,7 @@ But writing queries yourself can be slow, repetitive, and error-prone.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>'SELECT TOP 10 \nsource_id, ra, dec, pmra, pmdec, parallax, radial_velocity\nFROM gaiadr2.gaia_source\nWHERE parallax < 1\n AND bp_rp BETWEEN -0.75 AND 2\n'
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>'SELECT \nTOP 10 \nsource_id, ra, dec, pmra, pmdec\nFROM gaiadr2.gaia_source\nWHERE parallax < 1\n AND bp_rp BETWEEN -0.75 AND 2\n'
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1245,8 +1244,9 @@ But writing queries yourself can be slow, repetitive, and error-prone.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT TOP 10
|
||||
source_id, ra, dec, pmra, pmdec, parallax, radial_velocity
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
|
||||
TOP 10
|
||||
source_id, ra, dec, pmra, pmdec
|
||||
FROM gaiadr2.gaia_source
|
||||
WHERE parallax < 1
|
||||
AND bp_rp BETWEEN -0.75 AND 2
|
||||
@@ -1258,26 +1258,24 @@ WHERE parallax < 1
|
||||
<p>Let’s run it and see if it works:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job3</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query3</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">job3</span><span class="p">)</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">job</span> <span class="o">=</span> <span class="n">Gaia</span><span class="o">.</span><span class="n">launch_job</span><span class="p">(</span><span class="n">query3</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">job</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><Table length=10>
|
||||
name dtype unit description n_bad
|
||||
--------------- ------- -------- ------------------------------------------------------------------ -----
|
||||
source_id int64 Unique source identifier (unique within a particular Data Release) 0
|
||||
ra float64 deg Right ascension 0
|
||||
dec float64 deg Declination 0
|
||||
pmra float64 mas / yr Proper motion in right ascension direction 0
|
||||
pmdec float64 mas / yr Proper motion in declination direction 0
|
||||
parallax float64 mas Parallax 0
|
||||
radial_velocity float64 km / s Radial velocity 10
|
||||
name dtype unit description
|
||||
--------- ------- -------- ------------------------------------------------------------------
|
||||
source_id int64 Unique source identifier (unique within a particular Data Release)
|
||||
ra float64 deg Right ascension
|
||||
dec float64 deg Declination
|
||||
pmra float64 mas / yr Proper motion in right ascension direction
|
||||
pmdec float64 mas / yr Proper motion in declination direction
|
||||
Jobid: None
|
||||
Phase: COMPLETED
|
||||
Owner: None
|
||||
Output file: sync_20201229114653.xml.gz
|
||||
Output file: sync_20210315091929.xml.gz
|
||||
Results: None
|
||||
</pre></div>
|
||||
</div>
|
||||
@@ -1285,27 +1283,27 @@ Results: None
|
||||
</div>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results3</span> <span class="o">=</span> <span class="n">job3</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="n">results3</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">get_results</span><span class="p">()</span>
|
||||
<span class="n">results</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_html"><i>Table length=10</i>
|
||||
<table id="table139836148327184" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>parallax</th><th>radial_velocity</th></tr></thead>
|
||||
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th>mas</th><th>km / s</th></tr></thead>
|
||||
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
|
||||
<tr><td>4660466371507774848</td><td>79.49100199261952</td><td>-66.41411777672668</td><td>2.6073927760139983</td><td>-1.1968907801968522</td><td>-0.13534455558687877</td><td>--</td></tr>
|
||||
<tr><td>4660498704031984000</td><td>80.19346436358076</td><td>-66.04907675232856</td><td>1.644062563874402</td><td>-1.954975321006136</td><td>-0.1774586376397</td><td>--</td></tr>
|
||||
<tr><td>4660458949824817024</td><td>79.68666778641992</td><td>-66.52220959662557</td><td>1.4777697275880581</td><td>0.3328961930362448</td><td>-0.030149510331454386</td><td>--</td></tr>
|
||||
<tr><td>4660451974773926528</td><td>79.89972073493868</td><td>-66.75037858071191</td><td>2.4635048563021065</td><td>-0.7474574729840501</td><td>-0.005219591141134416</td><td>--</td></tr>
|
||||
<tr><td>4660474033731357056</td><td>78.93214036467484</td><td>-66.46483644591667</td><td>1.2369118132455594</td><td>-1.7187143034638677</td><td>0.6018032937392243</td><td>--</td></tr>
|
||||
<tr><td>4660454105077874688</td><td>79.91045649235578</td><td>-66.68106846135971</td><td>1.4747855407533677</td><td>0.022464361420165786</td><td>-0.19949109843083218</td><td>--</td></tr>
|
||||
<tr><td>4660494855753740544</td><td>79.63101881952036</td><td>-66.15063315813536</td><td>1.6227863263222113</td><td>0.20490893261905152</td><td>0.0013030724292998944</td><td>--</td></tr>
|
||||
<tr><td>4660439536571308160</td><td>80.24817961417894</td><td>-66.46303270664092</td><td>1.8652105429518493</td><td>-0.5618584442373111</td><td>-0.10578442360531497</td><td>--</td></tr>
|
||||
<tr><td>4660470185440466560</td><td>79.25942306061864</td><td>-66.51120777549605</td><td>3.373561172003989</td><td>0.25326243152876704</td><td>-0.6932631338638376</td><td>--</td></tr>
|
||||
<tr><td>4660441460690365568</td><td>80.38862155241748</td><td>-66.34404494135046</td><td>1.6588478675146066</td><td>-0.09646951910977163</td><td>-0.2100859303087445</td><td>--</td></tr>
|
||||
<table id="table139985564118512" class="table-striped table-bordered table-condensed">
|
||||
<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th></tr></thead>
|
||||
<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th></tr></thead>
|
||||
<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>
|
||||
<tr><td>5895272561481374080</td><td>213.2606587905109</td><td>-56.55044401535715</td><td>0.3894438898301715</td><td>1.2299266281737415</td></tr>
|
||||
<tr><td>5895261875598904576</td><td>213.5508930114549</td><td>-56.61037780154348</td><td>0.16203641364393007</td><td>-4.672602679543312</td></tr>
|
||||
<tr><td>5895247444506644992</td><td>213.33215109206796</td><td>-56.975347759380995</td><td>-7.474003156859284</td><td>-3.538080792097856</td></tr>
|
||||
<tr><td>5895259470417635968</td><td>213.78815034206346</td><td>-56.64585047451594</td><td>-5.287202255231853</td><td>-0.8163762113468646</td></tr>
|
||||
<tr><td>5895265925746051584</td><td>213.17082359534547</td><td>-56.74540885107754</td><td>-7.880749306158471</td><td>-4.8585444120179595</td></tr>
|
||||
<tr><td>5895260913525974528</td><td>213.66936020541976</td><td>-56.66655190442016</td><td>-4.7820929042428215</td><td>-1.5566420086447643</td></tr>
|
||||
<tr><td>5895264212062283008</td><td>213.7755742121852</td><td>-56.51570859067397</td><td>-6.657690998559842</td><td>-1.7616494482071872</td></tr>
|
||||
<tr><td>5895253457497979136</td><td>213.30929960610283</td><td>-56.78849448744587</td><td>-5.242106718924749</td><td>-0.18655636353898095</td></tr>
|
||||
<tr><td>4143614130253524096</td><td>269.1749117455479</td><td>-18.53415139972117</td><td>2.6164274510804826</td><td>1.3244248889980894</td></tr>
|
||||
<tr><td>4065443904433108992</td><td>273.26868565443743</td><td>-24.421651815402857</td><td>-1.663096652191022</td><td>-2.6514745376067683</td></tr>
|
||||
</table></div></div>
|
||||
</div>
|
||||
<p>Good so far.</p>
|
||||
@@ -1317,25 +1315,27 @@ Results: None
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Solution</span>
|
||||
|
||||
<span class="n">query4_base</span> <span class="o">=</span> <span class="s2">"""SELECT TOP 10</span>
|
||||
<span class="n">query_base</span> <span class="o">=</span> <span class="s2">"""SELECT </span>
|
||||
<span class="s2">TOP 10</span>
|
||||
<span class="si">{columns}</span><span class="s2"></span>
|
||||
<span class="s2">FROM gaiadr2.gaia_source</span>
|
||||
<span class="s2">WHERE parallax < </span><span class="si">{max_parallax}</span><span class="s2"> AND </span>
|
||||
<span class="s2">bp_rp BETWEEN -0.75 AND 2</span>
|
||||
<span class="s2">"""</span>
|
||||
|
||||
<span class="n">query4</span> <span class="o">=</span> <span class="n">query4_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
|
||||
<span class="n">max_parallax</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
|
||||
<span class="n">query</span> <span class="o">=</span> <span class="n">query_base</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
|
||||
<span class="n">max_parallax</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT TOP 10
|
||||
source_id, ref_epoch, ra, dec, parallax
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>SELECT
|
||||
TOP 10
|
||||
source_id, ra, dec, pmra, pmdec
|
||||
FROM gaiadr2.gaia_source
|
||||
WHERE parallax < 1
|
||||
AND bp_rp BETWEEN -0.75 AND 2
|
||||
WHERE parallax < 0.5 AND
|
||||
bp_rp BETWEEN -0.75 AND 2
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1360,22 +1360,16 @@ WHERE parallax < 1
|
||||
<li><p>Read the metadata and the documentation to make sure you understand the tables, their columns, and what they mean.</p></li>
|
||||
<li><p>Develop queries incrementally: start with something simple, test it, and add a little bit at a time.</p></li>
|
||||
<li><p>Use ADQL features like <code class="docutils literal notranslate"><span class="pre">TOP</span></code> and <code class="docutils literal notranslate"><span class="pre">COUNT</span></code> to test before you run a query that might return a lot of data.</p></li>
|
||||
<li><p>If you know your query will return fewer than 3000 rows, you can run it synchronously, which might complete faster (but it doesn’t seem to make much difference). If it might return more than 3000 rows, you should run it asynchronously.</p></li>
|
||||
<li><p>If you know your query will return fewer than 2000 rows, you can run it synchronously, which might complete faster. If it might return more than 2000 rows, you should run it asynchronously.</p></li>
|
||||
<li><p>ADQL and SQL are not case-sensitive, so you don’t have to capitalize the keywords, but you should.</p></li>
|
||||
<li><p>ADQL and SQL don’t require you to break a query into multiple lines, but you should.</p></li>
|
||||
</ul>
|
||||
<p>Jupyter notebooks can be good for developing and testing code, but they have some drawbacks. In particular, if you run the cells out of order, you might find that variables don’t have the values you expect.</p>
|
||||
<p>There are a few things you can do to mitigate these problems:</p>
|
||||
<p>To mitigate these problems:</p>
|
||||
<ul class="simple">
|
||||
<li><p>Make each section of the notebook self-contained. Try not to use the same variable name in more than one section.</p></li>
|
||||
<li><p>Keep notebooks short. Look for places where you can break your analysis into phases with one notebook per phase.</p></li>
|
||||
</ul>
|
||||
<blockquote>
|
||||
<div><p>One of the other tables we’ll use is
|
||||
<code class="docutils literal notranslate"><span class="pre">gaiadr2.panstarrs1_original_valid</span></code>. Use <code class="docutils literal notranslate"><span class="pre">load_table</span></code> to get the
|
||||
metadata for this table. How many columns are there and what are
|
||||
their names?</p>
|
||||
</div></blockquote>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -1408,7 +1402,7 @@ their names?</p>
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="README.html" title="previous page">Astronomical Data in Python</a>
|
||||
<a class='right-next' id="next-link" href="02_coords.html" title="next page">Coordinates and units</a>
|
||||
<a class='right-next' id="next-link" href="02_coords.html" title="next page">2. Coordinates and Units</a>
|
||||
|
||||
</div>
|
||||
<footer class="footer mt-5 mt-md-0">
|
||||
|
||||
Reference in New Issue
Block a user