Revising notebooks

This commit is contained in:
Allen Downey
2020-12-07 15:02:26 -05:00
parent 211402462c
commit 165243aac6
7 changed files with 1148 additions and 848 deletions
+276 -422
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Chapter 2\n",
"# Coordinates and units\n",
"\n",
"This is the second in a series of notebooks related to astronomy data.\n",
"\n",
@@ -75,7 +75,7 @@
"IN_COLAB = 'google.colab' in sys.modules\n",
"\n",
"if IN_COLAB:\n",
" !pip install astroquery astro-gala pyia"
" !pip install astroquery astro-gala"
]
},
{
+300 -78
View File
@@ -79,7 +79,7 @@
"IN_COLAB = 'google.colab' in sys.modules\n",
"\n",
"if IN_COLAB:\n",
" !pip install astroquery astro-gala pyia python-wget"
" !pip install astroquery astro-gala python-wget"
]
},
{
@@ -331,7 +331,7 @@
"data": {
"text/html": [
"<i>Row index=0</i>\n",
"<table id=\"table139758619539344\">\n",
"<table id=\"table140419756472688\">\n",
"<thead><tr><th>source_id</th><th>ra</th><th>dec</th><th>pmra</th><th>pmdec</th><th>parallax</th><th>parallax_error</th><th>radial_velocity</th></tr></thead>\n",
"<thead><tr><th></th><th>deg</th><th>deg</th><th>mas / yr</th><th>mas / yr</th><th>mas</th><th>mas</th><th>km / s</th></tr></thead>\n",
"<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th></tr></thead>\n",
@@ -555,64 +555,49 @@
"\n",
"To plot them, we will transform them back to the `GD1Koposov10` frame; that way, the axes of the figure are aligned with the GD-1, which will make it easy to select stars near the centerline of the stream.\n",
"\n",
"To do that, we'll put the results into a `GaiaData` object, provided by the [pyia library](https://pyia.readthedocs.io/en/latest/api/pyia.GaiaData.html).\n",
"\n",
"TODO: Do we need pyia, or could we do this with astropy and gala?"
"To do that, we'll put the results into a `SkyCoord` object, \n",
"which is an \"interface for celestial coordinate representation, manipulation, and transformation between systems\", provided by Astropy."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"pyia.data.GaiaData"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pyia import GaiaData\n",
"\n",
"gaia_data = GaiaData(results)\n",
"type(gaia_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can extract sky coordinates from the `GaiaData` object, like this:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"import astropy.coordinates as coord\n",
"import astropy.units as u\n",
"\n",
"skycoord = gaia_data.get_skycoord(\n",
" distance=8*u.kpc, \n",
" radial_velocity=0*u.km/u.s)"
"skycoord = coord.SkyCoord(\n",
" ra=results['ra'], \n",
" dec=results['dec'],\n",
" pm_ra_cosdec=results['pmra'],\n",
" pm_dec=results['pmdec'], \n",
" distance=8*u.kpc, \n",
" radial_velocity=0*u.km/u.s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We provide `distance` and `radial_velocity` to prepare the data for reflex correction, which we explain below."
"Most of the arguments we send to `SkyCoord` come directly from `results`.\n",
"\n",
"We provide `distance` and `radial_velocity` to prepare the data for reflex correction, which we explain below.\n",
"\n",
"The result is a `SkyCoord` object ([documentation here](https://docs.astropy.org/en/stable/api/astropy.coordinates.SkyCoord.html#astropy.coordinates.SkyCoord))."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result is an Astropy `SkyCoord` object."
]
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 59,
"metadata": {},
"outputs": [
{
@@ -621,7 +606,7 @@
"astropy.coordinates.sky_coordinate.SkyCoord"
]
},
"execution_count": 16,
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
@@ -634,12 +619,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The result is an Astropy `SkyCoord` object ([documentation here](https://docs.astropy.org/en/stable/api/astropy.coordinates.SkyCoord.html#astropy.coordinates.SkyCoord)), which provides `transform_to`, so we can transform the coordinates to other frames."
"`SkyCoord` provides `transform_to`, so we can transform the coordinates to other frames."
]
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 60,
"metadata": {},
"outputs": [
{
@@ -648,7 +633,7 @@
"astropy.coordinates.sky_coordinate.SkyCoord"
]
},
"execution_count": 17,
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
@@ -695,7 +680,7 @@
},
{
"cell_type": "code",
"execution_count": 18,
"execution_count": 61,
"metadata": {},
"outputs": [
{
@@ -704,7 +689,7 @@
"astropy.coordinates.sky_coordinate.SkyCoord"
]
},
"execution_count": 18,
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
@@ -730,7 +715,7 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 57,
"metadata": {
"scrolled": true
},
@@ -749,7 +734,7 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": 58,
"metadata": {},
"outputs": [
{
@@ -785,7 +770,7 @@
"source": [
"## Pandas DataFrame\n",
"\n",
"At this point we have three objects containing different subsets of the data."
"At this point we have two objects containing different subsets of the data. `results` is the Astropy `Table` we downloaded from Gaia."
]
},
{
@@ -809,28 +794,15 @@
]
},
{
"cell_type": "code",
"execution_count": 22,
"cell_type": "markdown",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"pyia.data.GaiaData"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(gaia_data)"
"And `gd1_coord` is a `SkyCoord` object that contains the transformed coordinates and proper motions."
]
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": 63,
"metadata": {},
"outputs": [
{
@@ -839,7 +811,7 @@
"astropy.coordinates.sky_coordinate.SkyCoord"
]
},
"execution_count": 23,
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
@@ -852,7 +824,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"On one hand, this makes sense, since each object provides different capabilities. But working with three different object types can be awkward.\n",
"On one hand, this division of labor makes sense because each object provides different capabilities. But working with multiple object types can be awkward.\n",
"\n",
"It will be more convenient to choose one object and get all of the data into it. We'll use a Pandas DataFrame, for two reasons:\n",
"\n",
@@ -1027,7 +999,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Python detail: `shape` is an attribute, so we can display it's value without calling it as a function; `head` is a function, so we need the parentheses."
"Python detail: `shape` is an attribute, so we display its value without calling it as a function; `head` is a function, so we need the parentheses."
]
},
{
@@ -1039,16 +1011,16 @@
},
{
"cell_type": "code",
"execution_count": 26,
"execution_count": 64,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(140340, 10)"
"(140340, 12)"
]
},
"execution_count": 26,
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
@@ -1068,7 +1040,7 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": 65,
"metadata": {},
"outputs": [
{
@@ -1077,7 +1049,7 @@
"(140340, 12)"
]
},
"execution_count": 27,
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
@@ -1097,6 +1069,256 @@
"We could have: `proper_motion` contains the same data as `pm_phi1_cosphi2` and `pm_phi2`, but in a different format."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exploring data\n",
"\n",
"One benefit of using Pandas is that it provides function for exploring the data and checking for problems.\n",
"\n",
"One of the most useful of these functions is `describe`, which computes summary statistics for each column."
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>source_id</th>\n",
" <th>ra</th>\n",
" <th>dec</th>\n",
" <th>pmra</th>\n",
" <th>pmdec</th>\n",
" <th>parallax</th>\n",
" <th>parallax_error</th>\n",
" <th>radial_velocity</th>\n",
" <th>phi1</th>\n",
" <th>phi2</th>\n",
" <th>pm_phi1</th>\n",
" <th>pm_phi2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>1.403400e+05</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>1.403400e+05</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" <td>140340.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>6.792378e+17</td>\n",
" <td>143.822971</td>\n",
" <td>26.780161</td>\n",
" <td>-2.484410</td>\n",
" <td>-6.100784</td>\n",
" <td>0.179474</td>\n",
" <td>0.518068</td>\n",
" <td>9.931167e+19</td>\n",
" <td>-50.091337</td>\n",
" <td>-1.803264</td>\n",
" <td>-0.868980</td>\n",
" <td>1.409215</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>3.792015e+16</td>\n",
" <td>3.697824</td>\n",
" <td>3.052639</td>\n",
" <td>5.913923</td>\n",
" <td>7.202013</td>\n",
" <td>0.759622</td>\n",
" <td>0.505558</td>\n",
" <td>8.267982e+18</td>\n",
" <td>2.892321</td>\n",
" <td>3.444439</td>\n",
" <td>6.657700</td>\n",
" <td>6.518573</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>6.214900e+17</td>\n",
" <td>135.425699</td>\n",
" <td>19.286617</td>\n",
" <td>-106.755260</td>\n",
" <td>-138.065163</td>\n",
" <td>-15.287602</td>\n",
" <td>0.020824</td>\n",
" <td>-1.792684e+02</td>\n",
" <td>-54.999989</td>\n",
" <td>-8.029159</td>\n",
" <td>-115.275637</td>\n",
" <td>-161.150142</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>6.443515e+17</td>\n",
" <td>140.967807</td>\n",
" <td>24.592348</td>\n",
" <td>-5.038746</td>\n",
" <td>-8.341641</td>\n",
" <td>-0.035983</td>\n",
" <td>0.141108</td>\n",
" <td>1.000000e+20</td>\n",
" <td>-52.603097</td>\n",
" <td>-4.750410</td>\n",
" <td>-2.948851</td>\n",
" <td>-1.107074</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>6.888056e+17</td>\n",
" <td>143.734183</td>\n",
" <td>26.746169</td>\n",
" <td>-1.834971</td>\n",
" <td>-4.689570</td>\n",
" <td>0.362705</td>\n",
" <td>0.336103</td>\n",
" <td>1.000000e+20</td>\n",
" <td>-50.147567</td>\n",
" <td>-1.671497</td>\n",
" <td>0.585038</td>\n",
" <td>1.987196</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>6.976578e+17</td>\n",
" <td>146.607180</td>\n",
" <td>28.990490</td>\n",
" <td>0.452995</td>\n",
" <td>-1.937833</td>\n",
" <td>0.657636</td>\n",
" <td>0.751171</td>\n",
" <td>1.000000e+20</td>\n",
" <td>-47.593466</td>\n",
" <td>1.160632</td>\n",
" <td>3.001761</td>\n",
" <td>4.628859</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>7.974418e+17</td>\n",
" <td>152.777393</td>\n",
" <td>34.285481</td>\n",
" <td>104.319923</td>\n",
" <td>20.981070</td>\n",
" <td>0.999957</td>\n",
" <td>4.171221</td>\n",
" <td>1.000000e+20</td>\n",
" <td>-45.000086</td>\n",
" <td>4.014794</td>\n",
" <td>39.802471</td>\n",
" <td>79.275199</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" source_id ra dec pmra \\\n",
"count 1.403400e+05 140340.000000 140340.000000 140340.000000 \n",
"mean 6.792378e+17 143.822971 26.780161 -2.484410 \n",
"std 3.792015e+16 3.697824 3.052639 5.913923 \n",
"min 6.214900e+17 135.425699 19.286617 -106.755260 \n",
"25% 6.443515e+17 140.967807 24.592348 -5.038746 \n",
"50% 6.888056e+17 143.734183 26.746169 -1.834971 \n",
"75% 6.976578e+17 146.607180 28.990490 0.452995 \n",
"max 7.974418e+17 152.777393 34.285481 104.319923 \n",
"\n",
" pmdec parallax parallax_error radial_velocity \\\n",
"count 140340.000000 140340.000000 140340.000000 1.403400e+05 \n",
"mean -6.100784 0.179474 0.518068 9.931167e+19 \n",
"std 7.202013 0.759622 0.505558 8.267982e+18 \n",
"min -138.065163 -15.287602 0.020824 -1.792684e+02 \n",
"25% -8.341641 -0.035983 0.141108 1.000000e+20 \n",
"50% -4.689570 0.362705 0.336103 1.000000e+20 \n",
"75% -1.937833 0.657636 0.751171 1.000000e+20 \n",
"max 20.981070 0.999957 4.171221 1.000000e+20 \n",
"\n",
" phi1 phi2 pm_phi1 pm_phi2 \n",
"count 140340.000000 140340.000000 140340.000000 140340.000000 \n",
"mean -50.091337 -1.803264 -0.868980 1.409215 \n",
"std 2.892321 3.444439 6.657700 6.518573 \n",
"min -54.999989 -8.029159 -115.275637 -161.150142 \n",
"25% -52.603097 -4.750410 -2.948851 -1.107074 \n",
"50% -50.147567 -1.671497 0.585038 1.987196 \n",
"75% -47.593466 1.160632 3.001761 4.628859 \n",
"max -45.000086 4.014794 39.802471 79.275199 "
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise\n",
"\n",
"Review the summary statistics in this table.\n",
"\n",
"* Do the values makes senses based on what you know about the context?\n",
"\n",
"* Do you see any values that seem problematic, or evidence of other data issues?"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [],
"source": [
"# Solution\n",
"\n",
"# A few issues that are likely to come up:\n",
"\n",
"# 1. Why are some of the parallax values negative?\n",
"# Some parallax measurements are inaccurate, especially\n",
"# stars that are far away.\n",
"\n",
"# 2. Why are some of the radial velocities 1e20?\n",
"# It seems like this value is used to indicate invalid data.\n",
"# Notice that the 25th percentile is 1e20, which indicates\n",
"# that at least 75% of these values are invalid."
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -1441,12 +1663,12 @@
"\n",
"We'll use a simple rectangle for now, but in a later lesson we'll see how to select a polygonal region as well.\n",
"\n",
"Here are bounds on proper motion we chose by eye,"
"Here are bounds on proper motion we chose by eye:"
]
},
{
"cell_type": "code",
"execution_count": 38,
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
@@ -1465,7 +1687,7 @@
},
{
"cell_type": "code",
"execution_count": 39,
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
@@ -1482,7 +1704,7 @@
},
{
"cell_type": "code",
"execution_count": 40,
"execution_count": 46,
"metadata": {},
"outputs": [
{
+50 -329
View File
File diff suppressed because one or more lines are too long
+10 -10
View File
@@ -71,7 +71,7 @@
"IN_COLAB = 'google.colab' in sys.modules\n",
"\n",
"if IN_COLAB:\n",
" !pip install astroquery astro-gala pyia python-wget"
" !pip install astroquery astro-gala python-wget"
]
},
{
@@ -184,16 +184,16 @@
"\n",
"Fortunately, smart people have worked on this problem, and the Gaia database includes cross-matching tables that suggest a best neighbor in the Pan-STARRS catalog for many stars in the Gaia catalog.\n",
"\n",
"[This document describes the cross matching process](https://gea.esac.esa.int/archive/documentation/GDR2/Catalogue_consolidation/chap_cu9val_cu9val/ssec_cu9xma/sssec_cu9xma_extcat.html). Briefly, it uses a cone search to find possible matches in approximately the right position, then uses attributes like color and magnitude to choose pairs of stars most likely to be identical."
"[This document describes the cross matching process](https://gea.esac.esa.int/archive/documentation/GDR2/Catalogue_consolidation/chap_cu9val_cu9val/ssec_cu9xma/sssec_cu9xma_extcat.html). Briefly, it uses a cone search to find possible matches in approximately the right position, then uses attributes like color and magnitude to choose pairs of observations most likely to be the same star."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So the hard part of cross-matching has been done for us. However, using the results is a little tricky.\n",
"## Joining tables\n",
"\n",
"But, it is also an opportunity to learn about one of the most important tools for working with databases: \"joining\" tables.\n",
"So the hard part of cross-matching has been done for us. Using the results is a little tricky, but it gives us a chance to learn about one of the most important tools for working with databases: \"joining\" tables.\n",
"\n",
"In general, a \"join\" is an operation where you match up records from one table with records from another table using as a \"key\" a piece of information that is common to both tables, usually some kind of ID code.\n",
"\n",
@@ -285,8 +285,8 @@
"metadata": {},
"outputs": [],
"source": [
"table = candidate_table[['source_id']]\n",
"table.write('candidate_df.xml', format='votable', overwrite=True)"
"table_id = candidate_table[['source_id']]\n",
"table_id.write('candidate_df.xml', format='votable', overwrite=True)"
]
},
{
@@ -395,8 +395,8 @@
}
],
"source": [
"table = candidate_table[['source_id']]\n",
"type(table)"
"table_id = candidate_table[['source_id']]\n",
"type(table_id)"
]
},
{
@@ -952,8 +952,8 @@
"source": [
"# Solution\n",
"\n",
"table = results1[['source_id', 'original_ext_source_id']]\n",
"table.write('external.xml', format='votable', overwrite=True)"
"table_ext = results1[['source_id', 'original_ext_source_id']]\n",
"table_ext.write('external.xml', format='votable', overwrite=True)"
]
},
{
+509 -6
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -69,7 +69,7 @@
"IN_COLAB = 'google.colab' in sys.modules\n",
"\n",
"if IN_COLAB:\n",
" !pip install astroquery astro-gala pyia python-wget"
" !pip install astroquery astro-gala python-wget"
]
},
{