diff --git a/aas-2021-workshop/01-Search-and-Download-Solar-Data.ipynb b/aas-2021-workshop/01-Search-and-Download-Solar-Data.ipynb new file mode 100644 index 0000000..a8260be --- /dev/null +++ b/aas-2021-workshop/01-Search-and-Download-Solar-Data.ipynb @@ -0,0 +1,2512 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "8b36dd99", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "Searching and Downloading Data with Sunpy\n", + "=========================================\n", + "\n", + "In this notebook we will look at how sunpy can be used to search and download solar physics data. Here is what this notebook contains:\n", + "\n", + "1. Overview of Fido\n", + "2. Attributes of Fido\n", + "3. Constructing a data search query\n", + "4. Inspecting data search query\n", + "5. More complex queries\n", + "6. Using Fido to query the HEK\n", + "7. Downloading data\n", + "8. Extending Fido : New clients and use for your own packages \"logo\"" + ] + }, + { + "cell_type": "markdown", + "id": "6007ebb7", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "\n", + "Fido Unified Downloader\n", + "====================\n", + "\n", + "## 1. Overview of Fido\n", + "\n", + "\n", + "\n", + "* Fido is sunpy's interface for searching and downloading solar physics data.\n", + "\n", + "\n", + "* It offers a unified interface for searching and fetching data irrespective of the underlying client or webservice from where the data is obtained.\n", + "\n", + "\n", + "* Offers a way to search and accesses multiple instruments and all available data providers in a single query.\n", + "\n", + "\n", + "* It supplies a single, easy, consistent and *extendable* way to get most forms of solar physics data the community need \n", + "\n", + "Fido offers access to data available through:\n", + "\n", + " * **VSO**\n", + " * **JSOC** (through `drms`)\n", + " * **Individual data providers** from web accessible sources (http, ftp, etc)\n", + " * **HEK**\n", + " * **HELIO**\n", + " \n", + "Lets first import `Fido` and the `attrs` submodule, and inspect Fido. \n", + "As described here Fido provides access to many sources of data through different `clients`, these clients can be defined inside sunpy or in other packages. Lets print the current list of available clients within sunpy." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "c6596d6f", + "metadata": {}, + "outputs": [], + "source": [ + "from sunpy.net import Fido, attrs as a" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "05c7100b", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "

sunpy.net.Fido

\n", + "

\n", + "Fido is a unified data search and retrieval tool.\n", + "\n", + "It provides simultaneous access to a variety of online data sources, some\n", + "cover multiple instruments and data products like the Virtual Solar\n", + "Observatory and some are specific to a single source.\n", + "\n", + "For details of using `~sunpy.net.Fido` see :ref:`fido_guide`.\n", + "\n", + "

\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
ClientDescription
EVEClientProvides access to Level 0C Extreme ultraviolet Variability Experiment (EVE) data.
GBMClientProvides access to data from the Gamma-Ray Burst Monitor (GBM) instrument on board the Fermi satellite.
XRSClientProvides access to the GOES XRS fits files archive.
SUVIClientProvides access to data from the GOES Solar Ultraviolet Imager (SUVI).
GONGClientProvides access to the Magnetogram products of NSO-GONG synoptic Maps.
......
NoRHClientProvides access to the Nobeyama RadioHeliograph (NoRH) averaged correlation time series data.
RHESSIClientProvides access to the RHESSI observing summary time series data.
HEKClientProvides access to the Heliophysics Event Knowledgebase (HEK).
HECClientProvides access to the HELIO webservices.
JSOCClientProvides access to the JSOC Data Export service.
VSOClientProvides access to query and download from Virtual Solar Observatory (VSO).
\n", + "Length = 15 rows" + ], + "text/plain": [ + "\n", + "sunpy.net.Fido\n", + "\n", + "Fido is a unified data search and retrieval tool.\n", + "\n", + "It provides simultaneous access to a variety of online data sources, some\n", + "cover multiple instruments and data products like the Virtual Solar\n", + "Observatory and some are specific to a single source.\n", + "\n", + "For details of using `~sunpy.net.Fido` see :ref:`fido_guide`.\n", + "\n", + "\n", + " Client ...\n", + "------------ ...\n", + "EVEClient ...\n", + "GBMClient ...\n", + "XRSClient ...\n", + "SUVIClient ...\n", + "GONGClient ...\n", + "... ...\n", + "NoRHClient ...\n", + "RHESSIClient ...\n", + "HEKClient ...\n", + "HECClient ...\n", + "JSOCClient ...\n", + "VSOClient ...\n", + "Length = 15 rows" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "Fido" + ] + }, + { + "cell_type": "markdown", + "id": "eabc9a77", + "metadata": {}, + "source": [ + "## 2. Using attributes to search for data with Fido\n", + "\n", + "Sunpy uses specified *attributes* to search for data using Fido. The range of these attributes is located in the `attrs` submodule. These `attr` parameters can be combined together to construct data search queries, such as searching over a certain time period, for data from a certain instrument with a certain wavelength etc.\n", + "\n", + "Different clients and provides will have client-specific attributes, but the core attributes are:\n", + "\n", + "* `a.Time`\n", + "* `a.Instrument`\n", + "* `a.Wavelength`\n", + "\n", + "\n", + "Lets look at how these attributes work in more detail.\n", + "\n", + "First we can look at a.Time, which is used to specify the timerange of a query." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "82e5945e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "a.Time(\"2021-05-23 02:00\", \"2021-05-23 04:00\")" + ] + }, + { + "cell_type": "markdown", + "id": "8be2a838", + "metadata": {}, + "source": [ + "We can inspect the instrument attribute to see what instrument `attrs` are currently supported through sunpy. Here we can see the instrument name (i.e. the name to be passed to the `a.Instrument` attribute, the client from which the data is available to access, and the full name of the instrument." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "13fdc573", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "

sunpy.net.attrs.Instrument

\n", + "

\n", + "Specifies the Instrument name for the search.\n", + "

\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Attribute NameClientFull NameDescription
aiaVSOAIAAtmospheric Imaging Assembly
bbiVSOBBINone
bcsVSOBCSBragg Crystal Spectrometer
be_continuumVSOBE-ContinuumINAF-OACT Barra Equatoriale Continuum Instrument
be_halphaVSOBE-HalphaINAF-OACT Barra Equatoriale Hα Instrument
bic_hifiVSOBIC-HIFINone
bigbearVSOBig BearBig Bear Solar Observatory, California TON and GONG+ sites
caiiVSOCAIIKanzelhöhe Ca II k Instrument
cdsVSOCDSCoronal Diagnostic Spectrometer
celiasVSOCELIASCharge, Element, and Isotope Analysis System
cerrotololoVSOCerro TololoCerro Tololo, Chile GONG+ site
chpVSOchpChromospheric Helium-I Imaging Photometer
chrotelVSOChroTelChromospheric Telescope
climsoVSOCLIMSOChristian Latouche IMageur SOlaire
cookeVSOCookeNone
costepVSOCOSTEPComprehensive Suprathermal and Energetic Particle Analyzer
cpVSOcpCoronagraph/Polarimeter
dpmVSOdpmDigital Prominence Monitor
eisVSOEISEUV Imaging Spectrometer
eitVSOEITExtreme ultraviolet Imaging Telescope
elteideVSOEl TeideCanary Islands GONG+ site
erneVSOERNEEnergetic and Relativistic Nuclei and Electron experiment
eveEVEEVEExtreme ultraviolet Variability Experiment, which is part of the NASA Solar D...
eveVSOEVEExtreme Ultraviolet Variability Experiment
filmVSOFILMKanzelhöhe Hα Film Camera
five_12_channelmagnetographVSO512-channel magnetograph512-channel Magnetograph
foxsiVSOFOXSIFocusing Optics X-ray Solar Imager
gbmGBMGBMGamma-Ray Burst Monitor on board the Fermi satellite.
gfpiVSOGFPINone
goesXRSGOESThe Geostationary Operational Environmental Satellite Program.
golfVSOGOLFGlobal Oscillations at Low Frequencies
gongGONGGONGGlobal Oscillation Network Group.
grisVSOGRISNone
ha2VSOHA2Kanzelhöhe Hα Patrol Instrument
hi_cVSOHi-CHigh Resolution Coronal Imager (193 A)
hi_c21VSOHi-C21High Resolution Coronal Imager 2.1 (172 A)
hmiVSOHMIHelioseismic and Magnetic Imager
hxeclipseVSOHXECLIPSEUCAR HAO Historical Eclipse Archive
hxtVSOHXTHard X-Ray Telescope
imaxVSOIMaXNone
impactVSOIMPACTIn-situ Measurements of Particles and CME Transients
irisVSOIRISInterface Region Imaging Spectrograph
isoonVSOISOONImproved Solar Observing Optical Network
issVSOISSNone
ivmVSOIVMNone
k_corVSOK-CorNone
kpdcVSOKPDCKanzelhöhe Photosphere Digital Camera
lascoVSOLASCOLarge Angle and Spectrometric Coronagraph
learmonthVSOLearmonthAustralian GONG+ site
longwave_lobe_06VSOLONGWAVE-LOBE-06Full longwave EUV observation of Sun with EUNIS (2006 flight)
longwave_lobe_07VSOLONGWAVE-LOBE-07Full longwave EUV observation of Sun with EUNIS (2007 flight)
longwave_slit_06VSOLONGWAVE-SLIT-06Full longwave EUV observation of Sun with EUNIS (trimmed/2006 flight)
longwave_slit_07VSOLONGWAVE-SLIT-07Full longwave EUV observation of Sun with EUNIS (trimmed/2007 flight)
lyraLYRALYRALyman Alpha Radiometer is the solar UV radiometer on board Proba-2.
maunaloaVSOMauna LoaHawai'ian GONG+ site
mdiVSOMDIMichelson Doppler Imager
meesVSOMEESMees Solar Observatory Imaging Spectrograph on Haleakala, Maui
mergedgongVSOMERGED GONGNone
meudonspectroheliographVSOMeudon SpectroheliographNone
mk4VSOmk4Mk. IV coronagraph
noaa_indicesNOAAIndicesNOAA-IndicesRecent Solar Indices of Observed Monthly Mean Values
noaa_predictNOAAPredictNOAA-PredictPredicted Sunspot Number And Radio Flux Values With Expected Ranges.
norhNoRHNORHNobeyama Radio Heliograph is an imaging radio telescope at 17 or 34GHz locate...
ovsaVSOOVSAOwens Valley Solar Array
phoenixVSOPHOENIXPHOENIX detector on flight #3 of FOXSI
phokaVSOPHOKAKanzelhöhe Photoheliograph
plasticVSOPLASTICPLasma And SupraThermal Ion Composition
ptmcVSOPTMCNone
rhessiRHESSIRHESSIReuven Ramaty High Energy Solar Spectroscopic Imager.
rhessiVSORHESSIReuven Ramaty High Energy Solar Spectroscopic Imager
secchiVSOSECCHISun Earth Connection Coronal and Heliospheric Investigation
shortwave_lobe_06VSOSHORTWAVE-LOBE-06Shortwave EUV observation of Sun with EUNIS (2006 flight)
shortwave_lobe_07VSOSHORTWAVE-LOBE-07Shortwave EUV observation of Sun with EUNIS (2007 flight)
shortwave_slit_06VSOSHORTWAVE-SLIT-06Shortwave EUV observation of Sun with EUNIS (trimmed/2006 flight)
shortwave_slit_07VSOSHORTWAVE-SLIT-07Shortwave EUV observation of Sun with EUNIS (trimmed/2007 flight)
six_0_ftshgVSO60-ft SHG60-foot Tower Spectroheliograph
sjVSOSJSlit-Jaw
solarftsspectrometerVSOsolar fts spectrometerSolar FTS Spectrometer
soonSRSSOONSolar Region Summary.
sotVSOSOTSolar Optical Telescope
sp1VSOSP1Spectropolarimeter1
sp2VSOSP2Spectropolarimeter2
spectroheliographVSOspectroheliographSpectroHeliograph
spectromagnetographVSOspectromagnetographSpectroMagnetograph
srs_tableSRSSRS-TableSolar Region Summary.
sumerVSOSUMERSolar Ultraviolet Measurements of Emitted Radiation
suviSUVISUVIGOES Solar Ultraviolet Imager.
suviVSOSUVIGOES-R Series Solar Ultraviolet Imager
swanVSOSWANSolar Wind Anisotropies
swapVSOSWAPSun Watcher using Active pixel system detector and image Processing
swavesVSOSWAVESSTEREO/WAVES
sxi_0VSOSXI-0Solar X-ray Imager
sxtVSOSXTSoft X-Ray Telescope
tm_1001VSOTM-1001Kanzelhöhe Hα Pulnix TM-1001 Camera
tm_1010VSOTM-1010Kanzelhöhe Hα Pulnix TM-1010 Camera
traceVSOTRACETransition Region And Coronal Explorer
udaipurVSOUdaipurIndian GONG+ site
uvcsVSOUVCSUltraviolet Coronagraph Spectrometer
vault_1999VSOVAULT-1999Very High Angular Resolution Ultraviolet Telescope (1999 flight)
vault_2002VSOVAULT-2002Very High Angular Resolution Ultraviolet Telescope (2002 flight)
vault_2014VSOVAULT-2014Very High Angular Resolution Ultraviolet Telescope (2014 flight)
virgoVSOVIRGOVariability of Solar Irradiance and Gravity Oscillations
vsmVSOvsmVector SpecroMagnetograph
wbsVSOWBSWide Band Spectrometer
wisprVSOWISPRWide-Field Imager for Solar Probe Plus
x123VSOX123X123 SXR Spectometer
xrsXRSXRSGOES X-ray Sensor
xrtVSOXRTX-Ray Telescope
zimpolVSOZIMPOLNone
" + ], + "text/plain": [ + "\n", + "sunpy.net.attrs.Instrument\n", + "\n", + "Specifies the Instrument name for the search.\n", + "\n", + " Attribute Name ...\n", + "--------------------------- ...\n", + "aia ...\n", + "bbi ...\n", + "bcs ...\n", + "be_continuum ...\n", + "be_halpha ...\n", + "bic_hifi ...\n", + "bigbear ...\n", + "caii ...\n", + "cds ...\n", + "celias ...\n", + "cerrotololo ...\n", + "chp ...\n", + "chrotel ...\n", + "climso ...\n", + "cooke ...\n", + "costep ...\n", + "cp ...\n", + "dpm ...\n", + "eis ...\n", + "eit ...\n", + "elteide ...\n", + "erne ...\n", + "eve ...\n", + "eve ...\n", + "film ...\n", + "five_12_channelmagnetograph ...\n", + "foxsi ...\n", + "gbm ...\n", + "gfpi ...\n", + "goes ...\n", + "golf ...\n", + "gong ...\n", + "gris ...\n", + "ha2 ...\n", + "hi_c ...\n", + "hi_c21 ...\n", + "hmi ...\n", + "hxeclipse ...\n", + "hxt ...\n", + "imax ...\n", + "impact ...\n", + "iris ...\n", + "isoon ...\n", + "iss ...\n", + "ivm ...\n", + "k_cor ...\n", + "kpdc ...\n", + "lasco ...\n", + "learmonth ...\n", + "longwave_lobe_06 ...\n", + "longwave_lobe_07 ...\n", + "longwave_slit_06 ...\n", + "longwave_slit_07 ...\n", + "lyra ...\n", + "maunaloa ...\n", + "mdi ...\n", + "mees ...\n", + "mergedgong ...\n", + "meudonspectroheliograph ...\n", + "mk4 ...\n", + "noaa_indices ...\n", + "noaa_predict ...\n", + "norh ...\n", + "ovsa ...\n", + "phoenix ...\n", + "phoka ...\n", + "plastic ...\n", + "ptmc ...\n", + "rhessi ...\n", + "rhessi ...\n", + "secchi ...\n", + "shortwave_lobe_06 ...\n", + "shortwave_lobe_07 ...\n", + "shortwave_slit_06 ...\n", + "shortwave_slit_07 ...\n", + "six_0_ftshg ...\n", + "sj ...\n", + "solarftsspectrometer ...\n", + "soon ...\n", + "sot ...\n", + "sp1 ...\n", + "sp2 ...\n", + "spectroheliograph ...\n", + "spectromagnetograph ...\n", + "srs_table ...\n", + "sumer ...\n", + "suvi ...\n", + "suvi ...\n", + "swan ...\n", + "swap ...\n", + "swaves ...\n", + "sxi_0 ...\n", + "sxt ...\n", + "tm_1001 ...\n", + "tm_1010 ...\n", + "trace ...\n", + "udaipur ...\n", + "uvcs ...\n", + "vault_1999 ...\n", + "vault_2002 ...\n", + "vault_2014 ...\n", + "virgo ...\n", + "vsm ...\n", + "wbs ...\n", + "wispr ...\n", + "x123 ...\n", + "xrs ...\n", + "xrt ...\n", + "zimpol ..." + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "a.Instrument" + ] + }, + { + "cell_type": "markdown", + "id": "f1e5dded", + "metadata": {}, + "source": [ + "Sunpy also now provides tab completion to auto-fill the attribute name" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "705a3d84", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "a.Instrument.eit" + ] + }, + { + "cell_type": "markdown", + "id": "28a90a90", + "metadata": {}, + "source": [ + "To search for certain wavelengths, we need to specify the input as an `astropy Quantity` which is a the combination of a value and an associated unit. This is something is universal in the sunpy stack - that every physical input/output is a `Quantity`." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "9436a819", + "metadata": {}, + "outputs": [], + "source": [ + "from astropy import units as u" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "d234fae7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/latex": [ + "$171 \\; \\mathrm{\\mathring{A}}$" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "171*u.angstrom" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "ce1faff9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "a.Wavelength(17.1*u.angstrom)" + ] + }, + { + "cell_type": "markdown", + "id": "ede00dbf", + "metadata": {}, + "source": [ + "## 3. Constructing a search query\n", + " ### A simple query\n", + "\n", + "Lets create a simple query to search for data from AIA over a particular time period" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "1bd72a37", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: SunpyUserWarning: VSO-C500 :soap:Server.Transport : 200 OK [sunpy.net.vso.vso]\n" + ] + } + ], + "source": [ + "result = Fido.search(a.Time(\"2021-05-23 11:00\", \"2021-05-23 11:20\"), \n", + " a.Instrument(\"AIA\"))" + ] + }, + { + "cell_type": "markdown", + "id": "536c9dec", + "metadata": {}, + "source": [ + "The `Fido.search` will return `UnifiedResponse` object with all the information available from the search that fit the criteria specified by the attrs. This `result` can then be passed to `Fido.fetch` to download the queried data (We will discuss this further below). Lets now inspect our `result` which holds the results of our query, if we print this we should see a summary of our query:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "b7816751", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "Results from 1 Provider:

801 Results from the VSOClient:
VSOQueryResponseTable length=801\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Start TimeEnd TimeSourceInstrumentWavelength [2]ProviderPhysobsWavetypeExtent WidthExtent LengthExtent TypeSizeInfo
AngstromMibyte
objectobjectstr3str3float64str4str9str6str4str4str8float64str57
2021-05-23 11:00:00.0002021-05-23 11:00:01.000SDOAIA335.0 .. 335.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:00:04.0002021-05-23 11:00:05.000SDOAIA193.0 .. 193.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:05.0002021-05-23 11:00:06.000SDOAIA304.0 .. 304.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.902 exposure] [100.00 percentd]
2021-05-23 11:00:05.0002021-05-23 11:00:06.000SDOAIA4500.0 .. 4500.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [0.300 exposure] [100.00 percentd]
2021-05-23 11:00:06.0002021-05-23 11:00:07.000SDOAIA131.0 .. 131.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA211.0 .. 211.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:00:11.0002021-05-23 11:00:12.000SDOAIA94.0 .. 94.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:00:12.0002021-05-23 11:00:13.000SDOAIA335.0 .. 335.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
.......................................
2021-05-23 11:19:47.0002021-05-23 11:19:48.000SDOAIA94.0 .. 94.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:19:48.0002021-05-23 11:19:49.000SDOAIA335.0 .. 335.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:19:50.0002021-05-23 11:19:51.000SDOAIA1600.0 .. 1600.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:19:52.0002021-05-23 11:19:53.000SDOAIA193.0 .. 193.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:53.0002021-05-23 11:19:54.000SDOAIA304.0 .. 304.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.902 exposure] [100.00 percentd]
2021-05-23 11:19:54.0002021-05-23 11:19:55.000SDOAIA131.0 .. 131.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:19:57.0002021-05-23 11:19:58.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:57.0002021-05-23 11:19:58.000SDOAIA211.0 .. 211.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:19:59.0002021-05-23 11:20:00.000SDOAIA94.0 .. 94.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]
2021-05-23 11:20:00.0002021-05-23 11:20:01.000SDOAIA335.0 .. 335.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.901 exposure] [100.00 percentd]

" + ], + "text/plain": [ + "\n", + "Results from 1 Provider:\n", + "\n", + "801 Results from the VSOClient:\n", + " Start Time ...\n", + " ...\n", + "----------------------- ...\n", + "2021-05-23 11:00:00.000 ...\n", + "2021-05-23 11:00:04.000 ...\n", + "2021-05-23 11:00:05.000 ...\n", + "2021-05-23 11:00:05.000 ...\n", + "2021-05-23 11:00:06.000 ...\n", + "2021-05-23 11:00:09.000 ...\n", + "2021-05-23 11:00:09.000 ...\n", + "2021-05-23 11:00:11.000 ...\n", + "2021-05-23 11:00:12.000 ...\n", + "2021-05-23 11:00:14.000 ...\n", + " ... ...\n", + "2021-05-23 11:19:47.000 ...\n", + "2021-05-23 11:19:48.000 ...\n", + "2021-05-23 11:19:50.000 ...\n", + "2021-05-23 11:19:52.000 ...\n", + "2021-05-23 11:19:53.000 ...\n", + "2021-05-23 11:19:54.000 ...\n", + "2021-05-23 11:19:57.000 ...\n", + "2021-05-23 11:19:57.000 ...\n", + "2021-05-23 11:19:59.000 ...\n", + "2021-05-23 11:20:00.000 ...\n", + "Length = 801 rows\n" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result" + ] + }, + { + "cell_type": "markdown", + "id": "1f213a9f", + "metadata": {}, + "source": [ + "Now lets make our query a bit more specific, say, say we only want one wavelength band from AIA. This can be achieved by specifying the `Wavelength` attribute within the search. The `Wavelength` attribute is passed as an astropy Quantity:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "abfbe41d", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: SunpyUserWarning: VSO-C500 :soap:Server.Transport : 200 OK [sunpy.net.vso.vso]\n" + ] + } + ], + "source": [ + "result = Fido.search(a.Time(\"2021-05-23 11:00\", \"2021-05-23 11:20\"), \n", + " a.Instrument(\"AIA\"), \n", + " a.Wavelength(171*u.angstrom))" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "1475b4ce", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "Results from 1 Provider:

100 Results from the VSOClient:
VSOQueryResponseTable length=100\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Start TimeEnd TimeSourceInstrumentWavelength [2]ProviderPhysobsWavetypeExtent WidthExtent LengthExtent TypeSizeInfo
AngstromMibyte
objectobjectstr3str3float64str4str9str6str4str4str8float64str57
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:21.0002021-05-23 11:00:22.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:33.0002021-05-23 11:00:34.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:45.0002021-05-23 11:00:46.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:00:57.0002021-05-23 11:00:58.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:01:09.0002021-05-23 11:01:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:01:21.0002021-05-23 11:01:22.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:01:33.0002021-05-23 11:01:34.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:01:45.0002021-05-23 11:01:46.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
.......................................
2021-05-23 11:18:09.0002021-05-23 11:18:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:18:21.0002021-05-23 11:18:22.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:18:33.0002021-05-23 11:18:34.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:18:45.0002021-05-23 11:18:46.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:18:57.0002021-05-23 11:18:58.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:09.0002021-05-23 11:19:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:21.0002021-05-23 11:19:22.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:33.0002021-05-23 11:19:34.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:45.0002021-05-23 11:19:46.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:19:57.0002021-05-23 11:19:58.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]

" + ], + "text/plain": [ + "\n", + "Results from 1 Provider:\n", + "\n", + "100 Results from the VSOClient:\n", + " Start Time ...\n", + " ...\n", + "----------------------- ...\n", + "2021-05-23 11:00:09.000 ...\n", + "2021-05-23 11:00:21.000 ...\n", + "2021-05-23 11:00:33.000 ...\n", + "2021-05-23 11:00:45.000 ...\n", + "2021-05-23 11:00:57.000 ...\n", + "2021-05-23 11:01:09.000 ...\n", + "2021-05-23 11:01:21.000 ...\n", + "2021-05-23 11:01:33.000 ...\n", + "2021-05-23 11:01:45.000 ...\n", + "2021-05-23 11:01:57.000 ...\n", + " ... ...\n", + "2021-05-23 11:18:09.000 ...\n", + "2021-05-23 11:18:21.000 ...\n", + "2021-05-23 11:18:33.000 ...\n", + "2021-05-23 11:18:45.000 ...\n", + "2021-05-23 11:18:57.000 ...\n", + "2021-05-23 11:19:09.000 ...\n", + "2021-05-23 11:19:21.000 ...\n", + "2021-05-23 11:19:33.000 ...\n", + "2021-05-23 11:19:45.000 ...\n", + "2021-05-23 11:19:57.000 ...\n", + "Length = 100 rows\n" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result" + ] + }, + { + "cell_type": "markdown", + "id": "414f2640", + "metadata": {}, + "source": [ + "We can further specify this query by choosing the cadence (time-sampling) of the data we want to search for and download. This can be achieved by using the `Sample` attribute. Similar to the `Wavelength` attributes, this needs to be an `astropy Quantity`. Lets further specify the search above to only search for data with a cadence of 2 minutes." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "5fb4038b", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: SunpyUserWarning: VSO-C500 :soap:Server.Transport : 200 OK [sunpy.net.vso.vso]\n" + ] + } + ], + "source": [ + "result = Fido.search(a.Time(\"2021-05-23 11:00\", \"2021-05-23 11:20\"), \n", + " a.Instrument(\"AIA\"), \n", + " a.Wavelength(171*u.angstrom),\n", + " a.Sample(2*u.min))" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "e8a81bc5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "Results from 1 Provider:

10 Results from the VSOClient:
VSOQueryResponseTable length=10\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Start TimeEnd TimeSourceInstrumentWavelength [2]ProviderPhysobsWavetypeExtent WidthExtent LengthExtent TypeSizeInfo
AngstromMibyte
objectobjectstr3str3float64str4str9str6str4str4str8float64str57
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:02:09.0002021-05-23 11:02:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:04:09.0002021-05-23 11:04:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:06:09.0002021-05-23 11:06:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:08:09.0002021-05-23 11:08:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:10:09.0002021-05-23 11:10:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:12:09.0002021-05-23 11:12:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:14:09.0002021-05-23 11:14:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:16:09.0002021-05-23 11:16:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
2021-05-23 11:18:09.0002021-05-23 11:18:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]

" + ], + "text/plain": [ + "\n", + "Results from 1 Provider:\n", + "\n", + "10 Results from the VSOClient:\n", + " Start Time ...\n", + " ...\n", + "----------------------- ...\n", + "2021-05-23 11:00:09.000 ...\n", + "2021-05-23 11:02:09.000 ...\n", + "2021-05-23 11:04:09.000 ...\n", + "2021-05-23 11:06:09.000 ...\n", + "2021-05-23 11:08:09.000 ...\n", + "2021-05-23 11:10:09.000 ...\n", + "2021-05-23 11:12:09.000 ...\n", + "2021-05-23 11:14:09.000 ...\n", + "2021-05-23 11:16:09.000 ...\n", + "2021-05-23 11:18:09.000 ...\n" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result" + ] + }, + { + "cell_type": "markdown", + "id": "7c0a5c3e", + "metadata": {}, + "source": [ + "## 4. Dealing with search results and the `UnifiedResponse`\n", + "\n", + "So far we have shown how queries can be constructed and returned as a `UnifiedResponse` object. This object can then be passed to `fido.fetch` to download the data. However lets first inspect the `UnifiedResponse` object to get a better idea of what it consists of. \n", + "\n", + "From above, we can see that what is returned from a `Fido.search` is the `UnifiedResponse` object that consists of a one record (for a simple query) or multiple records depending on the search. These records (stored as tables) can be accessed by indexing the `UnifiedResponse` object. It acts both like a two dimensional array, where the first dimension is the response index and the second index is the row index, and a dictionary where you can index the responses by the name of the client. \n", + "\n", + "\n", + "In this case here, we only have one query returned as it was a simple query for AIA data" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "375011d9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "10" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result.file_num" + ] + }, + { + "cell_type": "markdown", + "id": "38f8a5aa", + "metadata": {}, + "source": [ + "As mentioned above the first index is the response index, and the second is the row, so we can pull out the individual rows. In this example, there is only one response (a simple query)\n", + "\n", + "So we can index to get the first row of the query results" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "8606b1fe", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "QueryResponseRow index=0\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Start TimeEnd TimeSourceInstrumentWavelength [2]ProviderPhysobsWavetypeExtent WidthExtent LengthExtent TypeSizeInfofileid
AngstromMibyte
objectobjectstr3str3float64str4str9str6str4str4str8float64str57str24
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]aia__lev1:171:1400842847
" + ], + "text/plain": [ + "\n", + " Start Time End Time Source Instrument Wavelength [2] Provider Physobs Wavetype Extent Width Extent Length Extent Type Size Info fileid \n", + " Angstrom Mibyte \n", + " object object str3 str3 float64 str4 str9 str6 str4 str4 str8 float64 str57 str24 \n", + "----------------------- ----------------------- ------ ---------- -------------- -------- --------- -------- ------------ ------------- ----------- -------- --------------------------------------------------------- ------------------------\n", + "2021-05-23 11:00:09.000 2021-05-23 11:00:10.000 SDO AIA 171.0 .. 171.0 JSOC intensity NARROW 4096 4096 FULLDISK 64.64844 AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd] aia__lev1:171:1400842847" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result[0, 0]" + ] + }, + { + "cell_type": "markdown", + "id": "96c16560", + "metadata": {}, + "source": [ + "The normal slices operations that work with other Python sequences will also work here (i.e. result[0, ::5] will return every fifth row etc" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "87d437b3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "VSOQueryResponseTable length=1\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
Start TimeEnd TimeSourceInstrumentWavelength [2]ProviderPhysobsWavetypeExtent WidthExtent LengthExtent TypeSizeInfo
AngstromMibyte
objectobjectstr3str3float64str4str9str6str4str4str8float64str57
2021-05-23 11:00:09.0002021-05-23 11:00:10.000SDOAIA171.0 .. 171.0JSOCintensityNARROW40964096FULLDISK64.64844AIA level 1, 4096x4096 [2.000 exposure] [100.00 percentd]
" + ], + "text/plain": [ + "\n", + " Start Time ...\n", + " ...\n", + "----------------------- ...\n", + "2021-05-23 11:00:09.000 ..." + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result[0,::50]" + ] + }, + { + "cell_type": "markdown", + "id": "d20dc0ff", + "metadata": {}, + "source": [ + "### Inspecting the result tables\n", + "\n", + "Each of these tables indexed from the `UnifiedResponse` object are `QueryResponseTable` objects which are just `astropy.table` objects that can easily interacted with (e.g. indexed, filtered etc).\n", + "\n", + "This is particularly handy when the results that are searched for are from metadata tables such as the HEK! But lets first look at how we can interact with the results table, and filter etc. Lets first look at how we can play around with each table" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "33c8b73f", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Start Time',\n", + " 'End Time',\n", + " 'Source',\n", + " 'Instrument',\n", + " 'Wavelength',\n", + " 'Provider',\n", + " 'Physobs',\n", + " 'Wavetype',\n", + " 'Extent Width',\n", + " 'Extent Length',\n", + " 'Extent Type',\n", + " 'Size',\n", + " 'Info',\n", + " 'fileid']" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result[0].colnames" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "5c2c8023", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "