jsoc_data_export.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSOC data export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import drms\n",
    "from astropy.io import fits\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import dates\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the download directory, if it does not exist yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_dir = 'downloads'\n",
    "if not os.path.exists(out_dir):\n",
    "    os.makedirs(out_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the `myemail` address below to your own email address. Note that you need to register you email address on the [JSOC email registration](http://jsoc.stanford.edu/ajax/register_email.html) webpage first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "myemail = 'name@example.com'\n",
    "client = drms.Client(email=myemail, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download a HMI full-disk image\n",
    "\n",
    "Export requests are created using the [`drms.Client.export()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.export.html) method. The default protocol is `'as-is'` which is the fastest way to export files from JSOC, but does not include the metadata from the DRMS database in the FITS header. If metadata is required in the FITS headers, the `protocol='fits'` argument must be specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "request = client.export('hmi.ic_720s[2015.12.19_00:00_TAI]')\n",
    "print(request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = request.download(out_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the downloaded FITS file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fname_ic = res.download[0]\n",
    "img_data = fits.getdata(fname_ic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "im = ax.imshow(img_data/1e3, cmap='gray', origin='lower')\n",
    "fig.colorbar(im, label='Intensity [kDN/s]')\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ax.set_xlim(1900, 2200)\n",
    "ax.set_ylim(1440, 1740)\n",
    "plt.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open and inspect the downloaded FITS file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = fits.open(fname_ic)\n",
    "f.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f[1].header"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export FITS files with metadata\n",
    "\n",
    "In order to download FITS files with header data, it is necessary to use the `protocol='fits'` argument when calling [`drms.Client.export()`](http://drms.readthedocs.io/en/stable/generated/drms.Client.export.html). This creates a 'real' export request, where new FITS files are created at Stanford, containing the image data from the stored files combined with the metadata from the DRMS database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "request = client.export('hmi.ic_720s[2015.12.19_01:00_TAI]', protocol='fits')\n",
    "print(request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(request.request_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = request.download(out_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open and inspect the new downloaded FITS file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fname_ic2 = res.download[0]\n",
    "f2 = fits.open(fname_ic2)\n",
    "f2.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS files from JSOC, that include metadata in their headers are unfortunately not standard-conform. In order to read them with `astropy.io.fits`, it is necessary to fix their headers first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "f2[1].verify('silentfix')\n",
    "f2[1].header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f2[1].data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synoptic magnetic map\n",
    "\n",
    "This example shows how to download synoptic magnetic maps from JSOC and query the metadata separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "series = 'hmi.synoptic_mr_720s'\n",
    "info = client.info(series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info.primekeys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "info.segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cr = 2150\n",
    "segment = 'synopMr'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Query the metadata for CR 2150:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "qstr = '{}[{}]'.format(series, cr)\n",
    "\n",
    "print('Querying \"{}\"...'.format(qstr))\n",
    "keys = client.query(qstr, key=drms.const.all)\n",
    "print('-> {} line(s) retrieved.'.format(len(keys)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "keys.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform an `'as-is'` export using only the `'synopMr'` segment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "expstr = qstr + '{' + segment + '}'\n",
    "\n",
    "print('Export request: \"{}\"'.format(expstr))\n",
    "request = client.export(expstr)\n",
    "print(request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = request.download(out_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is only one record / one file, so we can just use the first row here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "h = keys.iloc[0]\n",
    "fname_syn = res.download[0]\n",
    "#print(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the image data from the downloaded FITS file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = fits.getdata(fname_syn)\n",
    "ny, nx = data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the metadata from the DRMS query to transform pixel to world coordinates and determine the correct extent and aspect ratio for the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert pixel to world coordinates using WCS keywords\n",
    "xmin = (1 - h.CRPIX1)*h.CDELT1 + h.CRVAL1\n",
    "xmax = (nx - h.CRPIX1)*h.CDELT1 + h.CRVAL1\n",
    "ymin = (1 - h.CRPIX2)*h.CDELT2 + h.CRVAL2\n",
    "ymax = (ny - h.CRPIX2)*h.CDELT2 + h.CRVAL2\n",
    "\n",
    "# Convert to Carrington longitude\n",
    "xmin = h.LON_LAST - xmin\n",
    "xmax = h.LON_LAST - xmax\n",
    "\n",
    "# Compute the plot extent used with imshow\n",
    "extent = (xmin - abs(h.CDELT1)/2, xmax + abs(h.CDELT1)/2,\n",
    "          ymin - abs(h.CDELT2)/2, ymax + abs(h.CDELT2)/2)\n",
    "\n",
    "# Aspect ratio for imshow in respect to the extent computed above\n",
    "aspect = abs((xmax - xmin)/nx * ny/(ymax - ymin))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(9, 5))\n",
    "\n",
    "title_str = '{}, Time: {} ... {}'.format(qstr, h.T_START, h.T_STOP)\n",
    "ax.set_title(title_str, fontsize='medium')\n",
    "\n",
    "ax.imshow(data, vmin=-300, vmax=300, origin='lower', cmap='gray', extent=extent, aspect=aspect)\n",
    "ax.invert_xaxis()\n",
    "ax.set_xlabel('Carrington longitude')\n",
    "ax.set_ylabel('Sine latitude')\n",
    "\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}