Using-Cactus/Cactus-Funwave.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![cactus](http://cactuscode.org/global/images/cactuslogo.png)\n",
    "# Compiling Cactus!\n",
    "Step 1 is to download the code. Cactus uses a script named \"GetComponents\" to find and prepare all the source code modules that it needs for a given installation. The GetComponents script can be downloaded with a simple invocation of curl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "!curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2016_11/GetComponents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2 is to download your thornlist. In this tutorial, we are going to use Funwave, a collection of thorns\n",
    "designed to simulate water waves using the Boussinesq equations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -kLO https://bitbucket.org/stevenrbrandt/cajunwave/raw/master/funwave_carpet.th"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view a file in the notebook by using the \"magic\" command \"%pycat filename\". However, it tries to highlight\n",
    "syntax as if the file is written in python. In those cases you can simply use \"%cat filename.\" Unfortunately, unlike %pycat, %cat leaves the contents of the file on the screen.\n",
    "\n",
    "Note that at the top of the file is \"DEFINE_ROOT = CactusFW2\". This means that Cactus, and all its thorns, will be checked out under that directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat ~/funwave_carpet.th"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to checkout the components listed in the thornlist. We do this with the GetComponents command.\n",
    "Before we can execute it, however, we need to turn on its execute bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "!chmod a+x GetComponents\n",
    "!echo no|./GetComponents --parallel funwave_carpet.th"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://simfactory.org/global/images/factory.png\" width=\"200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simfactory\n",
    "Cactus is normally built with a tool called Simfactory. Simfactory, in turn, will call make.\n",
    "Before it can work, however, it needs to be configured. Please replace... my email address in the\n",
    "command below with yours. The email address isn't sent anywhere, all it's used for is allowing\n",
    "Cactus to send job change state notifications to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2\n",
    "!./simfactory/bin/sim setup-silent --setup-email=sbrandt@cct.lsu.edu "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At long last, we are ready to actually build Cactus. Cactus can often figure out what compilers and build\n",
    "options to use automatically, but in some cases it is necessary to specify it by hand (you can do this by adding --optionlist=centos.cfg to the build command below). The file containing\n",
    "this information is called the Option List. You might want to take a look at it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the command to build Cactus using our thornlist. As written, it will build in parallel using two processes. That's what the -j option does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!rm -fr configs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!time ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=./repos/cajunwave/funwave_carpet.th | cat -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The build command creates a configuration called \"sim\". It is found in the \"configs/sim\" directory. One of the files in this directory is the ThornList. It contains the list of thorns Cactus will compile. If you wish to add or remove a thorn from your configuration, you can do it by editing this file. However, by doing so you risk confusing yourself by forgetting what you've done. Proceed at your own risk!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%ls ~/CactusFW2/configs/sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat ~/CactusFW2/configs/sim/ThornList"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"OptionList\" file contains all the configuration options (the things you saw in centos.cfg). Unlike the ThornList file, however, changing this file will have no effect. If you wish to change your configuration options without starting over from scratch, you should edit the file \"configs/sim/config-data/make.config.defn.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat ~/CactusFW2/configs/sim/config-data/make.config.defn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Running Cactus!</h1>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we are going to run a simple Gaussian water wave over a flat seabed. We will use MPI and run on two processes. You can edit the parameter file below and hit shift-Enter to write it to disk. The special sequence \"%%writefile filename\" at the top makes this possible. Alternatively, you can load an existing file by putting the special sequence \"%load filename\" at the top of a cell and hitting shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ~/CactusFW2/wave.par\n",
    "\n",
    "#Reorder the parameters for easy comparison to the input.txt in example 3\n",
    "ActiveThorns = \"\n",
    "  CoordBase FunWave FunwaveCoord CartGrid3D Carpet CarpetIOASCII\n",
    "  CartGrid3D IOUtil CarpetIOBasic CarpetSlab Boundary SymBase MoL\n",
    "  CarpetReduce LocalReduce InitBase CarpetLib LoopControl Tridiagonal\n",
    "  CarpetIOScalar \"\n",
    "\n",
    "#----------------------------------------------------\n",
    "# Flesh and CCTK parameters\n",
    "#----------------------------------------------------\n",
    "\n",
    "# flesh\n",
    "Cactus::cctk_run_title = \"Test Run\"\n",
    "Cactus::cctk_show_schedule = \"yes\"\n",
    "Cactus::cctk_itlast = 300\n",
    "Cactus::allow_mixeddim_gfs = \"yes\"\n",
    "\n",
    "# CartGrid3D\n",
    "CartGrid3D::type = \"coordbase\"\n",
    "CartGrid3D::avoid_origin = \"no\"\n",
    "CoordBase::domainsize = \"minmax\"\n",
    "CoordBase::spacing    = \"gridspacing\"\n",
    "CoordBase::xmin =  0\n",
    "CoordBase::xmax =  30\n",
    "CoordBase::ymin =  0\n",
    "CoordBase::ymax =  30\n",
    "CoordBase::zmin =  0.0\n",
    "CoordBase::zmax =  0.0\n",
    "CoordBase::dx   =  0.25\n",
    "CoordBase::dy   =  0.25\n",
    "\n",
    "CoordBase::boundary_size_x_lower     = 3\n",
    "CoordBase::boundary_size_x_upper     = 3\n",
    "CoordBase::boundary_size_y_lower     = 3\n",
    "CoordBase::boundary_size_y_upper     = 3\n",
    "CoordBase::boundary_size_z_lower     = 0\n",
    "CoordBase::boundary_size_z_upper     = 0\n",
    "CoordBase::boundary_shiftout_x_lower = 1\n",
    "CoordBase::boundary_shiftout_x_upper = 1\n",
    "CoordBase::boundary_shiftout_y_lower = 1\n",
    "CoordBase::boundary_shiftout_y_upper = 1\n",
    "CoordBase::boundary_shiftout_z_lower = 1\n",
    "CoordBase::boundary_shiftout_z_upper = 1\n",
    "\n",
    "# Carpet\n",
    "Carpet::domain_from_coordbase = \"yes\"\n",
    "Carpet::ghost_size_x = 3\n",
    "Carpet::ghost_size_y = 3\n",
    "Carpet::ghost_size_z = 1\n",
    "carpet::adaptive_stepsize = yes\n",
    "\n",
    "# MoL\n",
    "MoL::ODE_Method = \"RK3\"\n",
    "MoL::disable_prolongation        = \"yes\"\n",
    "\n",
    "# the output dir will be named after the parameter file name\n",
    "IO::out_dir = $parfile\n",
    "IO::out_fileinfo=\"none\"\n",
    "IOBasic::outInfo_every = 1\n",
    "IOBasic::outInfo_vars = \"FunWave::eta FunWave::u FunWave::v\"\n",
    "\n",
    "#IOASCII::out1D_every = 1\n",
    "#IOASCII::out1d_vars = \"FunWave::eta Funwave::depth\"\n",
    "CarpetIOASCII::compact_format = false\n",
    "IOASCII::out2D_every = 30\n",
    "IOASCII::out2D_xyplane_z = 0\n",
    "IOASCII::out2D_vars = \"FunWave::eta FunWave::u FunWave::v\"\n",
    "IOASCII::out2D_xz = \"no\"\n",
    "IOASCII::out2D_yz = \"no\"\n",
    "IOASCII::output_ghost_points = \"no\"\n",
    "\n",
    "IOScalar::outScalar_every = 1\n",
    "IOScalar::outScalar_vars = \"FunWave::eta FunWave::u FunWave::v\"\n",
    "\n",
    "#& = \"Funwave::eta\"\n",
    "\n",
    "#----------------------------------------------------\n",
    "# Funwave parameters\n",
    "#----------------------------------------------------\n",
    "\n",
    "# Funwave depth \n",
    "FunWave::depth_file_offset_x = 3\n",
    "FunWave::depth_file_offset_y = 3\n",
    "FunWave::depth_type = \"flat\"\n",
    "FunWave::depth_format = \"ele\"\n",
    "FunWave::depth_file = \"/tmp/__depth__.txt\"\n",
    "FunWave::depth_flat = 0.8\n",
    "#Funwave::test_depth_shore_x = 80\n",
    "#Funwave::test_depth_island_x = 40\n",
    "#Funwave::test_depth_island_y = 40\n",
    "FunWave::depth_xslp = 10.0\n",
    "FunWave::depth_slope = 0.05\n",
    "FunWave::dt_size = 0\n",
    "Funwave::generate_test_depth_data = true\n",
    "Funwave::num_wave_components = 1\n",
    "Funwave::wave_component_file = \"/home/sbrandt/workspace/shi_funwave/example_2/fft/wavemk_per_amp_pha.txt\"\n",
    "Funwave::peak_period = 1\n",
    "\n",
    "# import\n",
    "Funwave::time_ramp = 1.0\n",
    "Funwave::delta_wk = 0.5\n",
    "Funwave::dep_wk = 0.45\n",
    "Funwave::xc_wk = 3.0\n",
    "Funwave::ywidth_wk = 10000.0\n",
    "Funwave::tperiod = 1.0\n",
    "Funwave::amp_wk = 0.0232\n",
    "Funwave::theta_wk = 0.0\n",
    "Funwave::freqpeak = 0.2\n",
    "Funwave::freqmin = 0.1\n",
    "Funwave::freqmax = 0.4\n",
    "Funwave::hmo = 1.0\n",
    "Funwave::gammatma = 5.0\n",
    "Funwave::thetapeak = 10.0\n",
    "Funwave::sigma_theta = 15.0\n",
    "\n",
    "# Funwave wind forcing\n",
    "Funwave::wind_force = false\n",
    "Funwave::use_wind_mask = false\n",
    "Funwave::num_time_wind_data = 2\n",
    "Funwave::timewind[0] = 0\n",
    "Funwave::wu[0] = 25\n",
    "Funwave::wv[0] = 50\n",
    "Funwave::timewind[1] = 1000\n",
    "Funwave::wu[1] = 100\n",
    "Funwave::wv[1] = 100\n",
    "Funwave::boundary = funwave\n",
    "\n",
    "# Funwave wave maker\n",
    "FunWave::wavemaker_type = \"ini_gau\"\n",
    "FunWave::xc = 26.5\n",
    "FunWave::yc = 26.9\n",
    "FunWave::amp =  2.0\n",
    "FunWave::wid =  1\n",
    "Funwave::wdep = 0.78\n",
    "Funwave::xwavemaker = 25.0\n",
    "\n",
    "# Funwave sponge \n",
    "FunWave::sponge_on = false\n",
    "FunWave::sponge_west_width = 2.0\n",
    "FunWave::sponge_east_width = 2.0\n",
    "FunWave::sponge_north_width = 0.0\n",
    "FunWave::sponge_south_width = 0.0\n",
    "FunWave::sponge_decay_rate = 0.9\n",
    "FunWave::sponge_damping_magnitude = 5.0\n",
    "\n",
    "# Funwave dispersion (example 3 enables dispersion)\n",
    "FunWave::dispersion_on = \"true\"\n",
    "FunWave::gamma1 = 1.0\n",
    "FunWave::gamma2 = 1.0\n",
    "FunWave::gamma3 = 1.0\n",
    "FunWave::beta_ref = -0.531\n",
    "FunWave::swe_eta_dep = 0.80\n",
    "FunWave::cd = 0.0\n",
    "\n",
    "# Funwave numerics (MoL parameter controls time integration scheme)\n",
    "FunWave::reconstruction_scheme = \"fourth\"\n",
    "FunWave::riemann_solver = \"HLLC\"\n",
    "FunWave::dtfac = 0.5\n",
    "FunWave::froudecap = 10.0\n",
    "FunWave::mindepth = 0.001\n",
    "FunWave::mindepthfrc = 0.001\n",
    "FunWave::enable_masks = \"true\"\n",
    "Funwave::estimate_dt_on = \"true\"\n",
    "\n",
    "FunwaveCoord::spherical_coordinates = false\n",
    "\n",
    "ActiveThorns = \"CarpetIOHDF5\"\n",
    "IOHDF5::out2D_xyplane_z = 0 \n",
    "IOHDF5::out2D_every = 10\n",
    "IOHDF5::out2D_vars = \" \n",
    "  FunWave::eta\n",
    "  FunWave::u\n",
    "  FunWave::v\n",
    "  Grid::Coordinates{out_every=1000000000}\n",
    "\"\n",
    "IOHDF5::out2D_xz = no\n",
    "IOHDF5::out2D_yz = no"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next cell deletes our simulation in case we want to throw it away and start over again for some reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -fr ~/simulations/wave"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At long last, we are ready run Cactus. This configuration specifies running on two threads, with 1 thread per process. To execute this command, Cactus uses a \"RunScript\" stored in configs/sim/RunScript. You might want to take a look at it. Identifiers sandwiched between @ symbols get replaced by Simfactory prior to execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ~/CactusFW2/configs/sim/RunScript"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enough already! Let's run Cactus!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2\n",
    "!./simfactory/bin/sim create-run --procs 2 --num-threads 1 wave.par"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be found in this directory. Using the next couple of commands, we will browse it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/simulations/wave/output-0000/wave"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%ls *.asc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell enables inline plotting in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top of %pycat command showed us what the columns mean:\n",
    "HYDROBASE::rho (hydrobase-rho)\n",
    "* 1 iteration\n",
    "* 2 time - how much time has passed in the simulation\n",
    "* 3 the data, in this case the variable rho\n",
    "\n",
    "Once we know all this, it is straightforward to plot the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin_data = np.genfromtxt(\"eta.maximum.asc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(lin_data[:,1],lin_data[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python knows how to read regularly formatted text\n",
    "files that use the # character for comments. Fortunately,\n",
    "that's what Cactus produces in its asc files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_data = np.genfromtxt(\"eta.xy.asc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "# https://matplotlib.org/examples/color/colormaps_reference.html\n",
    "cmap = cm.gist_rainbow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sets = np.unique(file_data[:,0])\n",
    "width = 8\n",
    "height = 4\n",
    "print(\"sets=\",sets)\n",
    "mn, mx = np.min(file_data[:,12]),np.max(file_data[:,12])\n",
    "for which in sets: \n",
    "    print(\"which=\",which)\n",
    "    g = file_data[file_data[:,0]==which,:]\n",
    "    x = g[:,5]\n",
    "    y = g[:,6]\n",
    "    z = g[:,12]\n",
    "    zi = z.reshape(len(np.unique(y)),len(np.unique(x)))\n",
    "    print('min/max=',np.min(zi),np.max(zi))\n",
    "    plt.figure(figsize=(width, height))\n",
    "    plt.imshow(zi[::-1,:],cmap,clim=(mn,mx))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%ls *.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Plotting HDF5 Data</h2>\n",
    "HDF5 (Hierarchical Data Format 5) is a portable binary data format. As such, it is far more efficient to read and\n",
    "write than ascii formats, and it is probably what you should normally use. Here, you can see how to read and display\n",
    "the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f5 = h5py.File(\"u.xy.h5\")\n",
    "for nm in f5:\n",
    "    if not hasattr(f5[nm],\"shape\"):\n",
    "        continue\n",
    "    print(\"nm=\",nm)\n",
    "    d=np.copy(f5[nm])\n",
    "    plt.figure()\n",
    "    plt.imshow(d[::-1,:])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, each set only has one component of the plot, i.e. the part belonging to one processor. To fix this, we'll\n",
    "collect data sets belonging to an iteration and display them all together. In order to make this happen, we'll need the\n",
    "x and y values for each component of the grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f5x = h5py.File(\"x.xy.h5\")\n",
    "f5y = h5py.File(\"y.xy.h5\")\n",
    "x_coords = {}\n",
    "y_coords = {}\n",
    "for nm in f5x:\n",
    "    print(nm)\n",
    "    m = re.search(r'rl=.*c=\\d+',nm)\n",
    "    if m:\n",
    "        k = m.group(0)\n",
    "        x_coords[k]=np.copy(f5x[nm])\n",
    "for nm in f5y:\n",
    "    m = re.search(r'rl=.*c=\\d+',nm)\n",
    "    if m:\n",
    "        k = m.group(0)\n",
    "        y_coords[k]=np.copy(f5y[nm])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f5 = h5py.File(\"u.xy.h5\")\n",
    "mn,mx = None,None\n",
    "\n",
    "# Compute the min and max\n",
    "for nm in f5:\n",
    "    if not hasattr(f5[nm],\"shape\"):\n",
    "        continue\n",
    "    d5 = np.copy(f5[nm])\n",
    "    tmin = np.min(d5)\n",
    "    tmax = np.max(d5)\n",
    "    if mn == None:\n",
    "        mn,mx = tmin,tmax\n",
    "    else:\n",
    "        if tmin < mn:\n",
    "            mn = tmin\n",
    "        if tmax > mx:\n",
    "            mx = tmax\n",
    "            \n",
    "# Collect all the pieces into the d5_tl dictionary\n",
    "d5_tl = {}            \n",
    "for nm in f5:\n",
    "    if not hasattr(f5[nm],\"shape\"):\n",
    "        continue\n",
    "    # Parse the string nm...\n",
    "    m = re.search(r'it=(\\d+)\\s+tl=\\d+\\s+(rl=(\\d+)\\s+c=(\\d+))',nm)\n",
    "    # group(1) is the iteration number\n",
    "    # group(2) is \"rl={number} c={number}\"\n",
    "    # group(3) is the number in \"rl={number}\"\n",
    "    # group(4) is the number in \"c={number}\"\n",
    "    grid = int(m.group(1))\n",
    "    comp = int(m.group(4))\n",
    "    k = m.group(2)\n",
    "    if grid in d5_tl:\n",
    "        d5_tl[grid][\"x\"] += [x_coords[k]] # append to the x array\n",
    "        d5_tl[grid][\"y\"] += [y_coords[k]] # append to the y array\n",
    "        d5_tl[grid][\"D\"] += [f5[nm]] # append to the data array\n",
    "    else:\n",
    "        d5_tl[grid] = {\n",
    "            \"x\":[x_coords[k]],\n",
    "            \"y\":[y_coords[k]],\n",
    "            \"D\":[f5[nm]]\n",
    "        }\n",
    "\n",
    "# Sort the keys so that we display time levels in order\n",
    "def keysetf(d):\n",
    "    a = [] # create an empty list\n",
    "    for k in d: # for each key in d\n",
    "        a.append(k) # append it to the list\n",
    "    return a\n",
    "kys = keysetf(d5_tl.keys())\n",
    "kys.sort()\n",
    "\n",
    "# Show the figures, combing data from the same time level\n",
    "for index in kys:\n",
    "    data = d5_tl[index]\n",
    "    print(\"iteration=\",index)\n",
    "    plt.figure() # put this before the plots you wish to combine\n",
    "    plt.pcolor(data[\"x\"][0],data[\"y\"][0],data[\"D\"][0],vmin=mn,vmax=mx)\n",
    "    plt.pcolor(data[\"x\"][1],data[\"y\"][1],data[\"D\"][1],vmin=mn,vmax=mx)\n",
    "    plt.show() # show the plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Questions and Exercises:</h3>\n",
    "\n",
    "* Run the above simulation using a single process instead of two. Do the plotting routines work? What changes did you have to make. What would you need to do to make it work with 3?\n",
    "* Run the code at 1/2 the resolution.\n",
    "* Position the Guassian wave at a different place on the grid.\n",
    "* If you wanted to change the compiler or a compiler flag, how would you go about doing that?\n",
    "* If you wanted to add another thorn to the list of thorns to compile, how would you go about doing that?\n",
    "* If you wanted to create a thornlist that would check out Cactus under the Foo directory instead of the CactusFW2 directory, how would you do it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<table><tr><td>This work sponsored by NSF grants <a href=\"https://www.nsf.gov/awardsearch/showAward?AWD_ID=1550551\"> OAC 1550551</a> and <a href=\"https://www.nsf.gov/awardsearch/showAward?AWD_ID=1539567\"> CCF 1539567</a></td><td><img src=\"https://www.nsf.gov/awardsearch/images/common/nsf_logo_bottom.png\"></tr></table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}