Using-Cactus/CreatingANewThorn-Part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Creating a New Thorn, Part 2</h1>\n",
    "So far we've created an empty thorn which does nothing. Never before did doing nothing feel like such an accomplishment!\n",
    "\n",
    "We can use it as a template for something more useful. To that end, we're going to create a thorn which computes an \"energy,\" the sum of the squares of the wave velocities at each point on the grid. First, we create the source file itself. It's simple enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./arrangements/FunwaveUtils/EnergyCalc/src/energy.cc\n",
    "// We pretty much always want to include these 3 headers\n",
    "#include <cctk.h>\n",
    "#include <cctk_Arguments.h>\n",
    "#include <cctk_Parameters.h>\n",
    "\n",
    "void compute_energy(CCTK_ARGUMENTS) // Cactus functions always have this prototype\n",
    "{\n",
    "    DECLARE_CCTK_ARGUMENTS; // Declare all grid functions (from interface.ccl)\n",
    "    DECLARE_CCTK_PARAMETERS; // Declare all parameters (from param.ccl)\n",
    "    \n",
    "    // Note that even though this is really a 2-d calculation, Cactus\n",
    "    // thinks of it as 3-d with 1 zone in the z direction.\n",
    "    for(int k=0;k<cctk_lsh[2];k++) { // loop over the z direction\n",
    "        for(int j=0;j<cctk_lsh[1];j++) { // loop over the y direction\n",
    "            for(int i=0;i<cctk_lsh[0];i++) { // loop over the x direction\n",
    "                int cc = CCTK_GFINDEX3D(cctkGH,i,j,k);\n",
    "                energy[cc] = u[cc]*u[cc]+v[cc]*v[cc];\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we update the makefile so that it will build. Are we done? Let's try to compile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile arrangements/FunwaveUtils/EnergyCalc/src/make.code.defn\n",
    "# Main make.code.defn file for thorn Energy\n",
    "\n",
    "# Source files in this directory\n",
    "SRCS = energy.cc\n",
    "\n",
    "# Subdirectories containing source files\n",
    "SUBDIRS =\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!time ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=./my_thorns.th | cat -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why did it fail? Two reasons. First, the grid function \"energy\" was not defined. Second, the code did not know how to access the velocity fields, \"u\" and \"v.\"\n",
    "\n",
    "To overcome the first problem, we decalare the \"energy_group\" below. To fix the second, we simply inherit from thorn Funwave where the velocities \"u\" and \"v\" are defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./arrangements/FunwaveUtils/EnergyCalc/interface.ccl\n",
    "## Interface definitions for thorn Energy\n",
    "inherits: Funwave\n",
    "## An implementation name is required for all thorns. No\n",
    "## two thorns in a configuration can implement the same\n",
    "## interface.\n",
    "implements: EnergyCalc\n",
    "\n",
    "## the groups declared below can be public, private, or protected.\n",
    "public:\n",
    "\n",
    "## A group defines a set of variables that are allocated together\n",
    "## and share common properties, i.e. timelevels, tags such as the\n",
    "## Prolongation=None tag. The type tag can take on the values\n",
    "## GF, Scalar, or Array.\n",
    "\n",
    "## Note that the number of timelevels can be an integer parameter\n",
    "## GF stands for \"Grid Function\" and refers to a distributed array\n",
    "## data structure.\n",
    "#cctk_real force_group type=GF timelevels=3 tags='Prolongation=\"None\"'\n",
    "#{\n",
    "#  force1, force2\n",
    "#}\n",
    "\n",
    "cctk_real energy_group type=GF\n",
    "{\n",
    "    energy\n",
    "}\n",
    "\n",
    "## Scalars are single variables that are available on all processors.\n",
    "#{\n",
    "#  scalar1, scalar2\n",
    "#}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this change, everything should compile..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!time ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=./my_thorns.th | cat -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far so good. We've created a new thorn, including all the boiler plate code. We also filled in some source code and succeeded in compiling it. That's great. The only problem is, our thorn won't actually run. It's not scheduled to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./arrangements/FunwaveUtils/EnergyCalc/schedule.ccl\n",
    "## Schedule definitions for thorn Energy\n",
    "\n",
    "## There won't be any storage allocated for a group\n",
    "## unless a corresponding storage declaration exists\n",
    "## for it in the schedule file. In square brackets,\n",
    "## we specify the number of storage levels to allocate.\n",
    "## These commented examples correspond to the commented\n",
    "## examples in the interface file above.\n",
    "# storage: force_group[3]\n",
    "# storage: scalar_group\n",
    "\n",
    "## Schedule a function defined in this thorn to run at the beginning\n",
    "## of the simulation. The minimum you need to specify for a schedule\n",
    "## item is what language it's written in. Choices are: C (which includes\n",
    "## C++) and Fortran (which means Fortran90).\n",
    "#SCHEDULE init_function at CCTK_INIT\n",
    "#{\n",
    "#  LANG: C\n",
    "#}\"Do some initial stuff\"\n",
    "\n",
    "schedule compute_energy in CCTK_ANALYSIS\n",
    "{\n",
    "    LANG: C\n",
    "} \"Compute the Energy for Analysis\" # This comment is required!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does CCTK_ANALYSIS mean? Cactus schedules it's work in several bins. While there are many of them, here we only consider three:\n",
    "\n",
    "1) CCTK_INITIAL - This runs once at the beginning. Initialize your grid functions here\n",
    "\n",
    "2) CCTK_EVOL - Evolve a single timestep forward. This step will run repeatedly until the simulation finishes. Actually, this is rarely used directly. Usually, MoL_CalcRHS is used instead. More on that in a little bit.\n",
    "\n",
    "3) CCTK_ANALYSIS - Called right before output routines. This ensures values are filled in sensibly. Call it to calculate values that you want to look at, but which aren't necessarily needed for the evolution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!time ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=./my_thorns.th | cat -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's find out what our code does! First, we'll need to create a par file. It will be similar to the one we created\n",
    "in the first lesson. We will, however, make two small modifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ~/CactusFW2/wave2.par\n",
    "\n",
    "#Reorder the parameters for easy comparison to the input.txt in example 3\n",
    "ActiveThorns = \"\n",
    "  CoordBase FunWave FunwaveCoord CartGrid3D Carpet CarpetIOASCII\n",
    "  CartGrid3D IOUtil CarpetIOBasic CarpetSlab Boundary SymBase MoL\n",
    "  CarpetReduce LocalReduce InitBase CarpetLib LoopControl Tridiagonal\n",
    "  CarpetIOScalar EnergyCalc \"\n",
    "\n",
    "#----------------------------------------------------\n",
    "# Flesh and CCTK parameters\n",
    "#----------------------------------------------------\n",
    "\n",
    "# flesh\n",
    "Cactus::cctk_run_title = \"Test Run\"\n",
    "Cactus::cctk_show_schedule = \"yes\"\n",
    "Cactus::cctk_itlast = 300\n",
    "Cactus::allow_mixeddim_gfs = \"yes\"\n",
    "\n",
    "# CartGrid3D\n",
    "CartGrid3D::type = \"coordbase\"\n",
    "CartGrid3D::avoid_origin = \"no\"\n",
    "CoordBase::domainsize = \"minmax\"\n",
    "CoordBase::spacing    = \"gridspacing\"\n",
    "CoordBase::xmin =  0\n",
    "CoordBase::xmax =  30\n",
    "CoordBase::ymin =  0\n",
    "CoordBase::ymax =  30\n",
    "CoordBase::zmin =  0.0\n",
    "CoordBase::zmax =  0.0\n",
    "CoordBase::dx   =  0.25\n",
    "CoordBase::dy   =  0.25\n",
    "\n",
    "CoordBase::boundary_size_x_lower     = 3\n",
    "CoordBase::boundary_size_x_upper     = 3\n",
    "CoordBase::boundary_size_y_lower     = 3\n",
    "CoordBase::boundary_size_y_upper     = 3\n",
    "CoordBase::boundary_size_z_lower     = 0\n",
    "CoordBase::boundary_size_z_upper     = 0\n",
    "CoordBase::boundary_shiftout_x_lower = 1\n",
    "CoordBase::boundary_shiftout_x_upper = 1\n",
    "CoordBase::boundary_shiftout_y_lower = 1\n",
    "CoordBase::boundary_shiftout_y_upper = 1\n",
    "CoordBase::boundary_shiftout_z_lower = 1\n",
    "CoordBase::boundary_shiftout_z_upper = 1\n",
    "\n",
    "# Carpet\n",
    "Carpet::domain_from_coordbase = \"yes\"\n",
    "Carpet::ghost_size_x = 3\n",
    "Carpet::ghost_size_y = 3\n",
    "Carpet::ghost_size_z = 1\n",
    "carpet::adaptive_stepsize = yes\n",
    "\n",
    "# MoL\n",
    "MoL::ODE_Method = \"RK3\"\n",
    "MoL::disable_prolongation        = \"yes\"\n",
    "\n",
    "# the output dir will be named after the parameter file name\n",
    "IO::out_dir = $parfile\n",
    "IO::out_fileinfo=\"none\"\n",
    "IOBasic::outInfo_every = 1\n",
    "IOBasic::outInfo_vars = \"FunWave::eta FunWave::u FunWave::v\"\n",
    "\n",
    "#IOASCII::out1D_every = 1\n",
    "#IOASCII::out1d_vars = \"FunWave::eta Funwave::depth\"\n",
    "CarpetIOASCII::compact_format = false\n",
    "IOASCII::out2D_every = 30\n",
    "IOASCII::out2D_xyplane_z = 0\n",
    "IOASCII::out2D_vars = \"FunWave::eta FunWave::u FunWave::v EnergyCalc::energy\"\n",
    "IOASCII::out2D_xz = \"no\"\n",
    "IOASCII::out2D_yz = \"no\"\n",
    "IOASCII::output_ghost_points = \"no\"\n",
    "\n",
    "IOScalar::outScalar_every = 1\n",
    "IOScalar::outScalar_vars = \"FunWave::eta FunWave::u FunWave::v\"\n",
    "\n",
    "#& = \"Funwave::eta\"\n",
    "\n",
    "#----------------------------------------------------\n",
    "# Funwave parameters\n",
    "#----------------------------------------------------\n",
    "\n",
    "# Funwave depth \n",
    "FunWave::depth_file_offset_x = 3\n",
    "FunWave::depth_file_offset_y = 3\n",
    "FunWave::depth_type = \"flat\"\n",
    "FunWave::depth_format = \"ele\"\n",
    "FunWave::depth_file = \"/tmp/__depth__.txt\"\n",
    "FunWave::depth_flat = 0.8\n",
    "#Funwave::test_depth_shore_x = 80\n",
    "#Funwave::test_depth_island_x = 40\n",
    "#Funwave::test_depth_island_y = 40\n",
    "FunWave::depth_xslp = 10.0\n",
    "FunWave::depth_slope = 0.05\n",
    "FunWave::dt_size = 0\n",
    "Funwave::generate_test_depth_data = true\n",
    "Funwave::num_wave_components = 1\n",
    "Funwave::wave_component_file = \"/home/sbrandt/workspace/shi_funwave/example_2/fft/wavemk_per_amp_pha.txt\"\n",
    "Funwave::peak_period = 1\n",
    "\n",
    "# import\n",
    "Funwave::time_ramp = 1.0\n",
    "Funwave::delta_wk = 0.5\n",
    "Funwave::dep_wk = 0.45\n",
    "Funwave::xc_wk = 3.0\n",
    "Funwave::ywidth_wk = 10000.0\n",
    "Funwave::tperiod = 1.0\n",
    "Funwave::amp_wk = 0.0232\n",
    "Funwave::theta_wk = 0.0\n",
    "Funwave::freqpeak = 0.2\n",
    "Funwave::freqmin = 0.1\n",
    "Funwave::freqmax = 0.4\n",
    "Funwave::hmo = 1.0\n",
    "Funwave::gammatma = 5.0\n",
    "Funwave::thetapeak = 10.0\n",
    "Funwave::sigma_theta = 15.0\n",
    "\n",
    "# Funwave wind forcing\n",
    "Funwave::wind_force = false\n",
    "Funwave::use_wind_mask = false\n",
    "Funwave::num_time_wind_data = 2\n",
    "Funwave::timewind[0] = 0\n",
    "Funwave::wu[0] = 25\n",
    "Funwave::wv[0] = 50\n",
    "Funwave::timewind[1] = 1000\n",
    "Funwave::wu[1] = 100\n",
    "Funwave::wv[1] = 100\n",
    "Funwave::boundary = funwave\n",
    "\n",
    "# Funwave wave maker\n",
    "FunWave::wavemaker_type = \"ini_gau\"\n",
    "FunWave::xc = 26.5\n",
    "FunWave::yc = 26.9\n",
    "FunWave::amp =  2.0\n",
    "FunWave::wid =  1\n",
    "Funwave::wdep = 0.78\n",
    "Funwave::xwavemaker = 25.0\n",
    "\n",
    "# Funwave sponge \n",
    "FunWave::sponge_on = false\n",
    "FunWave::sponge_west_width = 2.0\n",
    "FunWave::sponge_east_width = 2.0\n",
    "FunWave::sponge_north_width = 0.0\n",
    "FunWave::sponge_south_width = 0.0\n",
    "FunWave::sponge_decay_rate = 0.9\n",
    "FunWave::sponge_damping_magnitude = 5.0\n",
    "\n",
    "# Funwave dispersion (example 3 enables dispersion)\n",
    "FunWave::dispersion_on = \"true\"\n",
    "FunWave::gamma1 = 1.0\n",
    "FunWave::gamma2 = 1.0\n",
    "FunWave::gamma3 = 1.0\n",
    "FunWave::beta_ref = -0.531\n",
    "FunWave::swe_eta_dep = 0.80\n",
    "FunWave::cd = 0.0\n",
    "\n",
    "# Funwave numerics (MoL parameter controls time integration scheme)\n",
    "FunWave::reconstruction_scheme = \"fourth\"\n",
    "FunWave::riemann_solver = \"HLLC\"\n",
    "FunWave::dtfac = 0.5\n",
    "FunWave::froudecap = 10.0\n",
    "FunWave::mindepth = 0.001\n",
    "FunWave::mindepthfrc = 0.001\n",
    "FunWave::enable_masks = \"true\"\n",
    "Funwave::estimate_dt_on = \"true\"\n",
    "\n",
    "FunwaveCoord::spherical_coordinates = false\n",
    "\n",
    "ActiveThorns = \"CarpetIOHDF5\"\n",
    "IOHDF5::out2D_xyplane_z = 0 \n",
    "IOHDF5::out2D_every = 10\n",
    "IOHDF5::out2D_vars = \" \n",
    "  FunWave::eta\n",
    "  FunWave::u\n",
    "  FunWave::v\n",
    "  Grid::Coordinates{out_every=1000000000}\n",
    "\"\n",
    "IOHDF5::out2D_xz = no\n",
    "IOHDF5::out2D_yz = no"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What, exactly, did we change? We can use diff to find out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!diff wave.par wave2.par"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -fr ~/simulations/wave2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/CactusFW2\n",
    "!./simfactory/bin/sim create-run --procs 2 --num-threads 1 wave2.par"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where did things go wrong? We can re-run the simulation with gdb and get a stack trace to see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gdb ./exe/cactus_sim --eval-command=\"run wave2.par\" --eval-command=where < /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The segfault occurs when we first attempt to write the energy variable. The problem is, Cactus has not allocated storage\n",
    "for the variable. To do that, we must edit the schedule file and add a storage declaration for the energy_group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile arrangements/FunwaveUtils/EnergyCalc/schedule.ccl\n",
    "## Schedule definitions for thorn Energy\n",
    "\n",
    "storage: energycalc::energy_group\n",
    "\n",
    "## There won't be any storage allocated for a group\n",
    "## unless a corresponding storage declaration exists\n",
    "## for it in the schedule file. In square brackets,\n",
    "## we specify the number of storage levels to allocate.\n",
    "## These commented examples correspond to the commented\n",
    "## examples in the interface file above.\n",
    "# storage: force_group[3]\n",
    "# storage: scalar_group\n",
    "\n",
    "## Schedule a function defined in this thorn to run at the beginning\n",
    "## of the simulation. The minimum you need to specify for a schedule\n",
    "## item is what language it's written in. Choices are: C (which includes\n",
    "## C++) and Fortran (which means Fortran90).\n",
    "#SCHEDULE init_function at CCTK_INIT\n",
    "#{\n",
    "#  LANG: C\n",
    "#}\"Do some initial stuff\"\n",
    "\n",
    "schedule compute_energy in CCTK_ANALYSIS\n",
    "{\n",
    "    LANG: C\n",
    "} \"Compute the Energy for Analysis\" # This comment is required!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!time ./simfactory/bin/sim build -j 2 --thornlist=./my_thorns.th"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -fr ~/simulations/wave2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./simfactory/bin/sim create-run --procs 2 --num-threads 1 wave2.par"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot our energy function. Basically, we're just copying cells from the Cactus-Funwave notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell enables inline plotting in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "# https://matplotlib.org/examples/color/colormaps_reference.html\n",
    "cmap = cm.gist_rainbow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "dir = os.environ[\"HOME\"]+\"/simulations/wave2/output-0000/wave2/\"\n",
    "file_data = np.genfromtxt(dir+\"energy.xy.asc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sets = np.unique(file_data[:,0])\n",
    "width = 8\n",
    "height = 4\n",
    "print(\"sets=\",sets)\n",
    "mn, mx = np.min(file_data[:,12]),np.max(file_data[:,12])\n",
    "for which in sets: \n",
    "    print(\"which=\",which)\n",
    "    g = file_data[file_data[:,0]==which,:]\n",
    "    x = g[:,5]\n",
    "    y = g[:,6]\n",
    "    z = g[:,12]\n",
    "    zi = z.reshape(len(np.unique(y)),len(np.unique(x)))\n",
    "    print('min/max=',np.min(zi),np.max(zi))\n",
    "    plt.figure(figsize=(width, height))\n",
    "    plt.imshow(zi[::-1,:],cmap,clim=(mn,mx))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Exercise:\n",
    "Make a copy of both \"CreatingANewThorn\" notebooks and edit them to create another new thorn, \"LaplacianEta\" which computes the Laplacian of the eta variable, which is a measure of the displacement of the surface of the water from rest. Like the \"Energy\" thorn, this quantity will be computed at analysis time.\n",
    "\n",
    "The second derivative of a quantity is approximated by the finite difference formula:\n",
    "\n",
    "$f''[x] = (f[x+dx] - 2 f[x] + f[x-dx])/dx^2 \\approx ((f[x]+dx f'[x]+\\frac{1}{2}dx^2 f[x]+...) - 2 f[x] + (f[x]-dx f'[x]+\\frac{1}{2}dx^2 f[x]+...))/dx^2$\n",
    "\n",
    "In terms of Cactus, f[x] and f[x+dx] might look something like this...\n",
    "<pre>\n",
    "  int cc = CCTK_GFINDEX3D(cctkGH,i,j,k)\n",
    "  int cp1 = CCTK_GFINDEX3D(cctkGH,i+1,j,k)\n",
    "  CCTK_REAL fx = f[cc]; // If this is f[x]\n",
    "  CCTK_REAL fx1 = f[cp1]; // this is f[x+dx]\n",
    "</pre>\n",
    "\n",
    "Cactus provides an additional array of integers, like cctk_lsh, called cctk_delta_space, which provides the quantities dx, dy, and dz (these are cctk_delta_space[0], cctk_delta_space[1] and cctk_delta_space[2], respecitively).\n",
    "\n",
    "Because this is a 2-d code, the Laplacian is\n",
    "\n",
    "$\\Delta^2 \\eta = \\left( \\frac{d^2}{dx^2} + \\frac{d^2}{dy^2} \\right) \\eta$\n",
    "\n",
    "Note that you will not be able to calculate the value of the Laplacian at the borders of the grid, as that would result in a segfault. Please write zeroes in the borders instead.\n",
    "\n",
    "Note also that Funwave defines grid variables dx and dy. You can use dx[cc] (where cc = CCTK_GFINDEX3D(cctkGH,i,j,k)) in place of cctk_delta_space[0] if you want to. You can't, however, redefined dx or dy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<table><tr><td>This work sponsored by NSF grants <a href=\"https://www.nsf.gov/awardsearch/showAward?AWD_ID=1550551\"> OAC 1550551</a> and <a href=\"https://www.nsf.gov/awardsearch/showAward?AWD_ID=1539567\"> CCF 1539567</a></td><td><img src=\"https://www.nsf.gov/awardsearch/images/common/nsf_logo_bottom.png\"></tr></table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}