1.6_generators_and_comprehensions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Wonderful World of Generators!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is a generator, you ask? I'm glad you did!\n",
    "\n",
    "If you've any experience in Python, then you've seen the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "for i in range(0,3):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might naively think that range creates a list: \\[0, 1, 2], but in fact, compare the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2]\n",
      "range(0, 3)\n"
     ]
    }
   ],
   "source": [
    "print([0,1,2])\n",
    "print(range(0,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, range doesn't actually contain any integers at first. It is a class of object called, you guessed it, a __generator__ (well, a generator-iterator in range's case).\n",
    "Until (implicitly or explicitly) it's internal ```__iter__``` function is called (see the tutorial on magic functions), it just sits, waiting to be used for useful work. Here is what I mean by explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<range_iterator object at 0x7ff65c30f780>\n"
     ]
    }
   ],
   "source": [
    "x = range(2)\n",
    "print(x.__iter__())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, that didn't work as expected, that is, not like it would for any other iterative generator. Why not? After consulting documentation, it appears that range has become a special type as of Python 3. But that's okay, because...we can make our own generators!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making your own generator\n",
    "So, here we make use of the all-important ```yield``` statement, which transforms any old function into a generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_gen(alist):\n",
    "    for i in alist:\n",
    "        yield i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object my_gen at 0x7ff65c351570>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen = my_gen([1,2,3])\n",
    "gen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-67-0996b87ec51a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "print(next(gen))\n",
    "print(next(gen))\n",
    "print(next(gen))\n",
    "print(next(gen))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There you have it. The generator _yields_ the next element available, on-demand, as the generator function is called. When there is nothing left (when it reaches a stop point), the `__iter__` magic function being called under the hood throws the StopIteration exception, which is actually the same exception that allows a `for` loop to gracefully exit iterating over an generator-iterator like `range`. In fact: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "x = my_gen([3,4,5])\n",
    "for i in x:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila! No StopIteration exception!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, these are clearly trivial examples just to motivate your understanding. When does this actually come in handy?\n",
    "Well, imagine you have a massive data set, somewhere in the area of 1 or 2 GB of JSON or XML (or just plain text) data sitting on your laptop. Are you going to read that whole monster into memory just so you can take a look at it? Of course not!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterators\n",
    "In case you had no idea what I was talking about when I talked about the `__iter__` method or iterators generally, let me give a quick overview.\n",
    "\n",
    "An iterator is simply an object that can be...iterated...across. A list is an obvious example, as is any object that has ordered elements. At its most basic, this simply means the object needs to implement some version of `__iter__`. A simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chris is horny\n",
      "Raymond is hungry\n",
      "Mohammed is hilarious\n"
     ]
    }
   ],
   "source": [
    "class DogHouse():\n",
    "    def __init__(self, dogs, states):\n",
    "        self.dogs = dogs\n",
    "        self.states = states\n",
    "    def __iter__(self):\n",
    "        for dog, state in zip(self.dogs, self.states):\n",
    "            yield dog, state\n",
    "\n",
    "doghouse = DogHouse(('Chris', 'Raymond', 'Mohammed'), ('horny', 'hungry', 'hilarious'))\n",
    "\n",
    "for dog, state in doghouse:\n",
    "    print(dog + \" is \" + state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try turning the above `yield` statement into a `return` statement, if you're still wondering why we must use `yield`. \n",
    "\n",
    "\n",
    "Okay, so I lied a bit. Technically, this instance of class `DogHouse` is not an iterator. It is an object that has an iterator method, though, so it can be iterated over, as we have done above.\n",
    "\n",
    "We can explicitly turn it into an iterator, as it has an `__iter__` method, by calling the `iter` function (which creates objects of type `iter`, so don't confuse it with the magic method `__iter__`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Chris', 'hungry')\n",
      "('Raymond', 'horny')\n",
      "('Mohammed', 'hilarious')\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-102-2587dcd95de1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdog_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdog_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdog_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "dog_iter = iter(doghouse)\n",
    "print(next(dog_iter))\n",
    "print(next(dog_iter))\n",
    "print(next(dog_iter))\n",
    "print(next(dog_iter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, we get the StopIteration exception we expect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comprehensions\n",
    "\n",
    "Okay, comprende so far? Now we move onto a fancy use of generators called Comprehensions, which are very useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [i for i in range(10)]\n",
    "y = list(range(10))\n",
    "x == y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the True output shows, these methods of creating a list from a range generator lead to the same results. The creation of x simply makes explicit that the generator `range` yields the values on the fly, which are then place in the list, element by element. Fancier uses involve making lists within lists, as well as dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0, 1, 2], [0, 1, 2], [0, 1, 2]]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[i for i in range(3)] for j in range(3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 1, 'b': 2, 'c': 3}"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{e:i for e, i in zip(['a','b','c'], [1, 2, 3])}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your not sure what zip does, play around with it a bit."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}