notebooks/chunk_dataframe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split a DataFrame into chunks\n",
    "\n",
    "Use [`numpy.array_split()`](https://numpy.org/doc/stable/reference/generated/numpy.array_split.html) to split a dataframe into chunks. This is useful, for example, when doing batch geocoding where the geocoding API only accepts a certain number of addresses per request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ghing/.local/share/virtualenvs/python-data-cheatsheet-nEkgY1LY/lib/python3.6/site-packages/pandas/compat/__init__.py:84: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n",
      "  warnings.warn(msg)\n",
      "/Users/ghing/.local/share/virtualenvs/python-data-cheatsheet-nEkgY1LY/lib/python3.6/site-packages/pandas/compat/__init__.py:84: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "# Setup\n",
    "\n",
    "import math\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create some toy data\n",
    "\n",
    "Use [np.random.rand](https://docs.scipy.org/doc//numpy-1.14.0/reference/generated/numpy.random.randn.html#numpy.random.randn) to generate a DataFrame with 1,090 rows and three columns. `randn()` returns [samples from the “standard normal” distribution](http://amsi.org.au/ESA_Senior_Years/SeniorTopic4/4b/4b_2content_6.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column_1</th>\n",
       "      <th>column_2</th>\n",
       "      <th>column_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>-0.522229</td>\n",
       "      <td>0.227516</td>\n",
       "      <td>-0.700995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.376896</td>\n",
       "      <td>0.951010</td>\n",
       "      <td>1.231711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>-0.717082</td>\n",
       "      <td>0.319611</td>\n",
       "      <td>0.006255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.514478</td>\n",
       "      <td>-0.525063</td>\n",
       "      <td>0.423066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>-0.929870</td>\n",
       "      <td>0.864430</td>\n",
       "      <td>-0.618780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1085</td>\n",
       "      <td>-0.109836</td>\n",
       "      <td>-0.480967</td>\n",
       "      <td>-0.179000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1086</td>\n",
       "      <td>2.121514</td>\n",
       "      <td>-0.728259</td>\n",
       "      <td>-1.568832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1087</td>\n",
       "      <td>0.514900</td>\n",
       "      <td>-0.240476</td>\n",
       "      <td>0.039966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1088</td>\n",
       "      <td>-0.124603</td>\n",
       "      <td>-0.950393</td>\n",
       "      <td>1.695722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1089</td>\n",
       "      <td>0.484821</td>\n",
       "      <td>-0.609991</td>\n",
       "      <td>0.270005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1090 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      column_1  column_2  column_3\n",
       "0    -0.522229  0.227516 -0.700995\n",
       "1     0.376896  0.951010  1.231711\n",
       "2    -0.717082  0.319611  0.006255\n",
       "3     0.514478 -0.525063  0.423066\n",
       "4    -0.929870  0.864430 -0.618780\n",
       "...        ...       ...       ...\n",
       "1085 -0.109836 -0.480967 -0.179000\n",
       "1086  2.121514 -0.728259 -1.568832\n",
       "1087  0.514900 -0.240476  0.039966\n",
       "1088 -0.124603 -0.950393  1.695722\n",
       "1089  0.484821 -0.609991  0.270005\n",
       "\n",
       "[1090 rows x 3 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(\n",
    "    np.random.randn(1090, 3),\n",
    "    columns=[\"column_1\", \"column_2\", \"column_3\"]\n",
    ")\n",
    "                  \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine the number of chunks using `math.ceil()`\n",
    "\n",
    "If we define a chunk size of 100, since we have 1,090 rows, we'll have 11 chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunk_size = 100\n",
    "n_chunks = math.ceil(df.shape[0] / chunk_size)\n",
    "\n",
    "n_chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use `numpy.array_split()` to split the dataframe into chunks\n",
    "\n",
    "We use [`numpy.array_split()`](https://numpy.org/doc/stable/reference/generated/numpy.array_split.html) rather than [`numpy.split()`](https://numpy.org/doc/stable/reference/generated/numpy.split.html) because our odd number of records will result in uneven chunk sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n",
      "99\n"
     ]
    }
   ],
   "source": [
    "chunks = np.array_split(df, n_chunks)\n",
    "\n",
    "for chunk in chunks:\n",
    "    print(chunk.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,Rmd"
  },
  "kernelspec": {
   "display_name": "python-data-cheatsheet",
   "language": "python",
   "name": "python-data-cheatsheet"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}