doc/notebooks/interim_storage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interim Storage\n",
    "\n",
    "- Store intertim results in different formats.  \n",
    "- Uses **Pickle+BZ2** as default.\n",
    "- Other formats: JSON, JSON+BZ2, Pickle.\n",
    "- Uses **system temporary directory** as default.\n",
    "- JSON: Disadvantages in key mapping (see example formats) and serialization of different formats (see example json disadvantage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "# Reload modules every time before executing the Python code typed\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# Import from code root directory\n",
    "import sys; sys.path.insert(0, '../..')\n",
    "\n",
    "from access.interim_storage import InterimStorage\n",
    "\n",
    "from access.amazon_pickle_splitter import AmazonPickleSplitter\n",
    "import timeit\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "### Minimal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 'one', 'two': 2}\n"
     ]
    }
   ],
   "source": [
    "test_data = {1:'one', 'two':2}\n",
    "print(InterimStorage('test-id').write(test_data).read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write, read, info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 'one', 'two': 2} <class 'dict'>\n",
      "{1: 'one', 'two': 2} <class 'dict'>\n",
      "get_filepath  /tmp/InterimStorage/test-id.pickle.bz2\n",
      "get_directory /tmp/InterimStorage\n",
      "get_id        test-id\n"
     ]
    }
   ],
   "source": [
    "# Write\n",
    "print(test_data, type(test_data))\n",
    "storage = InterimStorage('test-id').write(test_data)\n",
    "\n",
    "# Read\n",
    "read_data = storage.read()\n",
    "print(read_data, type(read_data))\n",
    "\n",
    "# File location\n",
    "print('get_filepath ', storage.get_filepath())\n",
    "print('get_directory', storage.get_directory())\n",
    "print('get_id       ', storage.get_id())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 'one', 'two': 2} <class 'dict'>\n",
      "{'1': 'one', 'two': 2} <class 'dict'> /tmp/InterimStorage/test-id.json\n",
      "{'1': 'one', 'two': 2} <class 'dict'> /tmp/InterimStorage/test-id.json.bz2\n",
      "{1: 'one', 'two': 2} <class 'dict'> /tmp/InterimStorage/test-id.pickle\n",
      "{1: 'one', 'two': 2} <class 'dict'> /tmp/InterimStorage/test-id.pickle.bz2\n"
     ]
    }
   ],
   "source": [
    "print(test_data, type(test_data))\n",
    "\n",
    "storage = InterimStorage('test-id', type_=InterimStorage.JSON).write(test_data)\n",
    "read_data = storage.read()\n",
    "print(read_data, type(read_data), storage.get_filepath())\n",
    "\n",
    "storage = InterimStorage('test-id', type_=InterimStorage.JSON_BZ2).write(test_data)\n",
    "read_data = storage.read()\n",
    "print(read_data, type(read_data), storage.get_filepath())\n",
    "\n",
    "storage = InterimStorage('test-id', type_=InterimStorage.PICKLE).write(test_data)\n",
    "read_data = storage.read()\n",
    "print(read_data, type(read_data), storage.get_filepath())\n",
    "\n",
    "storage = InterimStorage('test-id', type_=InterimStorage.PICKLE_BZ).write(test_data)\n",
    "read_data = storage.read()\n",
    "print(read_data, type(read_data), storage.get_filepath())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storage directory, check and delete file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/InterimStorage/test-id.pickle.bz2\n",
      "/mnt/test-id.pickle.bz2\n",
      "\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(InterimStorage('test-id').get_filepath())\n",
    "print(InterimStorage('test-id', directory='/mnt').get_filepath())\n",
    "print()\n",
    "\n",
    "storage = InterimStorage('test-id')\n",
    "storage.write(test_data)\n",
    "print(storage.isfile())\n",
    "storage.delete_file()\n",
    "print(storage.isfile())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON disadvantage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<access.interim_storage.InterimStorage at 0x7fec282a7220>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "numpy_array = np.array([1, 'one', 'two', 2], dtype='O')\n",
    "InterimStorage('test-id', type_=InterimStorage.PICKLE).write(numpy_array)\n",
    "\n",
    "# InterimStorage('test-id', type_=InterimStorage.JSON).write(numpy_array)\n",
    "# -> TypeError: Object of type ndarray is not JSON serializable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Small benchmark\n",
    "\n",
    "### Get data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "directory_pickle = '/home/eml4u/EML4U/data/amazon-complete/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10 X 2 X 10,000 = 200,000\n",
    "years     = [year for year in range(2001, 2010+1)] \n",
    "stars     = [0, 4]\n",
    "max_items = 10 * 1000\n",
    "data = AmazonPickleSplitter(directory_pickle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 41.75812528701499\n"
     ]
    }
   ],
   "source": [
    "# Get indexes\n",
    "start = timeit.default_timer()\n",
    "indexes = data.get_data_splits(years=years, stars=stars, max_items=max_items)\n",
    "time_read = timeit.default_timer() - start\n",
    "print('Time:', time_read)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200000\n",
      "Time: 22.256728648906574\n",
      "Example: [0.34047234058380127, 0.2043135166168213, 0.3105195164680481, -0.1342063695192337, -0.7848458290100098, 0.19274359941482544, 0.13591687381267548, 0.2334279716014862, 0.4503822326660156, -0.44136351346969604, 0.064661905169487, 0.739519476890564, 0.25940191745758057, -0.0029060891829431057, 0.27662014961242676, -0.17198514938354492, -0.036287810653448105, -0.16849054396152496, 0.3470377027988434, 0.27252212166786194, -0.25882798433303833, 0.04169261083006859, -0.3211864233016968, 0.511594295501709, 0.11867643147706985, 0.11629629135131836, 0.901910662651062, 0.021725205704569817, -0.020022975280880928, -0.22301003336906433, -0.8718155026435852, 0.027210205793380737, 0.19232884049415588, -0.19159847497940063, -0.44342997670173645, -0.48215731978416443, -0.6705529093742371, 0.22982566058635712, -0.2696513533592224, -0.005211083684116602, 0.5327435731887817, 0.6535655856132507, -0.4448152780532837, 0.3689725995063782, 0.1422746181488037, 0.6372444033622742, -0.46162495017051697, -0.022056307643651962, 0.41951462626457214, 0.6869705319404602]\n"
     ]
    }
   ],
   "source": [
    "# Get data\n",
    "start = timeit.default_timer()\n",
    "bench_data = []\n",
    "for years in indexes.keys():\n",
    "    for stars in indexes[years].keys():\n",
    "        for item in indexes[years][stars]:\n",
    "            bench_data.append(data.get_bow50(item).tolist())\n",
    "print(len(bench_data))\n",
    "\n",
    "time_read_data = timeit.default_timer() - start\n",
    "print('Time:', time_read_data)\n",
    "print('Example:', bench_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_bench(type_, data):\n",
    "    storage = InterimStorage('bench'+type_, type_=type_)\n",
    "    \n",
    "    start = timeit.default_timer()\n",
    "    storage.write(data)\n",
    "    time_write = timeit.default_timer() - start\n",
    "    \n",
    "    start = timeit.default_timer()\n",
    "    storage.read()\n",
    "    time_read = timeit.default_timer() - start\n",
    "    \n",
    "    size = os.path.getsize(storage.get_filepath())\n",
    "    \n",
    "    storage.delete_file()\n",
    "    \n",
    "    return (type_, time_write, time_read, size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('json', 8.397693400969729, 5.131067170063034, 210338619), ('json.bz2', 29.49387290980667, 17.110431153792888, 77956243), ('pickle', 0.3819358260370791, 0.8716766810975969, 90812879), ('pickle.bz2', 8.203910200158134, 13.247875838074833, 38766345)]\n"
     ]
    }
   ],
   "source": [
    "bench_results = []\n",
    "bench_results.append(run_bench(InterimStorage.JSON, bench_data))\n",
    "bench_results.append(run_bench(InterimStorage.JSON_BZ2, bench_data))\n",
    "bench_results.append(run_bench(InterimStorage.PICKLE, bench_data))\n",
    "bench_results.append(run_bench(InterimStorage.PICKLE_BZ, bench_data))\n",
    "print(bench_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_result(title, bench_results, index):\n",
    "    max_ = float('-inf')\n",
    "    min_ = float('inf')\n",
    "    for t in bench_results:\n",
    "        if(t[index]>max_):\n",
    "            max_ = t[index]\n",
    "        if(t[index]<min_):\n",
    "            min_ = t[index]\n",
    "    \n",
    "    print(title.ljust(12), 'max'.ljust(8), 'min'.ljust(8), 'value')\n",
    "    \n",
    "    for t in bench_results:\n",
    "        print(t[0].ljust(12), format(t[index]/max_,\".4f\").ljust(8), format(t[index]/min_,\".4f\").rjust(8), t[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing      max      min      value\n",
      "json         0.2847    21.9872 8.397693400969729\n",
      "json.bz2     1.0000    77.2221 29.49387290980667\n",
      "pickle       0.0129     1.0000 0.3819358260370791\n",
      "pickle.bz2   0.2782    21.4798 8.203910200158134\n"
     ]
    }
   ],
   "source": [
    "print_result('Writing', bench_results, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading      max      min      value\n",
      "json         0.2999     5.8864 5.131067170063034\n",
      "json.bz2     1.0000    19.6293 17.110431153792888\n",
      "pickle       0.0509     1.0000 0.8716766810975969\n",
      "pickle.bz2   0.7743    15.1982 13.247875838074833\n"
     ]
    }
   ],
   "source": [
    "print_result('Reading', bench_results, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size         max      min      value\n",
      "json         1.0000     5.4258 210338619\n",
      "json.bz2     0.3706     2.0109 77956243\n",
      "pickle       0.4317     2.3426 90812879\n",
      "pickle.bz2   0.1843     1.0000 38766345\n"
     ]
    }
   ],
   "source": [
    "print_result('Size', bench_results, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Writing      max      min      value\n",
    "json         0.2869    23.9839 8.338726856047288  <-  24x slower\n",
    "json.bz2     1.0000    83.5866 29.061390546150506 <-   worst, 84x slower\n",
    "pickle       0.0120     1.0000 0.3476801458746195 <- best\n",
    "pickle.bz2   0.2746    22.9505 7.979426169069484  <-   23x slower\n",
    "\n",
    "Reading      max      min      value\n",
    "json         0.2949     6.2524 4.976031355094165  <-   6x slower\n",
    "json.bz2     1.0000    21.1991 16.871615912998095 <-  21x slower\n",
    "pickle       0.0472     1.0000 0.7958648079074919 <- best\n",
    "pickle.bz2   0.6658    14.1134 11.232359538087621 <-  14x slower\n",
    "\n",
    "Size         max      min      value\n",
    "json         1.0000     5.4258 210338619          <-   worst, 5x larger\n",
    "json.bz2     0.3706     2.0109 77956243           <-  2x larger\n",
    "pickle       0.4317     2.3426 90812879           <-  2x larger\n",
    "pickle.bz2   0.1843     1.0000 38766345           <- best\n",
    "\n",
    "JSON also can not serialize lots of formats\n",
    "JSON can be easyly read by lots of approaches\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (EML4U)",
   "language": "python",
   "name": "eml4u"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}