doc/notebooks/amazon_pickle_reader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Pickle Reader\n",
    "\n",
    "File: [amazon_pickle_reader.py](../../access/amazon_pickle_reader.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reload modules every time before executing the Python code typed\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# Import from code root directory\n",
    "import sys; sys.path.insert(0, '../..')\n",
    "\n",
    "# Imports\n",
    "import os.path\n",
    "from access.file_storage import FileStorage\n",
    "from access.amazon_pickle_reader import AmazonPickleReader\n",
    "from sklearn.metrics.pairwise import pairwise_distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get storage directory of amazon data\n",
    "amazon_directoy = os.path.dirname(FileStorage().get_filepath('amazon_raw.pickle'))\n",
    "print('amazon_directoy:', amazon_directoy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading data\n",
    "\n",
    "To create an instance, specify the directory containing prepared pickle files. The instance will store data read from pickle files in memory. The pickle files will be read on first data access, not when creating the instance.\n",
    "\n",
    "`data = AmazonPickleReader('/tmp/directory-with-pickle-files/')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = AmazonPickleReader(amazon_directoy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing texts\n",
    "\n",
    "Texts are available for the IDs **[0, 7911683]**.\n",
    "\n",
    "Regarding to the original file, the texts are taken from the fields 'summary' and 'text', see [code](https://github.com/EML4U/ExplainingDriftTextEmbeddings/blob/a9d833ba28fb46959901d081b44338f3c8499ec9/data/amazon_movie_sorter.py#L36)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('first text: ', data.get_text(0)[:100])       # 100 characters of first text\n",
    "print('last text:  ', data.get_text(-1)[:100])      # 100 characters of last text\n",
    "print('last text:  ', data.get_text(7911683)[:100]) # 100 characters of last text\n",
    "\n",
    "# first text:  The everyday man at war &quot;The Cruel Sea&quot; gives an excellent account of the real war at sea,\n",
    "# last text:   Little darlings looking I saw this movie when I was in high school and loved Tatum O'neal and Kristy\n",
    "# last text:   Little darlings looking I saw this movie when I was in high school and loved Tatum O'neal and Kristy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing BoW50 embeddings (Bag of Words with 50 dimensions)\n",
    "\n",
    "Texts are available for the IDs **[84090, 7911683]**, as the first 3 years are not included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('first bow50: ', data.get_bow50(84090)[:5],   '...') # 5 numbers of first bow50\n",
    "print('last bow50:  ', data.get_bow50(7911683)[:5], '...') # 5 numbers of last bow50\n",
    "\n",
    "# first bow50:  [-0.24985783  0.16864878  0.02703895 -0.11902861 -0.12623635] ...\n",
    "# last bow50:   [-0.0483431  -0.698007    0.29450932  0.3002716   0.35189843] ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding characteristics\n",
    "\n",
    "Be aware that the same texts do not result in the same embeddings. But they are more similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sameA1 = 84107\n",
    "sameA2 = 84119\n",
    "print('sameA1 text:  ', data.get_text(sameA1)[:100])\n",
    "print('sameA2 text:  ', data.get_text(sameA2)[:100])\n",
    "print('sameA1 bow50: ', data.get_bow50(sameA1)[:5])\n",
    "print('sameA2 bow50: ', data.get_bow50(sameA2)[:5])\n",
    "\n",
    "# sameA1 text:   Might have been profound for it's time but... When I pay almost 30. for a DVD I expect alot, especia\n",
    "# sameA2 text:   Might have been profound for it's time but... When I pay almost 30. for a DVD I expect alot, especia\n",
    "# sameA1 bow50:  [-0.02514367 -0.29414916  0.40210924 -0.09406912 -0.51019126]\n",
    "# sameA2 bow50:  [ 0.04848728 -0.2670002   0.3912263  -0.07553244 -0.46337175]\n",
    "\n",
    "sameB1 = 84201\n",
    "sameB2 = 84206\n",
    "print('sameB1 text:  ', data.get_text(sameB1)[:100])\n",
    "print('sameB2 text:  ', data.get_text(sameB2)[:100])\n",
    "print('sameB1 bow50: ', data.get_bow50(sameB1)[:5])\n",
    "print('sameB2 bow50: ', data.get_bow50(sameB2)[:5])\n",
    "\n",
    "# sameB1 text:   Must not be ignored &quot;Salo&quot; is one of the most controversial and debatable films of the cen\n",
    "# sameB2 text:   Must not be ignored &quot;Salo&quot; is one of the most controversial and debatable films of the cen\n",
    "# sameB1 bow50:  [ 0.6617752   0.4826813   0.21395582 -0.2834527   0.05723236]\n",
    "# sameB2 bow50:  [ 0.7021096   0.38454628  0.20149517 -0.29684734  0.05091358]\n",
    "\n",
    "print('distance sameA1 sameA2', pairwise_distances([data.get_bow50(sameA1)], [data.get_bow50(sameA2)]))\n",
    "print('distance sameB1 sameB2', pairwise_distances([data.get_bow50(sameB1)], [data.get_bow50(sameB2)]))\n",
    "print('distance sameA1 sameB1', pairwise_distances([data.get_bow50(sameA1)], [data.get_bow50(sameB1)]))\n",
    "print('distance sameA2 sameB2', pairwise_distances([data.get_bow50(sameA2)], [data.get_bow50(sameB2)]))\n",
    "\n",
    "# distance sameA1 sameA2 [[0.28922]]\n",
    "# distance sameB1 sameB2 [[0.3078685]]\n",
    "# distance sameA1 sameB1 [[3.7674677]]\n",
    "# distance sameA2 sameB2 [[3.812414]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing metadata\n",
    "\n",
    "Metadata is available for both, text (raw) and bow50. Equal IDs result in equal metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('metadata first text:  ', data.get_text(0,      metadata=True))\n",
    "print('metadata text  84090: ', data.get_text(84090,  metadata=True))\n",
    "print('metadata bow50 84090: ', data.get_bow50(84090, metadata=True))\n",
    "print('metadata last text:   ', data.get_text(-1,     metadata=True))\n",
    "\n",
    "# metadata first text:   ['2/2', 4, datetime.datetime(1997, 8, 20, 2, 0), 2381344, 0]\n",
    "# metadata text  84090:  ['2/3', 3, datetime.datetime(2000, 1, 1, 1, 0), 1771, 84090]\n",
    "# metadata bow50 84090:  ['2/3', 3, datetime.datetime(2000, 1, 1, 1, 0), 1771, 84090]\n",
    "# metadata last text:    ['0/0', 3, datetime.datetime(2012, 10, 25, 2, 0), 7908990, 7911683]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metadata keys\n",
    "\n",
    "The keys were created in [amazon_movie_sorter.py](https://github.com/EML4U/ExplainingDriftTextEmbeddings/blob/a9d833ba28fb46959901d081b44338f3c8499ec9/data/amazon_movie_sorter.py#L30).\n",
    "\n",
    "- [0] helpfulness\n",
    "- [1] score [0, 4] (corresponding to [1, 5] stars)\n",
    "- [2] date\n",
    "- [3] entry number in original file from stanford.edu\n",
    "- [4] id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tup = data.get_text(0, metadata=True)\n",
    "print(tup[data.META_HELPFULNESS].ljust(20),      type(tup[data.META_HELPFULNESS]))\n",
    "print(str(tup[data.META_SCORE]).ljust(20),       type(tup[data.META_SCORE]))\n",
    "print(str(tup[data.META_DATE]).ljust(20),        type(tup[data.META_DATE]))\n",
    "print(str(tup[data.META_ORIGINAL_NO]).ljust(20), type(tup[data.META_ORIGINAL_NO]))\n",
    "print(str(tup[data.META_ID]).ljust(20),          type(tup[data.META_ID]))\n",
    "\n",
    "# 2/2                  <class 'str'>\n",
    "# 4                    <class 'int'>\n",
    "# 1997-08-20 02:00:00  <class 'datetime.datetime'>\n",
    "# 2381344              <class 'int'>\n",
    "# 0                    <class 'int'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generated embeddings\n",
    "\n",
    "The embeddings were generated using:\n",
    "\n",
    "- [../../transformation/amazon_movie_sorter.py](../../transformation/amazon_movie_sorter.py)\n",
    "    - uses `amazon_reviews_reader.py`\n",
    "    - writes `amazon_raw.pickle`\n",
    "- [../../transformation/amazon_reviews_reader.py](../../transformation/amazon_reviews_reader.py)\n",
    "    - reads `movies.txt.gz` (original file)\n",
    "- [../../transformation/generator_amazon_movie_embeddings.py](../../transformation/generator_amazon_movie_embeddings.py)\n",
    "    - reads `amazon_raw.pickle`\n",
    "    - uses `doc_to_vec.py`\n",
    "    - writes `amazon_bow_50.pickle`\n",
    "- [../../transformation/doc_to_vec.py](../../transformation/doc_to_vec.py)\n",
    "    - uses model(s) available at [FTP](https://hobbitdata.informatik.uni-leipzig.de/EML4U/2021-05-17-Amazon-Doc2Vec/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (EML4U)",
   "language": "python",
   "name": "eml4u"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}