Create_sparse_matrices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Spare Word Count Matrix\n",
    "\n",
    "This notebook creates a sparse matrix from a large set of word counts derived from historical British newspapers. \n",
    "- The columns in this matrix corresponds with a chosen vocabulary\n",
    "- The rows are the words counts for one newspaper title in specific month.\n",
    "\n",
    "This notebook explains and covers the following stages\n",
    "\n",
    "- 1. Download original count data from Zenodo\n",
    "- 2. Process JSON files with word counts\n",
    "- 3. Constract a vocabulary from the word counts (the matrix columns)\n",
    "- 4. Convert all JSON word counts to a sparse matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tools.ngram_creation import *\n",
    "from tools.ngram_creation import CorpusProcessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get word counts from Zenodo and unzip .tar file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write instructions here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Process JSON files\n",
    "## 2.1 Manage JSON files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load all the JSON files with the `JSONHandler` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "unzipped_ngram_path = '/Volumes/X9 Pro/ngrams-output'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "handler = JSONHandler(unzipped_ngram_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove corrupted json files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#handler.check_json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`len` returns the number of JSON files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(handler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`print` also includes the distinct number of newspapers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(handler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Create Vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine location where we store all the vocab, sparse matrices and other files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_to = '/Volumes/X9 Pro/ngrams-by-nlp-all'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially we combine the counts by NLP. For each JSON file we remove tokens that only occur once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Vocab(handler,save_to=save_to,min_threshold=1,**{'n_cores':8})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time vocab.nlp_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this operation is stored in `.wc_by_nlp` attribute (\"word counts by nlp\"). Each element is a dictionary with word counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vocab.wc_by_nlp[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vocab.wc_by_nlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we combine these dictionary, we set 5 as the minimum threshold at the level of the NLP/newspaper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab.min_threshold = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time vocab.total_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vocab.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we remove words that appear less then 2500 time in total with `filter_total`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab.filter_by_min_threshold(2500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vocab.vocab),len(vocab.wc_total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -la {vocab.save_to}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Convert JSON data to sparse matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we process the whole collection using this vocabulary across the whole corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_json = '/Volumes/X9 Pro/ngrams-output'\n",
    "path_to_matrices = '/Volumes/X9 Pro/ngrams-by-nlp-all'\n",
    "\n",
    "handler = JSONHandler(path_to_json)\n",
    "vocab = json.load(open(Path(path_to_matrices) / 'vocab.json'))\n",
    "\n",
    "corpus_proc = CorpusProcessor(handler,\n",
    "                              vocab, \n",
    "                              path_to_matrices,\n",
    "                              n_cores=8)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_proc.process_ngrams()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls {save_to} | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!du -h {save_to}/.. --max-depth=1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f52db470370448f7a886c43137db55aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1204 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving metadata of size (269179, 40)\n"
     ]
    }
   ],
   "source": [
    "# create metadata file\n",
    "corpus_proc.merge_metadata(Path(path_to_matrices),\n",
    "                            totals=None,\n",
    "                            npd_links_path= 'data/newspapers_overview_with_links_JISC_NLPs.csv',\n",
    "                            npd_data_path ='data/MPD_export_1846_1920_20230504.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge sparse matrices\n",
    "\n",
    "The cells below we merge all the ngrams at the NLP level into one large sparse matrix with corresponding metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_json = '/Volumes/X9 Pro/ngrams-output'\n",
    "path_to_matrices = '/Volumes/X9 Pro/ngrams-by-nlp-all'\n",
    "save_to = '/Volumes/X9 Pro/unigram-matrix'\n",
    "\n",
    "handler = JSONHandler(path_to_json)\n",
    "vocab = json.load(open(Path(path_to_matrices) / 'vocab.json'))\n",
    "corpus_proc = CorpusProcessor(handler,\n",
    "                              vocab = vocab, \n",
    "                              save_to = save_to)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# WARNING: this operation requires lots of memory (ca. 128 GB) and will likely crash your kernel\n",
    "save_merged = '/Volumes/X9 Pro/unigram-matrix'\n",
    "corpus_proc.merge_sparse_matrices(save_merged, override=True,\n",
    "                                 **{'npd_links_path' : 'data/newspapers_overview_with_links_JISC_NLPs.csv',\n",
    "                                    'npd_data_path' : 'data/MPD_export_1846_1920_20230504.csv'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -la {save_merged} --block-size=G"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fin."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "presscounts",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}