iterate-like-a-grandmaster.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25ee3f9a",
   "metadata": {
    "papermill": {
     "duration": 0.072902,
     "end_time": "2022-05-26T23:48:13.047034",
     "exception": false,
     "start_time": "2022-05-26T23:48:12.974132",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Iterate like a grandmaster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f7bc4c",
   "metadata": {
    "papermill": {
     "duration": 0.072179,
     "end_time": "2022-05-26T23:48:13.192250",
     "exception": false,
     "start_time": "2022-05-26T23:48:13.120071",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Note**: If you're fairly new to Kaggle, NLP, or Transformers, I strongly recommend you read my [Getting Started](https://www.kaggle.com/code/jhoward/getting-started-with-nlp-for-absolute-beginners) notebook first, and then come back to this one.\n",
    "\n",
    "---\n",
    "\n",
    "There's a lot of impressive notebooks around on Kaggle, but they often  fall into one of two categories:\n",
    "\n",
    "- Exploratory Data Analysis (EDA) notebooks with lots of pretty charts, but not much focus on understanding the key issues that will make a difference in the competition\n",
    "- Training/inference notebooks with little detail about *why* each step was chosen.\n",
    "\n",
    "In this notebook I'll try to give a taste of how a competitions grandmaster might tackle the [U.S. Patent Phrase to Phrase Matching](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/) competition. The focus generally should be two things:\n",
    "\n",
    "1. Creating an effective validation set\n",
    "1. Iterating rapidly to find changes which improve results on the validation set.\n",
    "\n",
    "If you can do these two things, then you can try out lots of experiments and find what works, and what doesn't. Without these two things, it will be nearly impossible to do well in a Kaggle competition (and, indeed, to create highly accurate models in real life!)\n",
    "\n",
    "I will show a couple of different ways to create an appropriate validation set, and will explain how to expand them into an appropriate cross-validation system. I'll use just plain HuggingFace Transformers for everything, and will keep the code concise and simple. The more code you have, the more you have to maintain, and the more chances there are to make mistakes. So keep it simple!\n",
    "\n",
    "OK, let's get started..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0746f13",
   "metadata": {
    "papermill": {
     "duration": 0.070617,
     "end_time": "2022-05-26T23:48:13.333562",
     "exception": false,
     "start_time": "2022-05-26T23:48:13.262945",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "It's nice to be able to run things locally too, to save your Kaggle GPU hours, so set a variable to make it easy to see where we are, and download what we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2c49324f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:13.485856Z",
     "iopub.status.busy": "2022-05-26T23:48:13.484682Z",
     "iopub.status.idle": "2022-05-26T23:48:32.890898Z",
     "shell.execute_reply": "2022-05-26T23:48:32.890333Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.775763Z"
    },
    "papermill": {
     "duration": 19.487416,
     "end_time": "2022-05-26T23:48:32.891069",
     "exception": false,
     "start_time": "2022-05-26T23:48:13.403653",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading us-patent-phrase-to-phrase-matching.zip to /Users/Declan/Projects/fastai-kaggle\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 682k/682k [00:10<00:00, 65.1kB/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "iskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\n",
    "if iskaggle:\n",
    "    !pip install -Uqq fastai\n",
    "else:\n",
    "    import zipfile,kaggle\n",
    "    path = Path('us-patent-phrase-to-phrase-matching')\n",
    "    kaggle.api.competition_download_cli(str(path))\n",
    "    zipfile.ZipFile(f'{path}.zip').extractall(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e423e3dd",
   "metadata": {
    "papermill": {
     "duration": 0.072537,
     "end_time": "2022-05-26T23:48:33.040140",
     "exception": false,
     "start_time": "2022-05-26T23:48:32.967603",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "A lot of the basic imports you'll want (`np`, `pd`, `plt`, etc) are provided by fastai, so let's grab them in one line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b9d5ed93",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:33.191021Z",
     "iopub.status.busy": "2022-05-26T23:48:33.190226Z",
     "iopub.status.idle": "2022-05-26T23:48:33.564990Z",
     "shell.execute_reply": "2022-05-26T23:48:33.564375Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.343819Z"
    },
    "papermill": {
     "duration": 0.452503,
     "end_time": "2022-05-26T23:48:33.565137",
     "exception": false,
     "start_time": "2022-05-26T23:48:33.112634",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from fastai.imports import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f5e2c9",
   "metadata": {
    "heading_collapsed": true,
    "papermill": {
     "duration": 0.071831,
     "end_time": "2022-05-26T23:48:33.710753",
     "exception": false,
     "start_time": "2022-05-26T23:48:33.638922",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Import and EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "497ad6d9",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.07272,
     "end_time": "2022-05-26T23:48:33.855411",
     "exception": false,
     "start_time": "2022-05-26T23:48:33.782691",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Set a path to our data. Use `pathlib.Path` because it makes everything so much easier, and make it work automatically regardless if you're working on your own PC or on Kaggle!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a47d56e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:34.006307Z",
     "iopub.status.busy": "2022-05-26T23:48:34.004567Z",
     "iopub.status.idle": "2022-05-26T23:48:34.013106Z",
     "shell.execute_reply": "2022-05-26T23:48:34.013511Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.782844Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.086082,
     "end_time": "2022-05-26T23:48:34.013648",
     "exception": false,
     "start_time": "2022-05-26T23:48:33.927566",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#3) [Path('../input/us-patent-phrase-to-phrase-matching/sample_submission.csv'),Path('../input/us-patent-phrase-to-phrase-matching/train.csv'),Path('../input/us-patent-phrase-to-phrase-matching/test.csv')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if iskaggle: path = Path('../input/us-patent-phrase-to-phrase-matching')\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a4a87c",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.072097,
     "end_time": "2022-05-26T23:48:34.157770",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.085673",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's look at the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa46e7dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:34.307315Z",
     "iopub.status.busy": "2022-05-26T23:48:34.306637Z",
     "iopub.status.idle": "2022-05-26T23:48:34.399213Z",
     "shell.execute_reply": "2022-05-26T23:48:34.398751Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.79936Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.168885,
     "end_time": "2022-05-26T23:48:34.399345",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.230460",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>anchor</th>\n",
       "      <th>target</th>\n",
       "      <th>context</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>37d61fd2272659b1</td>\n",
       "      <td>abatement</td>\n",
       "      <td>abatement of pollution</td>\n",
       "      <td>A47</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7b9652b17b68b7a4</td>\n",
       "      <td>abatement</td>\n",
       "      <td>act of abating</td>\n",
       "      <td>A47</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36d72442aefd8232</td>\n",
       "      <td>abatement</td>\n",
       "      <td>active catalyst</td>\n",
       "      <td>A47</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5296b0c19e1ce60e</td>\n",
       "      <td>abatement</td>\n",
       "      <td>eliminating process</td>\n",
       "      <td>A47</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54c1e3b9184cb5b6</td>\n",
       "      <td>abatement</td>\n",
       "      <td>forest region</td>\n",
       "      <td>A47</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36468</th>\n",
       "      <td>8e1386cbefd7f245</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden article</td>\n",
       "      <td>B44</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36469</th>\n",
       "      <td>42d9e032d1cd3242</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden box</td>\n",
       "      <td>B44</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36470</th>\n",
       "      <td>208654ccb9e14fa3</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden handle</td>\n",
       "      <td>B44</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36471</th>\n",
       "      <td>756ec035e694722b</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden material</td>\n",
       "      <td>B44</td>\n",
       "      <td>0.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36472</th>\n",
       "      <td>8d135da0b55b8c88</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden substrate</td>\n",
       "      <td>B44</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36473 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     id        anchor                  target context  score\n",
       "0      37d61fd2272659b1     abatement  abatement of pollution     A47   0.50\n",
       "1      7b9652b17b68b7a4     abatement          act of abating     A47   0.75\n",
       "2      36d72442aefd8232     abatement         active catalyst     A47   0.25\n",
       "3      5296b0c19e1ce60e     abatement     eliminating process     A47   0.50\n",
       "4      54c1e3b9184cb5b6     abatement           forest region     A47   0.00\n",
       "...                 ...           ...                     ...     ...    ...\n",
       "36468  8e1386cbefd7f245  wood article          wooden article     B44   1.00\n",
       "36469  42d9e032d1cd3242  wood article              wooden box     B44   0.50\n",
       "36470  208654ccb9e14fa3  wood article           wooden handle     B44   0.50\n",
       "36471  756ec035e694722b  wood article         wooden material     B44   0.75\n",
       "36472  8d135da0b55b8c88  wood article        wooden substrate     B44   0.50\n",
       "\n",
       "[36473 rows x 5 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'train.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c247effd",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.072586,
     "end_time": "2022-05-26T23:48:34.545440",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.472854",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "...and the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3ee6dc49",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:34.695328Z",
     "iopub.status.busy": "2022-05-26T23:48:34.694754Z",
     "iopub.status.idle": "2022-05-26T23:48:34.703897Z",
     "shell.execute_reply": "2022-05-26T23:48:34.703483Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.907493Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.085887,
     "end_time": "2022-05-26T23:48:34.704035",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.618148",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df = pd.read_csv(path/'test.csv')\n",
    "len(eval_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aaddf79d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:34.860557Z",
     "iopub.status.busy": "2022-05-26T23:48:34.859896Z",
     "iopub.status.idle": "2022-05-26T23:48:34.862629Z",
     "shell.execute_reply": "2022-05-26T23:48:34.863048Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.936134Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.085769,
     "end_time": "2022-05-26T23:48:34.863189",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.777420",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>anchor</th>\n",
       "      <th>target</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4112d61851461f60</td>\n",
       "      <td>opc drum</td>\n",
       "      <td>inorganic photoconductor drum</td>\n",
       "      <td>G02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>09e418c93a776564</td>\n",
       "      <td>adjust gas flow</td>\n",
       "      <td>altering gas flow</td>\n",
       "      <td>F23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36baf228038e314b</td>\n",
       "      <td>lower trunnion</td>\n",
       "      <td>lower locating</td>\n",
       "      <td>B60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1f37ead645e7f0c8</td>\n",
       "      <td>cap component</td>\n",
       "      <td>upper portion</td>\n",
       "      <td>D06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>71a5b6ad068d531f</td>\n",
       "      <td>neural stimulation</td>\n",
       "      <td>artificial neural network</td>\n",
       "      <td>H04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id              anchor                         target context\n",
       "0  4112d61851461f60            opc drum  inorganic photoconductor drum     G02\n",
       "1  09e418c93a776564     adjust gas flow              altering gas flow     F23\n",
       "2  36baf228038e314b      lower trunnion                 lower locating     B60\n",
       "3  1f37ead645e7f0c8       cap component                  upper portion     D06\n",
       "4  71a5b6ad068d531f  neural stimulation      artificial neural network     H04"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b92e76b9",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.073673,
     "end_time": "2022-05-26T23:48:35.010806",
     "exception": false,
     "start_time": "2022-05-26T23:48:34.937133",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's look at the distribution of values of `target`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7ec8e9f3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:35.253910Z",
     "iopub.status.busy": "2022-05-26T23:48:35.245904Z",
     "iopub.status.idle": "2022-05-26T23:48:35.286062Z",
     "shell.execute_reply": "2022-05-26T23:48:35.287178Z",
     "shell.execute_reply.started": "2022-04-20T18:46:49.963128Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.160127,
     "end_time": "2022-05-26T23:48:35.287397",
     "exception": false,
     "start_time": "2022-05-26T23:48:35.127270",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "composition                    24\n",
       "data                           22\n",
       "metal                          22\n",
       "motor                          22\n",
       "assembly                       21\n",
       "                               ..\n",
       "switching switch over valve     1\n",
       "switching switch off valve      1\n",
       "switching over valve            1\n",
       "switching off valve             1\n",
       "wooden substrate                1\n",
       "Name: target, Length: 29340, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.target.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5394c852",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.115163,
     "end_time": "2022-05-26T23:48:35.522064",
     "exception": false,
     "start_time": "2022-05-26T23:48:35.406901",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We see that there's nearly as many unique targets as items in the training set, so they're nearly but not quite unique. Most importantly, we can see that these generally contain very few words (1-4 words in the above sample).\n",
    "\n",
    "Let's check `anchor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "03244f5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:35.782445Z",
     "iopub.status.busy": "2022-05-26T23:48:35.781655Z",
     "iopub.status.idle": "2022-05-26T23:48:35.786140Z",
     "shell.execute_reply": "2022-05-26T23:48:35.786787Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.014179Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.142466,
     "end_time": "2022-05-26T23:48:35.787003",
     "exception": false,
     "start_time": "2022-05-26T23:48:35.644537",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "component composite coating              152\n",
       "sheet supply roller                      150\n",
       "source voltage                           140\n",
       "perfluoroalkyl group                     136\n",
       "el display                               135\n",
       "                                        ... \n",
       "plug nozzle                                2\n",
       "shannon                                    2\n",
       "dry coating composition1                   2\n",
       "peripheral nervous system stimulation      1\n",
       "conduct conducting material                1\n",
       "Name: anchor, Length: 733, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.anchor.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1010717",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.121258,
     "end_time": "2022-05-26T23:48:36.030313",
     "exception": false,
     "start_time": "2022-05-26T23:48:35.909055",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can see here that there's far fewer unique values (just 733) and that again they're very short (2-4 words in this sample).\n",
    "\n",
    "Now we'll do `context`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "13008a6b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:36.348869Z",
     "iopub.status.busy": "2022-05-26T23:48:36.348057Z",
     "iopub.status.idle": "2022-05-26T23:48:36.360761Z",
     "shell.execute_reply": "2022-05-26T23:48:36.361562Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.038141Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.189377,
     "end_time": "2022-05-26T23:48:36.361771",
     "exception": false,
     "start_time": "2022-05-26T23:48:36.172394",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "H01    2186\n",
       "H04    2177\n",
       "G01    1812\n",
       "A61    1477\n",
       "F16    1091\n",
       "       ... \n",
       "B03      47\n",
       "F17      33\n",
       "B31      24\n",
       "A62      23\n",
       "F26      18\n",
       "Name: context, Length: 106, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.context.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91eecd8e",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.122002,
     "end_time": "2022-05-26T23:48:36.653034",
     "exception": false,
     "start_time": "2022-05-26T23:48:36.531032",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "These are just short codes. Some of them have very few examples (18 in the smallest case) The first character is the section the patent was filed under -- let's create a column for that and look at the distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "598f9eeb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:36.898812Z",
     "iopub.status.busy": "2022-05-26T23:48:36.883754Z",
     "iopub.status.idle": "2022-05-26T23:48:36.914661Z",
     "shell.execute_reply": "2022-05-26T23:48:36.915082Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.054721Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.140415,
     "end_time": "2022-05-26T23:48:36.915230",
     "exception": false,
     "start_time": "2022-05-26T23:48:36.774815",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "B    8019\n",
       "H    6195\n",
       "G    6013\n",
       "C    5288\n",
       "A    4094\n",
       "F    4054\n",
       "E    1531\n",
       "D    1279\n",
       "Name: section, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['section'] = df.context.str[0]\n",
    "df.section.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbd30564",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.07417,
     "end_time": "2022-05-26T23:48:37.063875",
     "exception": false,
     "start_time": "2022-05-26T23:48:36.989705",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "It seems likely that these sections might be useful, since they've got quite a bit more data in each.\n",
    "\n",
    "Finally, we'll take a look at a histogram of the scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "be536877",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:37.218726Z",
     "iopub.status.busy": "2022-05-26T23:48:37.218153Z",
     "iopub.status.idle": "2022-05-26T23:48:37.449464Z",
     "shell.execute_reply": "2022-05-26T23:48:37.450197Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.103833Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.311537,
     "end_time": "2022-05-26T23:48:37.450348",
     "exception": false,
     "start_time": "2022-05-26T23:48:37.138811",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAT6klEQVR4nO3cf6zd9X3f8eerdkkIJJiE7iqyWe0pbjYHVo1eAVWk7iauwJAKI5VGIFrszKullmRZi9aYVRNTEiSilrLA8qPe8GwiGkNZN1uFllqEK7SpJkDJMD9KuQMC9kicxsadQ37U2Xt/nM9tT10b33vOvef4+jwf0tX9fj/fz/f7/bzPOfbrfn+cb6oKSdJo+5FhD0CSNHyGgSTJMJAkGQaSJAwDSRKweNgD6NU555xTy5cv72nd73znO5xxxhlzO6CTnDWPhlGredTqhf5rfuKJJ/6yqn7s6PYFGwbLly/n8ccf72ndyclJJiYm5nZAJzlrHg2jVvOo1Qv915zk68dq9zSRJMkwkCQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJJYwN9Alk5WyzfdP7R9b10zWo9m0NzxyECSdOIwSLIlyf4kT3e1/VaSP0/yVJL/lmRJ17Ibk0wleT7JpV3ta1rbVJJNXe0rkjza2u9Jctoc1idJmoGZHBlsBdYc1bYLOK+q/inwF8CNAElWAVcD72vrfD7JoiSLgM8BlwGrgGtaX4DPALdV1XuAg8CGviqSJM3aCcOgqh4BDhzV9idVdaTN7gaWtem1wPaq+n5VvQRMARe2n6mqerGqfgBsB9YmCfBB4L62/jbgyv5KkiTN1lxcQP4XwD1teimdcJi2t7UBvHpU+0XAu4DXu4Klu//fk2QjsBFgbGyMycnJngZ8+PDhntddqKx5cG44/8iJO82TUXufR61emL+a+wqDJL8JHAHunpvhvLmq2gxsBhgfH69en+ntM9BHw7BqXj/ku4lG6X32cz13eg6DJOuBnwNWV1W15n3AuV3dlrU2jtP+bWBJksXt6KC7vyRpQHq6tTTJGuA3gCuq6o2uRTuBq5O8JckKYCXwVeAxYGW7c+g0OheZd7YQeRi4qq2/DtjRWymSpF7N5NbSLwN/Crw3yd4kG4D/CLwd2JXka0m+CFBVzwD3As8CfwxcX1U/bH/1fxR4EHgOuLf1BfgE8OtJpuhcQ7hzTiuUJJ3QCU8TVdU1x2g+7n/YVXUzcPMx2h8AHjhG+4t07jaSJA2J30CWJBkGkiQfVDcy9uw7NJRbHl++5UMD36ek2fPIQJJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAksQMwiDJliT7kzzd1fbOJLuSvNB+n93ak+T2JFNJnkpyQdc661r/F5Ks62r/qSR72jq3J8lcFylJenMzOTLYCqw5qm0T8FBVrQQeavMAlwEr289G4AvQCQ/gJuAi4ELgpukAaX1+uWu9o/clSZpnJwyDqnoEOHBU81pgW5veBlzZ1X5XdewGliR5N3ApsKuqDlTVQWAXsKYte0dV7a6qAu7q2pYkaUAW97jeWFW91qa/AYy16aXAq1399ra2N2vfe4z2Y0qykc4RB2NjY0xOTvY0+MOHD/e87kI1djrccP6Rge93mK/zsN7nYbzO00btsz1q9cL81dxrGPyNqqokNReDmcG+NgObAcbHx2tiYqKn7UxOTtLrugvVHXfv4NY9fb/ds/bytRMD3+e0Yb3P6zfdP/B9Ttu65oyR+myP4r/l+aq517uJvtlO8dB+72/t+4Bzu/ota21v1r7sGO2SpAHqNQx2AtN3BK0DdnS1X9fuKroYONROJz0IXJLk7Hbh+BLgwbbsr5Jc3O4iuq5rW5KkATnheYMkXwYmgHOS7KVzV9AtwL1JNgBfBz7cuj8AXA5MAW8AHwGoqgNJPgU81vp9sqqmL0r/Kp07lk4H/qj9SJIG6IRhUFXXHGfR6mP0LeD642xnC7DlGO2PA+edaBySpPnjN5AlSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CSRJ9hkOTXkjyT5OkkX07y1iQrkjyaZCrJPUlOa33f0uan2vLlXdu5sbU/n+TSPmuSJM1Sz2GQZCnwr4DxqjoPWARcDXwGuK2q3gMcBDa0VTYAB1v7ba0fSVa19d4HrAE+n2RRr+OSJM1ev6eJFgOnJ1kMvA14DfggcF9bvg24sk2vbfO05auTpLVvr6rvV9VLwBRwYZ/jkiTNwuJeV6yqfUl+G3gF+C7wJ8ATwOtVdaR12wssbdNLgVfbukeSHALe1dp3d226e52/I8lGYCPA2NgYk5OTPY398OHDPa+7UI2dDjecf+TEHefYMF/nYb3Pw3idp43aZ3vU6oX5q7nnMEhyNp2/6lcArwO/T+c0z7ypqs3AZoDx8fGamJjoaTuTk5P0uu5CdcfdO7h1T89vd89evnZi4PucNqz3ef2m+we+z2lb15wxUp/tUfy3PF8193Oa6GeBl6rqW1X118AfAO8HlrTTRgDLgH1teh9wLkBbfhbw7e72Y6wjSRqAfsLgFeDiJG9r5/5XA88CDwNXtT7rgB1temebpy3/SlVVa7+63W20AlgJfLWPcUmSZqmfawaPJrkP+DPgCPAknVM49wPbk3y6td3ZVrkT+FKSKeAAnTuIqKpnktxLJ0iOANdX1Q97HZckafb6OolcVTcBNx3V/CLHuBuoqr4H/MJxtnMzcHM/Y5Ek9c5vIEuSDANJkmEgSaLPawYL1Z59h4ZyL/jLt3xo4PuUpJnwyECSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJJEn2GQZEmS+5L8eZLnkvx0kncm2ZXkhfb77NY3SW5PMpXkqSQXdG1nXev/QpJ1/RYlSZqdfo8MPgv8cVX9Y+AngeeATcBDVbUSeKjNA1wGrGw/G4EvACR5J3ATcBFwIXDTdIBIkgaj5zBIchbwM8CdAFX1g6p6HVgLbGvdtgFXtum1wF3VsRtYkuTdwKXArqo6UFUHgV3Aml7HJUmavcV9rLsC+BbwX5L8JPAE8HFgrKpea32+AYy16aXAq13r721tx2v/e5JspHNUwdjYGJOTkz0NfOx0uOH8Iz2t249exzsXRrHmw4cPD2X/w3idpw2r5mEZtXph/mruJwwWAxcAH6uqR5N8lr89JQRAVVWS6meAR21vM7AZYHx8vCYmJnrazh137+DWPf2U3puXr50Y+D6njWLNk5OT9PoZ6cf6TfcPfJ/Ttq45Yyg1D8uw3uNhmq+a+7lmsBfYW1WPtvn76ITDN9vpH9rv/W35PuDcrvWXtbbjtUuSBqTnMKiqbwCvJnlva1oNPAvsBKbvCFoH7GjTO4Hr2l1FFwOH2umkB4FLkpzdLhxf0tokSQPS73mDjwF3JzkNeBH4CJ2AuTfJBuDrwIdb3weAy4Ep4I3Wl6o6kORTwGOt3yer6kCf45IkzUJfYVBVXwPGj7Fo9TH6FnD9cbazBdjSz1gkSb3zG8iSJMNAkmQYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEHIRBkkVJnkzyh21+RZJHk0wluSfJaa39LW1+qi1f3rWNG1v780ku7XdMkqTZmYsjg48Dz3XNfwa4rareAxwENrT2DcDB1n5b60eSVcDVwPuANcDnkyyag3FJkmaorzBIsgz4EPCf23yADwL3tS7bgCvb9No2T1u+uvVfC2yvqu9X1UvAFHBhP+OSJM3O4j7X/w/AbwBvb/PvAl6vqiNtfi+wtE0vBV4FqKojSQ61/kuB3V3b7F7n70iyEdgIMDY2xuTkZE+DHjsdbjj/yIk7zrFexzsXRrHmw4cPD2X/w3idpw2r5j37Dg18nwArzlo01M/YMMzXe9xzGCT5OWB/VT2RZGLORvQmqmozsBlgfHy8JiZ62+0dd+/g1j395uDsvXztxMD3OW0Ua56cnKTXz0g/1m+6f+D7nLZ1zRkjVfOw6h2m+fpc9/O/w/uBK5JcDrwVeAfwWWBJksXt6GAZsK/13wecC+xNshg4C/h2V/u07nUkSQPQ8zWDqrqxqpZV1XI6F4C/UlXXAg8DV7Vu64AdbXpnm6ct/0pVVWu/ut1ttAJYCXy113FJkmZvPs4bfALYnuTTwJPAna39TuBLSaaAA3QChKp6Jsm9wLPAEeD6qvrhPIxLknQccxIGVTUJTLbpFznG3UBV9T3gF46z/s3AzXMxFknS7PkNZEmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCTRRxgkOTfJw0meTfJMko+39ncm2ZXkhfb77NaeJLcnmUryVJILura1rvV/Icm6/suSJM1GP0cGR4AbqmoVcDFwfZJVwCbgoapaCTzU5gEuA1a2n43AF6ATHsBNwEXAhcBN0wEiSRqMnsOgql6rqj9r0/8XeA5YCqwFtrVu24Ar2/Ra4K7q2A0sSfJu4FJgV1UdqKqDwC5gTa/jkiTNXqqq/40ky4FHgPOAV6pqSWsPcLCqliT5Q+CWqvofbdlDwCeACeCtVfXp1v7vgO9W1W8fYz8b6RxVMDY29lPbt2/vabz7Dxzim9/tadW+nL/0rMHvtBnFmg8fPsyZZ5458P3u2Xdo4PuctuKsRSNV87DqHaZ+P9cf+MAHnqiq8aPbF/c1KiDJmcB/Bf51Vf1V5///jqqqJP2nzd9ubzOwGWB8fLwmJiZ62s4dd+/g1j19lz5rL187MfB9ThvFmicnJ+n1M9KP9ZvuH/g+p21dc8ZI1Tyseodpvj7Xfd1NlORH6QTB3VX1B635m+30D+33/ta+Dzi3a/Vlre147ZKkAennbqIAdwLPVdXvdC3aCUzfEbQO2NHVfl27q+hi4FBVvQY8CFyS5Ox24fiS1iZJGpB+zhu8H/glYE+Sr7W2fwvcAtybZAPwdeDDbdkDwOXAFPAG8BGAqjqQ5FPAY63fJ6vqQB/jkiTNUs9h0C4E5ziLVx+jfwHXH2dbW4AtvY5FktQfv4EsSTIMJEmGgSQJw0CShGEgScIwkCRhGEiSMAwkSczBg+okaRQtH+LD+eaDRwaSJMNAkmQYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAksRJFAZJ1iR5PslUkk3DHo8kjZKTIgySLAI+B1wGrAKuSbJquKOSpNFxUoQBcCEwVVUvVtUPgO3A2iGPSZJGRqpq2GMgyVXAmqr6l23+l4CLquqjR/XbCGxss+8Fnu9xl+cAf9njuguVNY+GUat51OqF/mv+8ar6saMbF/exwYGrqs3A5n63k+TxqhqfgyEtGNY8Gkat5lGrF+av5pPlNNE+4Nyu+WWtTZI0ACdLGDwGrEyyIslpwNXAziGPSZJGxklxmqiqjiT5KPAgsAjYUlXPzOMu+z7VtABZ82gYtZpHrV6Yp5pPigvIkqThOllOE0mShsgwkCSd2mFwokdcJHlLknva8keTLB/CMOfMDOr99STPJnkqyUNJfnwY45xLM32MSZKfT1JJFvxtiDOpOcmH23v9TJLfG/QY59oMPtv/MMnDSZ5sn+/LhzHOuZJkS5L9SZ4+zvIkub29Hk8luaDvnVbVKflD50L0/wb+EXAa8L+AVUf1+VXgi236auCeYY97nuv9APC2Nv0rC7nemdbc+r0deATYDYwPe9wDeJ9XAk8CZ7f5fzDscQ+g5s3Ar7TpVcDLwx53nzX/DHAB8PRxll8O/BEQ4GLg0X73eSofGczkERdrgW1t+j5gdZIMcIxz6YT1VtXDVfVGm91N5/scC9lMH2PyKeAzwPcGObh5MpOafxn4XFUdBKiq/QMe41ybSc0FvKNNnwX8nwGOb85V1SPAgTfpsha4qzp2A0uSvLuffZ7KYbAUeLVrfm9rO2afqjoCHALeNZDRzb2Z1NttA52/LBayE9bcDp/Prar7BzmweTST9/kngJ9I8j+T7E6yZmCjmx8zqfnfA7+YZC/wAPCxwQxtaGb77/2ETorvGWiwkvwiMA7882GPZT4l+RHgd4D1Qx7KoC2mc6pogs7R3yNJzq+q14c5qHl2DbC1qm5N8tPAl5KcV1X/b9gDWyhO5SODmTzi4m/6JFlM5/Dy2wMZ3dyb0SM9kvws8JvAFVX1/QGNbb6cqOa3A+cBk0lepnNudecCv4g8k/d5L7Czqv66ql4C/oJOOCxUM6l5A3AvQFX9KfBWOg90O1XN+SN8TuUwmMkjLnYC69r0VcBXql2dWYBOWG+Sfwb8Lp0gWOjnkeEENVfVoao6p6qWV9VyOtdJrqiqx4cz3Dkxk8/1f6dzVECSc+icNnpxgGOcazOp+RVgNUCSf0InDL410FEO1k7gunZX0cXAoap6rZ8NnrKnieo4j7hI8kng8araCdxJ53Byis7FmquHN+L+zLDe3wLOBH6/XSd/paquGNqg+zTDmk8pM6z5QeCSJM8CPwT+TVUt1CPemdZ8A/CfkvwanYvJ6xfwH3Yk+TKdQD+nXQe5CfhRgKr6Ip3rIpcDU8AbwEf63ucCfr0kSXPkVD5NJEmaIcNAkmQYSJIMA0kShoEkCcNAkoRhIEkC/j/+hJpxR/8AwwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.score.hist();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72eb890d",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.075325,
     "end_time": "2022-05-26T23:48:37.600945",
     "exception": false,
     "start_time": "2022-05-26T23:48:37.525620",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "There's a small number that are scored `1.0` - here's a sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "35dc7359",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:37.759844Z",
     "iopub.status.busy": "2022-05-26T23:48:37.758967Z",
     "iopub.status.idle": "2022-05-26T23:48:37.778086Z",
     "shell.execute_reply": "2022-05-26T23:48:37.778536Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.403266Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.101626,
     "end_time": "2022-05-26T23:48:37.778671",
     "exception": false,
     "start_time": "2022-05-26T23:48:37.677045",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>anchor</th>\n",
       "      <th>target</th>\n",
       "      <th>context</th>\n",
       "      <th>score</th>\n",
       "      <th>section</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>473137168ebf7484</td>\n",
       "      <td>abatement</td>\n",
       "      <td>abating</td>\n",
       "      <td>F24</td>\n",
       "      <td>1.0</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>621b048d70aa8867</td>\n",
       "      <td>absorbent properties</td>\n",
       "      <td>absorbent characteristics</td>\n",
       "      <td>D01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>bc20a1c961cb073a</td>\n",
       "      <td>absorbent properties</td>\n",
       "      <td>absorption properties</td>\n",
       "      <td>D01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>e955700dffd68624</td>\n",
       "      <td>acid absorption</td>\n",
       "      <td>absorption of acid</td>\n",
       "      <td>B08</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>3a09aba546aac675</td>\n",
       "      <td>acid absorption</td>\n",
       "      <td>acid absorption</td>\n",
       "      <td>B08</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36398</th>\n",
       "      <td>913141526432f1d6</td>\n",
       "      <td>wiring trough</td>\n",
       "      <td>wiring troughs</td>\n",
       "      <td>F16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36435</th>\n",
       "      <td>ee0746f2a8ecef97</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wood articles</td>\n",
       "      <td>B05</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36440</th>\n",
       "      <td>ecaf479135cf0dfd</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden article</td>\n",
       "      <td>B05</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36464</th>\n",
       "      <td>8ceaa2b5c2d56250</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wood article</td>\n",
       "      <td>B44</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36468</th>\n",
       "      <td>8e1386cbefd7f245</td>\n",
       "      <td>wood article</td>\n",
       "      <td>wooden article</td>\n",
       "      <td>B44</td>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1154 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     id                anchor                     target  \\\n",
       "28     473137168ebf7484             abatement                    abating   \n",
       "158    621b048d70aa8867  absorbent properties  absorbent characteristics   \n",
       "161    bc20a1c961cb073a  absorbent properties      absorption properties   \n",
       "311    e955700dffd68624       acid absorption         absorption of acid   \n",
       "315    3a09aba546aac675       acid absorption            acid absorption   \n",
       "...                 ...                   ...                        ...   \n",
       "36398  913141526432f1d6         wiring trough             wiring troughs   \n",
       "36435  ee0746f2a8ecef97          wood article              wood articles   \n",
       "36440  ecaf479135cf0dfd          wood article             wooden article   \n",
       "36464  8ceaa2b5c2d56250          wood article               wood article   \n",
       "36468  8e1386cbefd7f245          wood article             wooden article   \n",
       "\n",
       "      context  score section  \n",
       "28        F24    1.0       F  \n",
       "158       D01    1.0       D  \n",
       "161       D01    1.0       D  \n",
       "311       B08    1.0       B  \n",
       "315       B08    1.0       B  \n",
       "...       ...    ...     ...  \n",
       "36398     F16    1.0       F  \n",
       "36435     B05    1.0       B  \n",
       "36440     B05    1.0       B  \n",
       "36464     B44    1.0       B  \n",
       "36468     B44    1.0       B  \n",
       "\n",
       "[1154 rows x 6 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.score==1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74a6796",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.076061,
     "end_time": "2022-05-26T23:48:37.933914",
     "exception": false,
     "start_time": "2022-05-26T23:48:37.857853",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can see from this that these are just minor rewordings of the same concept, and isn't likely to be specific to `context`. Any pretrained model should be pretty good at finding these already."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e8f591",
   "metadata": {
    "heading_collapsed": true,
    "papermill": {
     "duration": 0.07727,
     "end_time": "2022-05-26T23:48:38.088140",
     "exception": false,
     "start_time": "2022-05-26T23:48:38.010870",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa19512",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.078886,
     "end_time": "2022-05-26T23:48:38.243132",
     "exception": false,
     "start_time": "2022-05-26T23:48:38.164246",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Time to import the stuff we'll need for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "592306d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:38.400276Z",
     "iopub.status.busy": "2022-05-26T23:48:38.399693Z",
     "iopub.status.idle": "2022-05-26T23:48:44.863949Z",
     "shell.execute_reply": "2022-05-26T23:48:44.863424Z",
     "shell.execute_reply.started": "2022-04-20T18:46:50.430672Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 6.545272,
     "end_time": "2022-05-26T23:48:44.864112",
     "exception": false,
     "start_time": "2022-05-26T23:48:38.318840",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "import warnings,transformers,logging,torch\n",
    "from transformers import TrainingArguments,Trainer\n",
    "from transformers import AutoModelForSequenceClassification,AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "68f7c708",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:45.022628Z",
     "iopub.status.busy": "2022-05-26T23:48:45.022113Z",
     "iopub.status.idle": "2022-05-26T23:48:52.491389Z",
     "shell.execute_reply": "2022-05-26T23:48:52.490842Z",
     "shell.execute_reply.started": "2022-04-20T18:46:57.749578Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 7.551731,
     "end_time": "2022-05-26T23:48:52.491519",
     "exception": false,
     "start_time": "2022-05-26T23:48:44.939788",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "if iskaggle:\n",
    "    !pip install -q datasets\n",
    "import datasets\n",
    "from datasets import load_dataset, Dataset, DatasetDict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b32b63e6",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.082233,
     "end_time": "2022-05-26T23:48:52.650944",
     "exception": false,
     "start_time": "2022-05-26T23:48:52.568711",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "HuggingFace Transformers tends to be rather enthusiastic about spitting out lots of warnings, so let's quieten it down for our sanity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "01335fbf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:52.817959Z",
     "iopub.status.busy": "2022-05-26T23:48:52.816950Z",
     "iopub.status.idle": "2022-05-26T23:48:52.820984Z",
     "shell.execute_reply": "2022-05-26T23:48:52.820565Z",
     "shell.execute_reply.started": "2022-04-20T18:47:07.041859Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.088878,
     "end_time": "2022-05-26T23:48:52.821135",
     "exception": false,
     "start_time": "2022-05-26T23:48:52.732257",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "warnings.simplefilter('ignore')\n",
    "logging.disable(logging.WARNING)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973b8bed",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.075715,
     "end_time": "2022-05-26T23:48:52.975046",
     "exception": false,
     "start_time": "2022-05-26T23:48:52.899331",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "I tried to find a model that I could train reasonably at home in under two minutes, but got reasonable accuracy from. I found that deberta-v3-small fits the bill, so let's use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a86fda90",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:53.133872Z",
     "iopub.status.busy": "2022-05-26T23:48:53.133200Z",
     "iopub.status.idle": "2022-05-26T23:48:53.135655Z",
     "shell.execute_reply": "2022-05-26T23:48:53.135254Z",
     "shell.execute_reply.started": "2022-04-20T18:47:07.053658Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.083988,
     "end_time": "2022-05-26T23:48:53.135761",
     "exception": false,
     "start_time": "2022-05-26T23:48:53.051773",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_nm = 'microsoft/deberta-v3-small'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4de4e426",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.075121,
     "end_time": "2022-05-26T23:48:53.287429",
     "exception": false,
     "start_time": "2022-05-26T23:48:53.212308",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can now create a tokenizer for this model. Note that pretrained models assume that text is tokenized in a particular way. In order to ensure that your tokenizer matches your model, use the `AutoTokenizer`, passing in your model name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f5f61cd0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:53.445607Z",
     "iopub.status.busy": "2022-05-26T23:48:53.444834Z",
     "iopub.status.idle": "2022-05-26T23:48:59.248761Z",
     "shell.execute_reply": "2022-05-26T23:48:59.248291Z",
     "shell.execute_reply.started": "2022-04-20T18:47:07.06182Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 5.885561,
     "end_time": "2022-05-26T23:48:59.248889",
     "exception": false,
     "start_time": "2022-05-26T23:48:53.363328",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f3854b5dae1c49fdb2b4681e9d318883",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/52.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "659e1d22c440485abcd2ad41d552118d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/578 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43c8b984bfc84d64972c1e711019caa2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/2.35M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokz = AutoTokenizer.from_pretrained(model_nm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "529122f1",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.077603,
     "end_time": "2022-05-26T23:48:59.405264",
     "exception": false,
     "start_time": "2022-05-26T23:48:59.327661",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We'll need to combine the context, anchor, and target together somehow. There's not much research as to the best way to do this, so we may need to iterate a bit. To start with, we'll just combine them all into a single string. The model will need to know where each section starts, so we can use the special separator token to tell it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a2032514",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:59.565257Z",
     "iopub.status.busy": "2022-05-26T23:48:59.564567Z",
     "iopub.status.idle": "2022-05-26T23:48:59.567178Z",
     "shell.execute_reply": "2022-05-26T23:48:59.567576Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.623301Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.08474,
     "end_time": "2022-05-26T23:48:59.567699",
     "exception": false,
     "start_time": "2022-05-26T23:48:59.482959",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[SEP]'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sep = tokz.sep_token\n",
    "sep"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a94c18bc",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.081577,
     "end_time": "2022-05-26T23:48:59.729030",
     "exception": false,
     "start_time": "2022-05-26T23:48:59.647453",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's now created our combined column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2c84dd31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:48:59.894475Z",
     "iopub.status.busy": "2022-05-26T23:48:59.893637Z",
     "iopub.status.idle": "2022-05-26T23:48:59.912278Z",
     "shell.execute_reply": "2022-05-26T23:48:59.911807Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.631622Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.103996,
     "end_time": "2022-05-26T23:48:59.912397",
     "exception": false,
     "start_time": "2022-05-26T23:48:59.808401",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df['inputs'] = df.context + sep + df.anchor + sep + df.target"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "786ef88e",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.077605,
     "end_time": "2022-05-26T23:49:00.067554",
     "exception": false,
     "start_time": "2022-05-26T23:48:59.989949",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Generally we'll get best performance if we convert pandas DataFrames into HuggingFace Datasets, so we'll convert them over, and also rename the score column to what Transformers expects for the dependent variable, which is `label`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a33d4fd7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:00.227171Z",
     "iopub.status.busy": "2022-05-26T23:49:00.226594Z",
     "iopub.status.idle": "2022-05-26T23:49:00.265372Z",
     "shell.execute_reply": "2022-05-26T23:49:00.264480Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.663871Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.120505,
     "end_time": "2022-05-26T23:49:00.265485",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.144980",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds = Dataset.from_pandas(df).rename_column('score', 'label')\n",
    "eval_ds = Dataset.from_pandas(eval_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2db21ab",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.076727,
     "end_time": "2022-05-26T23:49:00.419215",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.342488",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To tokenize the data, we'll create a function (since that's what `Dataset.map` will need):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "807d6c8d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:00.578037Z",
     "iopub.status.busy": "2022-05-26T23:49:00.577202Z",
     "iopub.status.idle": "2022-05-26T23:49:00.579054Z",
     "shell.execute_reply": "2022-05-26T23:49:00.579461Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.93232Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.083366,
     "end_time": "2022-05-26T23:49:00.579586",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.496220",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def tok_func(x): return tokz(x[\"inputs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce3981e",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.076625,
     "end_time": "2022-05-26T23:49:00.733520",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.656895",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's try tokenizing one input and see how it looks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d32bfc72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:00.898072Z",
     "iopub.status.busy": "2022-05-26T23:49:00.895876Z",
     "iopub.status.idle": "2022-05-26T23:49:00.901276Z",
     "shell.execute_reply": "2022-05-26T23:49:00.900807Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.940059Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.090515,
     "end_time": "2022-05-26T23:49:00.901391",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.810876",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [1, 336, 5753, 2, 47284, 2, 47284, 265, 6435, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tok_func(ds[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "299e87fd",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.078836,
     "end_time": "2022-05-26T23:49:01.057455",
     "exception": false,
     "start_time": "2022-05-26T23:49:00.978619",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The only bit we care about at the moment is `input_ids`. We can see in the tokens that it starts with a special token `1` (which represents the start of text), and then has our three fields separated by the separator token `2`. We can check the indices of the special token IDs like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b0aea492",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:01.219897Z",
     "iopub.status.busy": "2022-05-26T23:49:01.219251Z",
     "iopub.status.idle": "2022-05-26T23:49:01.221814Z",
     "shell.execute_reply": "2022-05-26T23:49:01.222288Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.951052Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.086698,
     "end_time": "2022-05-26T23:49:01.222428",
     "exception": false,
     "start_time": "2022-05-26T23:49:01.135730",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]', '[SEP]', '[UNK]', '[PAD]', '[MASK]']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokz.all_special_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9401b2",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.077401,
     "end_time": "2022-05-26T23:49:01.378694",
     "exception": false,
     "start_time": "2022-05-26T23:49:01.301293",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can now tokenize the input. We'll use batching to speed it up, and remove the columns we no longer need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5d848dc8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:01.544446Z",
     "iopub.status.busy": "2022-05-26T23:49:01.543695Z",
     "iopub.status.idle": "2022-05-26T23:49:08.560872Z",
     "shell.execute_reply": "2022-05-26T23:49:08.561282Z",
     "shell.execute_reply.started": "2022-04-20T18:47:14.960846Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 7.104938,
     "end_time": "2022-05-26T23:49:08.561440",
     "exception": false,
     "start_time": "2022-05-26T23:49:01.456502",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1467acc30bc443fba8ffbf7f1902f826",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/37 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "inps = \"anchor\",\"target\",\"context\"\n",
    "tok_ds = ds.map(tok_func, batched=True, remove_columns=inps+('inputs','id','section'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eea2523",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.080684,
     "end_time": "2022-05-26T23:49:08.722936",
     "exception": false,
     "start_time": "2022-05-26T23:49:08.642252",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Looking at the first item of the dataset we should see the same information as when we checked `tok_func` above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f16c285d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:08.899316Z",
     "iopub.status.busy": "2022-05-26T23:49:08.898606Z",
     "iopub.status.idle": "2022-05-26T23:49:08.901439Z",
     "shell.execute_reply": "2022-05-26T23:49:08.901841Z",
     "shell.execute_reply.started": "2022-04-20T18:47:21.821952Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.092635,
     "end_time": "2022-05-26T23:49:08.901981",
     "exception": false,
     "start_time": "2022-05-26T23:49:08.809346",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': 0.5,\n",
       " 'input_ids': [1, 336, 5753, 2, 47284, 2, 47284, 265, 6435, 2],\n",
       " 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tok_ds[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef0093e",
   "metadata": {
    "heading_collapsed": true,
    "papermill": {
     "duration": 0.081305,
     "end_time": "2022-05-26T23:49:09.064220",
     "exception": false,
     "start_time": "2022-05-26T23:49:08.982915",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Creating a validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f1927b",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.145391,
     "end_time": "2022-05-26T23:49:09.309208",
     "exception": false,
     "start_time": "2022-05-26T23:49:09.163817",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "According to [this post](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/315220), the private test anchors do not overlap with the training set. So let's do the same thing for our validation set.\n",
    "\n",
    "First, create a randomly shuffled list of anchors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f482ae35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:09.576982Z",
     "iopub.status.busy": "2022-05-26T23:49:09.576206Z",
     "iopub.status.idle": "2022-05-26T23:49:09.584481Z",
     "shell.execute_reply": "2022-05-26T23:49:09.585440Z",
     "shell.execute_reply.started": "2022-04-20T18:47:21.83246Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.146607,
     "end_time": "2022-05-26T23:49:09.585628",
     "exception": false,
     "start_time": "2022-05-26T23:49:09.439021",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['time digital signal', 'antiatherosclerotic', 'filled interior',\n",
       "       'dispersed powder', 'locking formation'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anchors = df.anchor.unique()\n",
    "np.random.seed(42)\n",
    "np.random.shuffle(anchors)\n",
    "anchors[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75182fc0",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.079819,
     "end_time": "2022-05-26T23:49:09.791606",
     "exception": false,
     "start_time": "2022-05-26T23:49:09.711787",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now we can pick some proportion (e.g 25%) of these anchors to go in the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f8a4ca9e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:09.955487Z",
     "iopub.status.busy": "2022-05-26T23:49:09.954714Z",
     "iopub.status.idle": "2022-05-26T23:49:09.957310Z",
     "shell.execute_reply": "2022-05-26T23:49:09.956874Z",
     "shell.execute_reply.started": "2022-04-20T18:47:21.845845Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.086896,
     "end_time": "2022-05-26T23:49:09.957427",
     "exception": false,
     "start_time": "2022-05-26T23:49:09.870531",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "val_prop = 0.25\n",
    "val_sz = int(len(anchors)*val_prop)\n",
    "val_anchors = anchors[:val_sz]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "916e03bb",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.079146,
     "end_time": "2022-05-26T23:49:10.115585",
     "exception": false,
     "start_time": "2022-05-26T23:49:10.036439",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now we can get a list of which rows match `val_anchors`, and get their indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "357304bb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:10.477599Z",
     "iopub.status.busy": "2022-05-26T23:49:10.476831Z",
     "iopub.status.idle": "2022-05-26T23:49:10.479645Z",
     "shell.execute_reply": "2022-05-26T23:49:10.480164Z",
     "shell.execute_reply.started": "2022-04-20T18:47:21.852371Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.285039,
     "end_time": "2022-05-26T23:49:10.480314",
     "exception": false,
     "start_time": "2022-05-26T23:49:10.195275",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9116, 27357)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_val = np.isin(df.anchor, val_anchors)\n",
    "idxs = np.arange(len(df))\n",
    "val_idxs = idxs[ is_val]\n",
    "trn_idxs = idxs[~is_val]\n",
    "len(val_idxs),len(trn_idxs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f75fb077",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.078856,
     "end_time": "2022-05-26T23:49:10.638706",
     "exception": false,
     "start_time": "2022-05-26T23:49:10.559850",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Our training and validation `Dataset`s can now be selected, and put into a `DatasetDict` ready for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d720f138",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:10.816201Z",
     "iopub.status.busy": "2022-05-26T23:49:10.815370Z",
     "iopub.status.idle": "2022-05-26T23:49:10.826389Z",
     "shell.execute_reply": "2022-05-26T23:49:10.825739Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.063211Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.10863,
     "end_time": "2022-05-26T23:49:10.826535",
     "exception": false,
     "start_time": "2022-05-26T23:49:10.717905",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dds = DatasetDict({\"train\":tok_ds.select(trn_idxs),\n",
    "             \"test\": tok_ds.select(val_idxs)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58d29d78",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.087747,
     "end_time": "2022-05-26T23:49:11.001347",
     "exception": false,
     "start_time": "2022-05-26T23:49:10.913600",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "BTW, a lot of people do more complex stuff for creating their validation set, but with a dataset this large there's not much point. As you can see, the mean scores in the two groups are very similar despite just doing a random shuffle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "edb62d46",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:11.168671Z",
     "iopub.status.busy": "2022-05-26T23:49:11.167767Z",
     "iopub.status.idle": "2022-05-26T23:49:11.180869Z",
     "shell.execute_reply": "2022-05-26T23:49:11.181271Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.086933Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.097132,
     "end_time": "2022-05-26T23:49:11.181421",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.084289",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3623021530138539, 0.3613426941641071)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[trn_idxs].score.mean(),df.iloc[val_idxs].score.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc0dc2c",
   "metadata": {
    "heading_collapsed": true,
    "papermill": {
     "duration": 0.08042,
     "end_time": "2022-05-26T23:49:11.345039",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.264619",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Initial model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed272818",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.079204,
     "end_time": "2022-05-26T23:49:11.504384",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.425180",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's now train our model! We'll need to specify a metric, which is the correlation coefficient provided by numpy (we need to return a dictionary since that's how Transformers knows what label to use):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "157cc8c8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:11.668505Z",
     "iopub.status.busy": "2022-05-26T23:49:11.667688Z",
     "iopub.status.idle": "2022-05-26T23:49:11.670194Z",
     "shell.execute_reply": "2022-05-26T23:49:11.669761Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.108412Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.086335,
     "end_time": "2022-05-26T23:49:11.670315",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.583980",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def corr(eval_pred): return {'pearson': np.corrcoef(*eval_pred)[0][1]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a756e9f4",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.081379,
     "end_time": "2022-05-26T23:49:11.831306",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.749927",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We pick a learning rate and batch size that fits our GPU, and pick a reasonable weight decay and small number of epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "70c9599f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:11.997397Z",
     "iopub.status.busy": "2022-05-26T23:49:11.995783Z",
     "iopub.status.idle": "2022-05-26T23:49:11.997963Z",
     "shell.execute_reply": "2022-05-26T23:49:11.998400Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.114437Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.086929,
     "end_time": "2022-05-26T23:49:11.998534",
     "exception": false,
     "start_time": "2022-05-26T23:49:11.911605",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "lr,bs = 8e-5,128\n",
    "wd,epochs = 0.01,4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ea9285b",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.087644,
     "end_time": "2022-05-26T23:49:12.166347",
     "exception": false,
     "start_time": "2022-05-26T23:49:12.078703",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Three epochs might not sound like much, but you'll see once we train that most of the progress can be made in that time, so this is good for experimentation.\n",
    "\n",
    "Transformers uses the `TrainingArguments` class to set up arguments. We'll use a cosine scheduler with warmup, since at fast.ai we've found that's pretty reliable. We'll use fp16 since it's much faster on modern GPUs, and saves some memory. We evaluate using double-sized batches, since no gradients are stored so we can do twice as many rows at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "dfeadf5a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:12.336874Z",
     "iopub.status.busy": "2022-05-26T23:49:12.336030Z",
     "iopub.status.idle": "2022-05-26T23:49:12.337753Z",
     "shell.execute_reply": "2022-05-26T23:49:12.338203Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.12488Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.090935,
     "end_time": "2022-05-26T23:49:12.338343",
     "exception": false,
     "start_time": "2022-05-26T23:49:12.247408",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_trainer(dds):\n",
    "    args = TrainingArguments('outputs', learning_rate=lr, warmup_ratio=0.1, lr_scheduler_type='cosine', fp16=True,\n",
    "        evaluation_strategy=\"epoch\", per_device_train_batch_size=bs, per_device_eval_batch_size=bs*2,\n",
    "        num_train_epochs=epochs, weight_decay=wd, report_to='none')\n",
    "    model = AutoModelForSequenceClassification.from_pretrained(model_nm, num_labels=1)\n",
    "    return Trainer(model, args, train_dataset=dds['train'], eval_dataset=dds['test'],\n",
    "                   tokenizer=tokz, compute_metrics=corr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "cc3509d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:12.504777Z",
     "iopub.status.busy": "2022-05-26T23:49:12.503811Z",
     "iopub.status.idle": "2022-05-26T23:49:12.581378Z",
     "shell.execute_reply": "2022-05-26T23:49:12.580759Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.134979Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 0.162751,
     "end_time": "2022-05-26T23:49:12.581500",
     "exception": false,
     "start_time": "2022-05-26T23:49:12.418749",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "args = TrainingArguments('outputs', learning_rate=lr, warmup_ratio=0.1, lr_scheduler_type='cosine', fp16=True,\n",
    "    evaluation_strategy=\"epoch\", per_device_train_batch_size=bs, per_device_eval_batch_size=bs*2,\n",
    "    num_train_epochs=epochs, weight_decay=wd, report_to='none')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2dbd62",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.080002,
     "end_time": "2022-05-26T23:49:12.742598",
     "exception": false,
     "start_time": "2022-05-26T23:49:12.662596",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can now create our model, and `Trainer`, which is a class which combines the data and model together (just like `Learner` in fastai):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "fcf94aee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:12.908374Z",
     "iopub.status.busy": "2022-05-26T23:49:12.907570Z",
     "iopub.status.idle": "2022-05-26T23:49:33.726504Z",
     "shell.execute_reply": "2022-05-26T23:49:33.725566Z",
     "shell.execute_reply.started": "2022-04-20T18:47:22.204868Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 20.903622,
     "end_time": "2022-05-26T23:49:33.726659",
     "exception": false,
     "start_time": "2022-05-26T23:49:12.823037",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7bbfe71ed4f347fe85fb30b5b2382e0f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/273M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = AutoModelForSequenceClassification.from_pretrained(model_nm, num_labels=1)\n",
    "trainer = Trainer(model, args, train_dataset=dds['train'], eval_dataset=dds['test'],\n",
    "               tokenizer=tokz, compute_metrics=corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c64f6ef",
   "metadata": {
    "hidden": true,
    "papermill": {
     "duration": 0.080501,
     "end_time": "2022-05-26T23:49:33.888455",
     "exception": false,
     "start_time": "2022-05-26T23:49:33.807954",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's train our model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "24690b4f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:49:34.059065Z",
     "iopub.status.busy": "2022-05-26T23:49:34.058263Z",
     "iopub.status.idle": "2022-05-26T23:53:54.575338Z",
     "shell.execute_reply": "2022-05-26T23:53:54.574877Z",
     "shell.execute_reply.started": "2022-04-20T18:47:43.411773Z"
    },
    "hidden": true,
    "papermill": {
     "duration": 260.607107,
     "end_time": "2022-05-26T23:53:54.575476",
     "exception": false,
     "start_time": "2022-05-26T23:49:33.968369",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='856' max='856' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [856/856 04:19, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Pearson</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.025167</td>\n",
       "      <td>0.798359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.025149</td>\n",
       "      <td>0.803286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.035300</td>\n",
       "      <td>0.024344</td>\n",
       "      <td>0.815202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.035300</td>\n",
       "      <td>0.024549</td>\n",
       "      <td>0.815378</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.train();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832e4491",
   "metadata": {
    "papermill": {
     "duration": 0.081378,
     "end_time": "2022-05-26T23:53:54.743198",
     "exception": false,
     "start_time": "2022-05-26T23:53:54.661820",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Improving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8565a139",
   "metadata": {
    "papermill": {
     "duration": 0.079935,
     "end_time": "2022-05-26T23:53:54.903886",
     "exception": false,
     "start_time": "2022-05-26T23:53:54.823951",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We now want to start iterating to improve this. To do that, we need to know whether the model gives stable results. I tried training it 3 times from scratch, and got a range of outcomes from 0.808-0.810. This is stable enough to make a start - if we're not finding improvements that are visible within this range, then they're not very significant! Later on, if and when we feel confident that we've got the basics right, we can use cross validation and more epochs of training.\n",
    "\n",
    "Iteration speed is critical, so we need to quickly be able to try different data processing and trainer parameters. So let's create a function to quickly apply tokenization and create our `DatasetDict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "fe9f32fe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:53:55.075230Z",
     "iopub.status.busy": "2022-05-26T23:53:55.074576Z",
     "iopub.status.idle": "2022-05-26T23:53:55.077065Z",
     "shell.execute_reply": "2022-05-26T23:53:55.077576Z",
     "shell.execute_reply.started": "2022-04-20T18:52:07.51356Z"
    },
    "papermill": {
     "duration": 0.092376,
     "end_time": "2022-05-26T23:53:55.077718",
     "exception": false,
     "start_time": "2022-05-26T23:53:54.985342",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_dds(df):\n",
    "    ds = Dataset.from_pandas(df).rename_column('score', 'label')\n",
    "    tok_ds = ds.map(tok_func, batched=True, remove_columns=inps+('inputs','id','section'))\n",
    "    return DatasetDict({\"train\":tok_ds.select(trn_idxs), \"test\": tok_ds.select(val_idxs)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b8396a2",
   "metadata": {
    "papermill": {
     "duration": 0.166417,
     "end_time": "2022-05-26T23:53:55.351838",
     "exception": false,
     "start_time": "2022-05-26T23:53:55.185421",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "...and also a function to create a `Trainer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "db1aad08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:53:55.641542Z",
     "iopub.status.busy": "2022-05-26T23:53:55.640114Z",
     "iopub.status.idle": "2022-05-26T23:53:55.642535Z",
     "shell.execute_reply": "2022-05-26T23:53:55.640800Z",
     "shell.execute_reply.started": "2022-04-20T18:52:07.522276Z"
    },
    "papermill": {
     "duration": 0.147877,
     "end_time": "2022-05-26T23:53:55.642702",
     "exception": false,
     "start_time": "2022-05-26T23:53:55.494825",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_model(): return AutoModelForSequenceClassification.from_pretrained(model_nm, num_labels=1)\n",
    "\n",
    "def get_trainer(dds, model=None):\n",
    "    if model is None: model = get_model()\n",
    "    args = TrainingArguments('outputs', learning_rate=lr, warmup_ratio=0.1, lr_scheduler_type='cosine', fp16=True,\n",
    "        evaluation_strategy=\"epoch\", per_device_train_batch_size=bs, per_device_eval_batch_size=bs*2,\n",
    "        num_train_epochs=epochs, weight_decay=wd, report_to='none')\n",
    "    return Trainer(model, args, train_dataset=dds['train'], eval_dataset=dds['test'],\n",
    "                   tokenizer=tokz, compute_metrics=corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdb358a",
   "metadata": {
    "papermill": {
     "duration": 0.081728,
     "end_time": "2022-05-26T23:53:55.831661",
     "exception": false,
     "start_time": "2022-05-26T23:53:55.749933",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's now try out some ideas...\n",
    "\n",
    "Perhaps using the special separator character isn't a good idea, and we should use something we create instead. Let's see if that makes things better. First we'll change the separator and create the `DatasetDict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "038e3d6f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:53:56.026397Z",
     "iopub.status.busy": "2022-05-26T23:53:56.003728Z",
     "iopub.status.idle": "2022-05-26T23:54:02.736791Z",
     "shell.execute_reply": "2022-05-26T23:54:02.737203Z",
     "shell.execute_reply.started": "2022-04-20T18:56:47.88635Z"
    },
    "papermill": {
     "duration": 6.823235,
     "end_time": "2022-05-26T23:54:02.737364",
     "exception": false,
     "start_time": "2022-05-26T23:53:55.914129",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cab7d1e3479748b285f90645beae28ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/37 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sep = \" [s] \"\n",
    "df['inputs'] = df.context + sep + df.anchor + sep + df.target\n",
    "dds = get_dds(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2992eedf",
   "metadata": {
    "papermill": {
     "duration": 0.081832,
     "end_time": "2022-05-26T23:54:02.903232",
     "exception": false,
     "start_time": "2022-05-26T23:54:02.821400",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "...and create and train a model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2bdd9185",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:54:03.076186Z",
     "iopub.status.busy": "2022-05-26T23:54:03.072754Z",
     "iopub.status.idle": "2022-05-26T23:58:35.064324Z",
     "shell.execute_reply": "2022-05-26T23:58:35.064770Z",
     "shell.execute_reply.started": "2022-04-20T18:56:55.225892Z"
    },
    "papermill": {
     "duration": 272.079294,
     "end_time": "2022-05-26T23:58:35.064938",
     "exception": false,
     "start_time": "2022-05-26T23:54:02.985644",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='856' max='856' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [856/856 04:29, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Pearson</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.024909</td>\n",
       "      <td>0.797869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.024800</td>\n",
       "      <td>0.812953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.031600</td>\n",
       "      <td>0.024418</td>\n",
       "      <td>0.818292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.031600</td>\n",
       "      <td>0.024037</td>\n",
       "      <td>0.819089</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=856, training_loss=0.023822079752093165, metrics={'train_runtime': 269.6383, 'train_samples_per_second': 405.833, 'train_steps_per_second': 3.175, 'total_flos': 582160588599300.0, 'train_loss': 0.023822079752093165, 'epoch': 4.0})"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_trainer(dds).train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "762927f7",
   "metadata": {
    "papermill": {
     "duration": 0.082215,
     "end_time": "2022-05-26T23:58:35.233774",
     "exception": false,
     "start_time": "2022-05-26T23:58:35.151559",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "That's looking quite a bit better, so we'll keep that change.\n",
    "\n",
    "Often changing to lowercase is helpful. Let's see if that helps too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "93d8518d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-26T23:58:35.411243Z",
     "iopub.status.busy": "2022-05-26T23:58:35.405717Z",
     "iopub.status.idle": "2022-05-27T00:03:18.960349Z",
     "shell.execute_reply": "2022-05-27T00:03:18.959862Z",
     "shell.execute_reply.started": "2022-04-20T19:01:39.375999Z"
    },
    "papermill": {
     "duration": 283.643852,
     "end_time": "2022-05-27T00:03:18.960499",
     "exception": false,
     "start_time": "2022-05-26T23:58:35.316647",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6384b710b9c547b6b65ebad15ef55b6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/37 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='856' max='856' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [856/856 04:34, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Pearson</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.025170</td>\n",
       "      <td>0.798002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.024433</td>\n",
       "      <td>0.815301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.031500</td>\n",
       "      <td>0.024575</td>\n",
       "      <td>0.818443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.031500</td>\n",
       "      <td>0.024150</td>\n",
       "      <td>0.818868</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=856, training_loss=0.023811190484840178, metrics={'train_runtime': 274.7918, 'train_samples_per_second': 398.222, 'train_steps_per_second': 3.115, 'total_flos': 582160588599300.0, 'train_loss': 0.023811190484840178, 'epoch': 4.0})"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['inputs'] = df.inputs.str.lower()\n",
    "dds = get_dds(df)\n",
    "get_trainer(dds).train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f341ac4",
   "metadata": {
    "papermill": {
     "duration": 0.088188,
     "end_time": "2022-05-27T00:03:19.136761",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.048573",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "That one is less clear. We'll keep that change too since most times I run it, it's a little better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "247c466d",
   "metadata": {
    "papermill": {
     "duration": 0.083467,
     "end_time": "2022-05-27T00:03:19.361893",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.278426",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Special tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a81b29c9",
   "metadata": {
    "papermill": {
     "duration": 0.0848,
     "end_time": "2022-05-27T00:03:19.530841",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.446041",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "What if we made the patent section a special token? Then potentially the model might learn to recognize that different sections need to be handled in different ways. To do that, we'll use, e.g. `[A]` for section A. We'll then add those as special tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e8f2faf9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:03:19.718578Z",
     "iopub.status.busy": "2022-05-27T00:03:19.710686Z",
     "iopub.status.idle": "2022-05-27T00:03:19.723492Z",
     "shell.execute_reply": "2022-05-27T00:03:19.723037Z",
     "shell.execute_reply.started": "2022-04-20T19:06:34.192388Z"
    },
    "papermill": {
     "duration": 0.105458,
     "end_time": "2022-05-27T00:03:19.723609",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.618151",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['sectok'] = '[' + df.section + ']'\n",
    "sectoks = list(df.sectok.unique())\n",
    "tokz.add_special_tokens({'additional_special_tokens': sectoks})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bc1454",
   "metadata": {
    "papermill": {
     "duration": 0.085,
     "end_time": "2022-05-27T00:03:19.894784",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.809784",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We concatenate the section token to the start of our inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "790df141",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:03:20.071605Z",
     "iopub.status.busy": "2022-05-27T00:03:20.070780Z",
     "iopub.status.idle": "2022-05-27T00:03:27.303590Z",
     "shell.execute_reply": "2022-05-27T00:03:27.304244Z",
     "shell.execute_reply.started": "2022-04-20T19:06:34.222287Z"
    },
    "papermill": {
     "duration": 7.324227,
     "end_time": "2022-05-27T00:03:27.304455",
     "exception": false,
     "start_time": "2022-05-27T00:03:19.980228",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "828b42f02cd648fa88c3e62b458dc941",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/37 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df['inputs'] = df.sectok + sep + df.context + sep + df.anchor.str.lower() + sep + df.target\n",
    "dds = get_dds(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49408de4",
   "metadata": {
    "papermill": {
     "duration": 0.143953,
     "end_time": "2022-05-27T00:03:27.593621",
     "exception": false,
     "start_time": "2022-05-27T00:03:27.449668",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Since we've added more tokens, we need to resize the embedding matrix in the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a9273412",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:03:27.822131Z",
     "iopub.status.busy": "2022-05-27T00:03:27.821586Z",
     "iopub.status.idle": "2022-05-27T00:03:31.674273Z",
     "shell.execute_reply": "2022-05-27T00:03:31.673585Z",
     "shell.execute_reply.started": "2022-04-20T19:06:43.885693Z"
    },
    "papermill": {
     "duration": 3.944905,
     "end_time": "2022-05-27T00:03:31.674405",
     "exception": false,
     "start_time": "2022-05-27T00:03:27.729500",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(128009, 768)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = get_model()\n",
    "model.resize_token_embeddings(len(tokz))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d06a0fdd",
   "metadata": {
    "papermill": {
     "duration": 0.084183,
     "end_time": "2022-05-27T00:03:31.843916",
     "exception": false,
     "start_time": "2022-05-27T00:03:31.759733",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now we're ready to train:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "7cbce073",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:03:32.017291Z",
     "iopub.status.busy": "2022-05-27T00:03:32.016412Z",
     "iopub.status.idle": "2022-05-27T00:08:28.336176Z",
     "shell.execute_reply": "2022-05-27T00:08:28.335673Z",
     "shell.execute_reply.started": "2022-04-20T19:06:48.116453Z"
    },
    "papermill": {
     "duration": 296.407944,
     "end_time": "2022-05-27T00:08:28.336313",
     "exception": false,
     "start_time": "2022-05-27T00:03:31.928369",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='856' max='856' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [856/856 04:55, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Pearson</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.024835</td>\n",
       "      <td>0.797996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.024412</td>\n",
       "      <td>0.812386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.031800</td>\n",
       "      <td>0.024019</td>\n",
       "      <td>0.820914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.031800</td>\n",
       "      <td>0.024220</td>\n",
       "      <td>0.820187</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=856, training_loss=0.023861238889605084, metrics={'train_runtime': 296.1519, 'train_samples_per_second': 369.5, 'train_steps_per_second': 2.89, 'total_flos': 695409809982180.0, 'train_loss': 0.023861238889605084, 'epoch': 4.0})"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer = get_trainer(dds, model=model)\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21ff3464",
   "metadata": {
    "papermill": {
     "duration": 0.085191,
     "end_time": "2022-05-27T00:08:28.510347",
     "exception": false,
     "start_time": "2022-05-27T00:08:28.425156",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "It looks like we've made another bit of an improvement!\n",
    "\n",
    "There's plenty more things you could try. Here's some thoughts:\n",
    "\n",
    "- Try a model pretrained on legal vocabulary. E.g. how about [BERT for patents](https://huggingface.co/anferico/bert-for-patents)?\n",
    "- You'd likely get better results by using a sentence similarity model. Did you know that there's a [patent similarity model](https://huggingface.co/AI-Growth-Lab/PatentSBERTa) you could try?\n",
    "- You could also fine-tune any HuggingFace model using the full patent database (which is provided in BigQuery), before applying it to this dataset\n",
    "- Replace the patent context field with the description of that context provided by the patent office\n",
    "- ...and try out your own ideas too!\n",
    "\n",
    "Before submitting a model, retrain it on the full dataset, rather than just the 75% training subset we've used here. Create a function like the ones above to make that easy for you!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aee600e6",
   "metadata": {
    "papermill": {
     "duration": 0.085454,
     "end_time": "2022-05-27T00:08:28.681157",
     "exception": false,
     "start_time": "2022-05-27T00:08:28.595703",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "178e9e39",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:28.861424Z",
     "iopub.status.busy": "2022-05-27T00:08:28.860447Z",
     "iopub.status.idle": "2022-05-27T00:08:28.864741Z",
     "shell.execute_reply": "2022-05-27T00:08:28.864341Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.414585Z"
    },
    "papermill": {
     "duration": 0.095336,
     "end_time": "2022-05-27T00:08:28.864853",
     "exception": false,
     "start_time": "2022-05-27T00:08:28.769517",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "n_folds = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "346f8f81",
   "metadata": {
    "papermill": {
     "duration": 0.086984,
     "end_time": "2022-05-27T00:08:29.038727",
     "exception": false,
     "start_time": "2022-05-27T00:08:28.951743",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once you've gotten the low hanging fruit, you might want to use cross-validation to see the impact of minor changes. This time we'll use [StratifiedGroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html#sklearn.model_selection.StratifiedGroupKFold), partly just to show a different approach to before, and partly because it will give us slightly better balanced datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "34ee63b9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:29.223781Z",
     "iopub.status.busy": "2022-05-27T00:08:29.223133Z",
     "iopub.status.idle": "2022-05-27T00:08:29.238553Z",
     "shell.execute_reply": "2022-05-27T00:08:29.238126Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.422096Z"
    },
    "papermill": {
     "duration": 0.105978,
     "end_time": "2022-05-27T00:08:29.238660",
     "exception": false,
     "start_time": "2022-05-27T00:08:29.132682",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedGroupKFold\n",
    "cv = StratifiedGroupKFold(n_splits=n_folds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4908de",
   "metadata": {
    "papermill": {
     "duration": 0.084724,
     "end_time": "2022-05-27T00:08:29.408695",
     "exception": false,
     "start_time": "2022-05-27T00:08:29.323971",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Here's how to split the data frame into `n_folds` groups, with non-overlapping anchors and matched scores, after randomly shuffling the rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ddf9b58e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:29.585388Z",
     "iopub.status.busy": "2022-05-27T00:08:29.584527Z",
     "iopub.status.idle": "2022-05-27T00:08:29.947281Z",
     "shell.execute_reply": "2022-05-27T00:08:29.947791Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.449715Z"
    },
    "papermill": {
     "duration": 0.453549,
     "end_time": "2022-05-27T00:08:29.947937",
     "exception": false,
     "start_time": "2022-05-27T00:08:29.494388",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(array([    0,     1,     2, ..., 36469, 36471, 36472]),\n",
       "  array([    8,    13,    14, ..., 36453, 36464, 36470])),\n",
       " (array([    0,     1,     5, ..., 36470, 36471, 36472]),\n",
       "  array([    2,     3,     4, ..., 36459, 36461, 36462])),\n",
       " (array([    1,     2,     3, ..., 36467, 36470, 36472]),\n",
       "  array([    0,     7,    11, ..., 36468, 36469, 36471])),\n",
       " (array([    0,     2,     3, ..., 36469, 36470, 36471]),\n",
       "  array([    1,     5,     9, ..., 36465, 36467, 36472]))]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.sample(frac=1, random_state=42)\n",
    "scores = (df.score*100).astype(int)\n",
    "folds = list(cv.split(idxs, scores, df.anchor))\n",
    "folds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bfb23e2",
   "metadata": {
    "papermill": {
     "duration": 0.085962,
     "end_time": "2022-05-27T00:08:30.119719",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.033757",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can now create a little function to split into training and validation sets based on a fold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "3ee351ea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:30.296872Z",
     "iopub.status.busy": "2022-05-27T00:08:30.296049Z",
     "iopub.status.idle": "2022-05-27T00:08:30.298063Z",
     "shell.execute_reply": "2022-05-27T00:08:30.298461Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.889254Z"
    },
    "papermill": {
     "duration": 0.092828,
     "end_time": "2022-05-27T00:08:30.298598",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.205770",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_fold(folds, fold_num):\n",
    "    trn,val = folds[fold_num]\n",
    "    return DatasetDict({\"train\":tok_ds.select(trn), \"test\": tok_ds.select(val)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c709ae",
   "metadata": {
    "papermill": {
     "duration": 0.086035,
     "end_time": "2022-05-27T00:08:30.469738",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.383703",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b23334b2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:30.646981Z",
     "iopub.status.busy": "2022-05-27T00:08:30.646251Z",
     "iopub.status.idle": "2022-05-27T00:08:30.665028Z",
     "shell.execute_reply": "2022-05-27T00:08:30.664610Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.90085Z"
    },
    "papermill": {
     "duration": 0.109301,
     "end_time": "2022-05-27T00:08:30.665148",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.555847",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
       "        num_rows: 27346\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
       "        num_rows: 9127\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dds = get_fold(folds, 0)\n",
    "dds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb23e14a",
   "metadata": {
    "papermill": {
     "duration": 0.086924,
     "end_time": "2022-05-27T00:08:30.839878",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.752954",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can now pass this into `get_trainer` as we did before. If we have, say, 4 folds, then doing that for each fold will give us 4 models, and 4 sets of predictions and metrics. You could ensemble the 4 models to get a stronger model, and can also average the 4 metrics to get a more accurate assessment of your model. Here's how to get the final epoch metrics from a trainer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "1e1398ce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-27T00:08:31.018493Z",
     "iopub.status.busy": "2022-05-27T00:08:31.017861Z",
     "iopub.status.idle": "2022-05-27T00:08:31.020485Z",
     "shell.execute_reply": "2022-05-27T00:08:31.020877Z",
     "shell.execute_reply.started": "2022-04-20T19:11:55.932608Z"
    },
    "papermill": {
     "duration": 0.094653,
     "end_time": "2022-05-27T00:08:31.021017",
     "exception": false,
     "start_time": "2022-05-27T00:08:30.926364",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8201874392079798"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics = [o['eval_pearson'] for o in trainer.state.log_history if 'eval_pearson' in o]\n",
    "metrics[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f27c5616",
   "metadata": {
    "papermill": {
     "duration": 0.086657,
     "end_time": "2022-05-27T00:08:31.196112",
     "exception": false,
     "start_time": "2022-05-27T00:08:31.109455",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "I hope you've found this a helpful guide to improving your results in this competition - and on Kaggle more generally! If you like it, please remember to give it an upvote, and don't hesitate to add a comment if you have any questions or thoughts to add. And if the ideas here are helpful to you in creating your models, I'd really appreciate a link back to this notebook or a comment below to let me know what helped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfe5b959",
   "metadata": {
    "papermill": {
     "duration": 0.087962,
     "end_time": "2022-05-27T00:08:31.371721",
     "exception": false,
     "start_time": "2022-05-27T00:08:31.283759",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1229.954534,
   "end_time": "2022-05-27T00:08:34.729023",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-05-26T23:48:04.774489",
   "version": "2.3.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "050231bb17744e4e826344ca8e281608": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_818cd4fa69da452d8854ff5ecdd95554",
       "placeholder": "​",
       "style": "IPY_MODEL_0bdf748db10044fdb4150d9854efb2f7",
       "value": " 37/37 [00:05&lt;00:00,  5.41ba/s]"
      }
     },
     "06b4ea22503946fc90340347b3fa5907": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0bdf748db10044fdb4150d9854efb2f7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "0d0121284ee24da3b00b1da4742b716d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "110c574feb3e47a8b17971c130fbb7aa": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_54f6428beee9491681612ee89d9445a9",
       "placeholder": "​",
       "style": "IPY_MODEL_1b2c25bf99a7441483655aeb990e50fd",
       "value": " 273M/273M [00:12&lt;00:00, 23.4MB/s]"
      }
     },
     "1467acc30bc443fba8ffbf7f1902f826": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_babf6ff76d274cf9938a3da46b444f96",
        "IPY_MODEL_fa969b8dcc5248ddb411ec64f24f680e",
        "IPY_MODEL_8d5db6bd65094b5195e0423918e168bc"
       ],
       "layout": "IPY_MODEL_fa877bca33d841d693a5dbfb9a0b5e59"
      }
     },
     "147ddffcf12b4e168ee235a18425a38d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "175b7f150f4544acbfe448aff53bfdb4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_f5fb5dff95e84a18909bcc7df17f0d43",
       "placeholder": "​",
       "style": "IPY_MODEL_47ad56e881664b819779ef96ac918c1d",
       "value": "Downloading: 100%"
      }
     },
     "19b1e9c3855a4154830525bab9fa35c4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_8aefaab967ee478992b35917d02848ff",
       "placeholder": "​",
       "style": "IPY_MODEL_6b692610c99d4a4ca7d11ba8e06c51aa",
       "value": " 578/578 [00:00&lt;00:00, 20.7kB/s]"
      }
     },
     "19f27ae15914434e9f60ac3378c05467": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "1b2c25bf99a7441483655aeb990e50fd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "1bd68bb6190340278bade7a25cea1783": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "21579d3a660c4aedb6b281b0428632be": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b2cc45f29e7b40ca81bc6c01d6702b3e",
       "max": 52,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_147ddffcf12b4e168ee235a18425a38d",
       "value": 52
      }
     },
     "21e583b7f8854e27a459bdac51e09ac3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_63db4e9aca6b4c34b149f2a54c26d64e",
       "max": 286059269,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_d75db2d90ad54aa58cba80c81f35278d",
       "value": 286059269
      }
     },
     "252cbf7aee864971aace7ce7df4e8735": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "2e7c4891c1694af8b58b4066e0ed88d2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_637dc11668ad42cd991702c171972c46",
       "placeholder": "​",
       "style": "IPY_MODEL_c6197f1a4f3c479580e0dc4fbe18d5da",
       "value": " 37/37 [00:06&lt;00:00,  6.15ba/s]"
      }
     },
     "2f1bdd816af64bd0a6bc136b7fcd61aa": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "35745679468d453097e87a74bc0916ee": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "370dcf1a1f774c728f64364d2afab8bb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_8e3838a4b0e54c609642c88c938e1637",
       "placeholder": "​",
       "style": "IPY_MODEL_8d5e23c9af0b468195b8cd60f7f70cd7",
       "value": "100%"
      }
     },
     "3a2cf86b2fac4c9996c0b49e1877c0e2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3c4cb343c8a147d7bdf3baa08e5e7111": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "422ce5f4010d4c71beac4b21fd3c110a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "43c8b984bfc84d64972c1e711019caa2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_8b275b1882014fae97770cb75caf072c",
        "IPY_MODEL_f27ee53d013e4650830b4e50e2f510e4",
        "IPY_MODEL_f27e4dda852d401b9d4f6ef2a09078c1"
       ],
       "layout": "IPY_MODEL_9fb6f54e469e42849b7c7c1ed5e098a2"
      }
     },
     "47ad56e881664b819779ef96ac918c1d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "546dcdde65dd427389d97c735983e3bd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_8be31b43571f4855871157743a25319b",
       "max": 37,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_252cbf7aee864971aace7ce7df4e8735",
       "value": 37
      }
     },
     "54f6428beee9491681612ee89d9445a9": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "578fb6544f0045a6aa14160acd74a546": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "58d2f87bc7d646cfb1cdf254061087b4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_a839a234dbeb464ebc518be49e9ffec8",
       "max": 37,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_d2850a8aec3f4bf89c2172c84224d9bc",
       "value": 37
      }
     },
     "637dc11668ad42cd991702c171972c46": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6384b710b9c547b6b65ebad15ef55b6a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_370dcf1a1f774c728f64364d2afab8bb",
        "IPY_MODEL_58d2f87bc7d646cfb1cdf254061087b4",
        "IPY_MODEL_050231bb17744e4e826344ca8e281608"
       ],
       "layout": "IPY_MODEL_19f27ae15914434e9f60ac3378c05467"
      }
     },
     "63db4e9aca6b4c34b149f2a54c26d64e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "659e1d22c440485abcd2ad41d552118d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e4a8c7e62f2f4c57a33071808ffe7522",
        "IPY_MODEL_9895557627e24a5abe1a1ac9f2eb370d",
        "IPY_MODEL_19b1e9c3855a4154830525bab9fa35c4"
       ],
       "layout": "IPY_MODEL_0d0121284ee24da3b00b1da4742b716d"
      }
     },
     "6b692610c99d4a4ca7d11ba8e06c51aa": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "6bb4e6d0610b4955a09a74c5a1483b98": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6ca4d325d75f4d3785e17017eb2796eb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bdc371752d474e5d82402e7a63d76aa8",
       "placeholder": "​",
       "style": "IPY_MODEL_d63f3cb652db424c9dab6d0021b53a46",
       "value": " 52.0/52.0 [00:00&lt;00:00, 2.08kB/s]"
      }
     },
     "7bbbcd210d1546869cad760af83e4829": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "7bbfe71ed4f347fe85fb30b5b2382e0f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_175b7f150f4544acbfe448aff53bfdb4",
        "IPY_MODEL_21e583b7f8854e27a459bdac51e09ac3",
        "IPY_MODEL_110c574feb3e47a8b17971c130fbb7aa"
       ],
       "layout": "IPY_MODEL_bcfb4f3e5adb4570a8dcf47f5b30ac6d"
      }
     },
     "818cd4fa69da452d8854ff5ecdd95554": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "828b42f02cd648fa88c3e62b458dc941": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e9b2d5a85a2b4c409aaab1b4a4f0f895",
        "IPY_MODEL_546dcdde65dd427389d97c735983e3bd",
        "IPY_MODEL_2e7c4891c1694af8b58b4066e0ed88d2"
       ],
       "layout": "IPY_MODEL_91a073c74ffa49258c43d64f718afcf7"
      }
     },
     "8a1ba1721f6a47a5b9773ad1738b0dfb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "8aefaab967ee478992b35917d02848ff": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8b275b1882014fae97770cb75caf072c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_422ce5f4010d4c71beac4b21fd3c110a",
       "placeholder": "​",
       "style": "IPY_MODEL_c147e1e4451d45f6bba56c16dec9ac3e",
       "value": "Downloading: 100%"
      }
     },
     "8be31b43571f4855871157743a25319b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8c72a81d99cf4194a8ac0ca825e66f42": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "8d5db6bd65094b5195e0423918e168bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_3c4cb343c8a147d7bdf3baa08e5e7111",
       "placeholder": "​",
       "style": "IPY_MODEL_90dc86f9a77b46958015996a98d00e33",
       "value": " 37/37 [00:05&lt;00:00,  4.67ba/s]"
      }
     },
     "8d5e23c9af0b468195b8cd60f7f70cd7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "8d82c454041c40169b63b5745537bcf3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b4299bcee1be48498016503b70f79720",
       "placeholder": "​",
       "style": "IPY_MODEL_35745679468d453097e87a74bc0916ee",
       "value": "Downloading: 100%"
      }
     },
     "8e3838a4b0e54c609642c88c938e1637": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "90dc86f9a77b46958015996a98d00e33": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "91a073c74ffa49258c43d64f718afcf7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "926085d7e5a746bbafdafe689a7d94da": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "9895557627e24a5abe1a1ac9f2eb370d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_2f1bdd816af64bd0a6bc136b7fcd61aa",
       "max": 578,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_ed1a34a42c3e437795773e5ad5d9c9f6",
       "value": 578
      }
     },
     "9d83bf944ac94d588f972b709dfa269d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9fb6f54e469e42849b7c7c1ed5e098a2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a50a097377bf4221b9995983a55cf12f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_3a2cf86b2fac4c9996c0b49e1877c0e2",
       "max": 37,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_f424ae271c0a49debcf10983ad51811f",
       "value": 37
      }
     },
     "a839a234dbeb464ebc518be49e9ffec8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "abd53aae6ed34630a992bf711bd74e32": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_f5d2b5fca38e489e80cc96c24b65fb6c",
       "placeholder": "​",
       "style": "IPY_MODEL_e444e7a940f04bf4832981ed2e7fd239",
       "value": " 37/37 [00:05&lt;00:00,  7.08ba/s]"
      }
     },
     "b2cc45f29e7b40ca81bc6c01d6702b3e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b4299bcee1be48498016503b70f79720": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b6fcfb8fb7a3450485f64f580039b6e5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b8c2b36f8e744a199fa6fb1e4ba8fdb3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "babf6ff76d274cf9938a3da46b444f96": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_cf7415fabf5c41ffaaa7bf8bd6641124",
       "placeholder": "​",
       "style": "IPY_MODEL_8c72a81d99cf4194a8ac0ca825e66f42",
       "value": "100%"
      }
     },
     "bc6d92e49648408c81813fbeee1e6801": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bcfb4f3e5adb4570a8dcf47f5b30ac6d": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bdc371752d474e5d82402e7a63d76aa8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "c147e1e4451d45f6bba56c16dec9ac3e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "c6197f1a4f3c479580e0dc4fbe18d5da": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "cab7d1e3479748b285f90645beae28ed": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_f9a83cccde254fbbafa6421847a6d95f",
        "IPY_MODEL_a50a097377bf4221b9995983a55cf12f",
        "IPY_MODEL_abd53aae6ed34630a992bf711bd74e32"
       ],
       "layout": "IPY_MODEL_1bd68bb6190340278bade7a25cea1783"
      }
     },
     "cf7415fabf5c41ffaaa7bf8bd6641124": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d2850a8aec3f4bf89c2172c84224d9bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "d609f7e1a71847bd8bdeab7ca521ca1d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "d63f3cb652db424c9dab6d0021b53a46": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "d75db2d90ad54aa58cba80c81f35278d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "e26a2e3ae5694d4baeffebd6a5796489": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e444e7a940f04bf4832981ed2e7fd239": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "e4a8c7e62f2f4c57a33071808ffe7522": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b6fcfb8fb7a3450485f64f580039b6e5",
       "placeholder": "​",
       "style": "IPY_MODEL_7bbbcd210d1546869cad760af83e4829",
       "value": "Downloading: 100%"
      }
     },
     "e9b2d5a85a2b4c409aaab1b4a4f0f895": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_6bb4e6d0610b4955a09a74c5a1483b98",
       "placeholder": "​",
       "style": "IPY_MODEL_f79bd979fd8a4dff8edeee7a26ec648d",
       "value": "100%"
      }
     },
     "ed1a34a42c3e437795773e5ad5d9c9f6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "f27e4dda852d401b9d4f6ef2a09078c1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9d83bf944ac94d588f972b709dfa269d",
       "placeholder": "​",
       "style": "IPY_MODEL_578fb6544f0045a6aa14160acd74a546",
       "value": " 2.35M/2.35M [00:00&lt;00:00, 4.07MB/s]"
      }
     },
     "f27ee53d013e4650830b4e50e2f510e4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bc6d92e49648408c81813fbeee1e6801",
       "max": 2464616,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_926085d7e5a746bbafdafe689a7d94da",
       "value": 2464616
      }
     },
     "f3854b5dae1c49fdb2b4681e9d318883": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_8d82c454041c40169b63b5745537bcf3",
        "IPY_MODEL_21579d3a660c4aedb6b281b0428632be",
        "IPY_MODEL_6ca4d325d75f4d3785e17017eb2796eb"
       ],
       "layout": "IPY_MODEL_e26a2e3ae5694d4baeffebd6a5796489"
      }
     },
     "f424ae271c0a49debcf10983ad51811f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "f5d2b5fca38e489e80cc96c24b65fb6c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f5fb5dff95e84a18909bcc7df17f0d43": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f79bd979fd8a4dff8edeee7a26ec648d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "f9a83cccde254fbbafa6421847a6d95f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b8c2b36f8e744a199fa6fb1e4ba8fdb3",
       "placeholder": "​",
       "style": "IPY_MODEL_d609f7e1a71847bd8bdeab7ca521ca1d",
       "value": "100%"
      }
     },
     "fa877bca33d841d693a5dbfb9a0b5e59": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "fa969b8dcc5248ddb411ec64f24f680e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_06b4ea22503946fc90340347b3fa5907",
       "max": 37,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_8a1ba1721f6a47a5b9773ad1738b0dfb",
       "value": 37
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}