deep-learning-with-python-notebooks/6.1-one-hot-encoding-of-words-or-characters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文代码作者：François Chollet\n",
    "\n",
    "github：https://github.com/fchollet/deep-learning-with-python-notebooks\n",
    "\n",
    "中文注释制作：黄海广\n",
    "\n",
    "github：https://github.com/fengdu78\n",
    "\n",
    "代码全部测试通过。\n",
    "\n",
    "配置环境：keras 2.2.1（原文是2.0.8，运行结果一致），tensorflow 1.8，python 3.6，\n",
    "\n",
    "主机：显卡：一块1080ti；内存：32g（注：绝大部分代码不需要GPU）\n",
    "![公众号](data/gongzhong.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-hot encoding of words or characters\n",
    "# 单词和字符的 one-hot 编码\n",
    "\n",
    "This notebook contains the first code sample found in Chapter 6, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "One-hot encoding is the most common, most basic way to turn a token into a vector. You already saw it in action in our initial IMDB and \n",
    "Reuters examples from chapter 3 (done with words, in our case). It consists in associating a unique integer index to every word, then \n",
    "turning this integer index i into a binary vector of size N, the size of the vocabulary, that would be all-zeros except for the i-th \n",
    "entry, which would be 1.\n",
    "\n",
    "Of course, one-hot encoding can be done at the character level as well. To unambiguously drive home what one-hot encoding is and how to \n",
    "implement it, here are two toy examples of one-hot encoding: one for words, the other for characters.\n",
    "\n",
    "one-hot 编码是将标记转换为向量的最常用、最基本的方法。在第 3 章的 IMDB 和路透社两 个例子中，你已经用过这种方法（都是处理单词）。它将每个单词与一个唯一的整数索引相关联， 然后将这个整数索引 i 转换为长度为 N 的二进制向量（N 是词表大小），这个向量只有第 i 个元 素是 1，其余元素都为 0。\n",
    "\n",
    "当然，也可以进行字符级的 one-hot 编码。为了让你完全理解什么是 one-hot 编码以及如何 实现 one-hot 编码，代码清单 6-1 和代码清单 6-2 给出了两个简单示例，一个是单词级的 one-hot 编码，另一个是字符级的 one-hot 编码。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word level one-hot encoding (toy example):\n",
    "\n",
    "单词级的 one-hot 编码（简单示例）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# This is our initial data; one entry per \"sample\"\n",
    "# (in this toy example, a \"sample\" is just a sentence, but\n",
    "# it could be an entire document).\n",
    "# 初始数据：每个样本是列表的一个元素（本例中的样本是一个句子，但也可以是一整篇文档）\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# First, build an index of all tokens in the data.\n",
    "# 构建数据中所有标记的索引\n",
    "\n",
    "token_index = {}\n",
    "for sample in samples:\n",
    "    # We simply tokenize the samples via the `split` method.\n",
    "    # in real life, we would also strip punctuation and special characters\n",
    "    # from the samples.\n",
    "  # 利用 split 方法对样本进行分词。在实际应用中，还需要从样本中去掉标点和特殊字符\n",
    "\n",
    "    for word in sample.split():\n",
    "        if word not in token_index:\n",
    "            # Assign a unique index to each unique word\n",
    "            #为每个唯一单词指定一个唯一索引。\n",
    "            token_index[word] = len(token_index) + 1\n",
    "            # Note that we don't attribute index 0 to anything.\n",
    "            #注意，没有为索引编号 0 指定单词\n",
    "# Next, we vectorize our samples.\n",
    "# We will only consider the first `max_length` words in each sample.\n",
    "# 对样本进行分词。只考虑每个 样本前 max_length 个单词\n",
    "max_length = 10\n",
    "\n",
    "# This is where we store our results:（将结果保存在 results 中）\n",
    "results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index = token_index.get(word)\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Character level one-hot encoding (toy example)\n",
    "\n",
    "字符级的 one-hot 编码（简单示例）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "characters = string.printable  # All printable ASCII characters.（所有可打印的 ASCII 字符）\n",
    "token_index = dict(zip(characters, range(1, len(characters) + 1)))\n",
    "\n",
    "max_length = 50\n",
    "results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, character in enumerate(sample[:max_length]):\n",
    "        index = token_index.get(character)\n",
    "        results[i, j, index] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that Keras has built-in utilities for doing one-hot encoding text at the word level or character level, starting from raw text data. \n",
    "This is what you should actually be using, as it will take care of a number of important features, such as stripping special characters \n",
    "from strings, or only taking into the top N most common words in your dataset (a common restriction to avoid dealing with very large input \n",
    "vector spaces).\n",
    "\n",
    "注意，Keras 的内置函数可以对原始文本数据进行单词级或字符级的 one-hot 编码。你应该使用这些函数，因为它们实现了许多重要的特性，比如从字符串中去除特殊字符、只考虑数据集中前 N 个最常见的单词（这是一种常用的限制，以避免处理非常大的输入向量空间）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Keras for word-level one-hot encoding:\n",
    "\n",
    "用 Keras 实现单词级的 one-hot 编码：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 9 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# We create a tokenizer, configured to only take\n",
    "# into account the top-1000 most common words\n",
    "# 创建一个分词器（tokenizer），设置为只考虑前 1000 个最常见的单词\n",
    "tokenizer = Tokenizer(num_words=1000)\n",
    "# This builds the word index（构建单词索引）\n",
    "tokenizer.fit_on_texts(samples)\n",
    "\n",
    "# This turns strings into lists of integer indices.\n",
    "# 将字符串转换为整数索引组成的列表\n",
    "sequences = tokenizer.texts_to_sequences(samples)\n",
    "\n",
    "# You could also directly get the one-hot binary representations.\n",
    "# Note that other vectorization modes than one-hot encoding are supported!\n",
    "# （也可以直接得到 one-hot 二进制表示。）这个分词器也支持除 one-hot 编码外的其他向量化模式\n",
    "\n",
    "one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')\n",
    "\n",
    "# This is how you can recover the word index that was computed（找回单词索引）\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "A variant of one-hot encoding is the so-called \"one-hot hashing trick\", which can be used when the number of unique tokens in your \n",
    "vocabulary is too large to handle explicitly. Instead of explicitly assigning an index to each word and keeping a reference of these \n",
    "indices in a dictionary, one may hash words into vectors of fixed size. This is typically done with a very lightweight hashing function. \n",
    "The main advantage of this method is that it does away with maintaining an explicit word index, which \n",
    "saves memory and allows online encoding of the data (starting to generate token vectors right away, before having seen all of the available \n",
    "data). The one drawback of this method is that it is susceptible to \"hash collisions\": two different words may end up with the same hash, \n",
    "and subsequently any machine learning model looking at these hashes won't be able to tell the difference between these words. The likelihood \n",
    "of hash collisions decreases when the dimensionality of the hashing space is much larger than the total number of unique tokens being hashed.\n",
    "\n",
    "one-hot 编码的一种变体是所谓的 one-hot 散列技巧（one-hot hashing trick），如果词表中唯 一标记的数量太大而无法直接处理，就可以使用这种技巧。这种方法没有为每个单词显式分配 一个索引并将这些索引保存在一个字典中，而是将单词散列编码为固定长度的向量，通常用一个非常简单的散列函数来实现。这种方法的主要优点在于，它避免了维护一个显式的单词索引， 从而节省内存并允许数据的在线编码（在读取完所有数据之前，你就可以立刻生成标记向量）。 这种方法有一个缺点，就是可能会出现散列冲突（hash collision），即两个不同的单词可能具有相同的散列值，随后任何机器学习模型观察这些散列值，都无法区分它们所对应的单词。如果散列空间的维度远大于需要散列的唯一标记的个数，散列冲突的可能性会减小。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word-level one-hot encoding with hashing trick (toy example):\n",
    "\n",
    "使用散列技巧的单词级的 one-hot 编码（简单示例）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
    "\n",
    "# We will store our words as vectors of size 1000.\n",
    "# Note that if you have close to 1000 words (or more)\n",
    "# you will start seeing many hash collisions, which\n",
    "# will decrease the accuracy of this encoding method.\n",
    "# 将单词保存为长度为 1000 的向量。如果单词数量接近 1000 个（或更多），\n",
    "# 那么会遇到很多散列冲突，这会降低这种编码方法的准确性\n",
    "dimensionality = 1000\n",
    "max_length = 10\n",
    "\n",
    "results = np.zeros((len(samples), max_length, dimensionality))\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        # Hash the word into a \"random\" integer index\n",
    "        # that is between 0 and 1000\n",
    "        #将单词散列为 0~1000 范围内的一个随机整数索引\n",
    "\n",
    "        index = abs(hash(word)) % dimensionality\n",
    "        results[i, j, index] = 1."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}