keras教程-n-10分钟入门LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "title: keras教程-n-10分钟入门LSTM\n",
    "date: 2018-09-02 20:17:55\n",
    "tags: [keras教程]\n",
    "toc: true\n",
    "xiongzhang: false\n",
    "\n",
    "---\n",
    "<span></span>\n",
    "<!-- more -->\n",
    "\n",
    "\n",
    "本文代码运行环境:\n",
    "\n",
    "- windows10\n",
    "- python3.6\n",
    "- jupyter notebook\n",
    "- tensorflow 1.x\n",
    "- keras 2.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用过keras的人可能都遇到过这个问题: 怎么用keras来实现一个序列到序列的LSTM网络, 因为这个网络相对于简单的多层感知机要复杂很多。今天我们就用10分钟来实现一个lstm神经网络。前提是你对这个网络结构已经有一些了解。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 什么是seq2seq模型(序列到序列的模型)\n",
    "\n",
    "序列到序列的模型指的是模型的输入和输出都是一个序列, 比如机器翻译, 输入是中文的句子, 输出是英文的句子。这样的模型还可以被用于对话机器人或者一些文字生成的场景。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 简单模型:输入的序列和输出序列长度都相同\n",
    "\n",
    "当输入序列和输出序列具有相同的长度时，您可以使用Keras LSTM或GRU层简单地实现这些模型。这个示例脚本就是这种情况，它显示了如何教RNN学习数字加法：\n",
    "\n",
    "<img src=\"images/add-nums.png\" />\n",
    "\n",
    "这种方法的一个警告是，它假设有了输入就能产生输出。这在某些情况下有效（数字字符串加法），但对大多数用例不起作用。在一般情况下，关于整个输入序列的信息是必要的，以便开始生成目标序列。\n",
    "\n",
    "#### 更通用的模型:输入和输出长度不同\n",
    "\n",
    "在一般情况下，输入序列和输出序列具有不同的长度（例如机器翻译），并且需要整个输入序列以便开始预测目标。这需要更高级的设置，这是人们在提及没有进一步上下文的“序列到序列模型”时通常所指的。以下是它的工作原理：\n",
    "\n",
    "- RNN层充当“编码器”：它处理输入序列并返回其自己的内部状态。注意，我们丢弃编码器RNN的输出，仅使用状态信息。该状态将用作下一步骤中解码器的“上下文”。\n",
    "- 另一个RNN层（或其堆栈）充当“解码器”：在给定目标序列的先前字符的情况下，训练它以预测目标序列的下一个字符。具体而言，训练模型将目标序列转换为相同的序列但偏移一个时间步骤，在此上下文中称为“教师强制”的训练过程。重要的是，解码器使用来自编码器的状态向量作为初始状态，这是解码器获得应该生成什么的信息。\n",
    "\n",
    "<img src=\"images/seq2seq.png\" />\n",
    "\n",
    "\n",
    "在预测阶段, 也就是在解码未知输入序列时, 我们经过的步骤有所不同:\n",
    "\n",
    "1）将输入序列编码为状态向量。\n",
    "2）从大小为1的目标序列开始（只是序列开始字符）。\n",
    "3）将状态向量和1-char目标序列馈送到解码器以产生对下一个字符的预测。\n",
    "4）使用这些预测对下一个字符进行采样（我们只使用argmax）。\n",
    "5）将采样的字符附加到目标序列\n",
    "6）重复，直到我们生成序列结束字符或我们达到序列最长限制。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### keras案例\n",
    "\n",
    "我们用代码来加强理解。\n",
    "\n",
    "对于我们的示例，我们将使用成对的英语句子及其中文翻译的数据集，您可以从[网站(mlln.cn)下载](http://mlln.cn/2018/08/26/机器翻译语料库大全(免费下)) 。要下载的文件名为 cmn-eng.zip。我们将实现一个字符级的序列到序列模型，逐个字符地处理输入并逐个字符地生成输出。另一种选择是单词级模型，这对于机器翻译而言更为常见。在本文的最后，您将找到一些关于使用嵌入层将模型转换为单词级模型的注释。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 总的流程\n",
    "\n",
    "- 1）将句子转换为3个Numpy数组，encoder_input_data，decoder_input_data，decoder_target_data：\n",
    "    - encoder_input_data是包含英语句子的one-hot向量的三维数组, 形状为（num_pairs，max_english_sentence_length，num_english_characters）。\n",
    "    - decoder_input_data是一个形状为（num_pairs，max_chi_sentence_length，num_chi_characters）的3D数组，包含中文句子的one-hot向量。\n",
    "    - decoder_target_data与decoder_input_data相同，但偏移一个timestep。 decoder_target_data [：，t，：]与decoder_input_data [：，t + 1，：]相同。\n",
    "    \n",
    "- 2）训练基本的基于LSTM的Seq2Seq模型以预测给出encoder_input_data和decoder_input_data的decoder_target_data。我们的模型使用\"teacher forcing\"。\n",
    "\n",
    "- 3）翻译一些句子以检查模型是否正常工作（将来自encoder_input_data的样本从decoder_target_data转换成相应的样本）。\n",
    "\n",
    "由于训练过程和推理过程（解码句子）完全不同，我们对两者使用不同的模型，尽管它们都利用相同的内层。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是我们的训练模型, 它利用了keras的三个关键参数:\n",
    "\n",
    "- return_state参数，配置RNN层以返回列表，其中第一个是输出，下一个是内部RNN状态。这用于恢复编码器的状态。\n",
    "- inital_state参数，指定RNN的初始状态。这用于将编码器状态作为初始状态传递给解码器。\n",
    "- return_sequences参数，配置RNN以返回其完整的输出序列（而不仅仅是最后一个输出，默认行为）。这用于解码器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, LSTM, Dense\n",
    "\n",
    "# 定义输入序列\n",
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "# 构建LSTM单元\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "# 获取输出和状态h和状态c\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_inputs)\n",
    "# 我们抛弃 `encoder_outputs` 只保留states.\n",
    "encoder_states = [state_h, state_c]\n",
    "\n",
    "# 配置解码器`的初始状态为encoder_states` \n",
    "decoder_inputs = Input(shape=(None, num_decoder_tokens))\n",
    "# 我们让编码器返回全部的output 序列,\n",
    "# 并且返回内部状态, 在训练阶段我们不使用内部状态信息\n",
    "# 但是在推断阶段会用到\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(decoder_inputs,\n",
    "                                     initial_state=encoder_states)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "\n",
    "# Define the model that will turn\n",
    "# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用两行代码来实现训练模型，同时监控20％样本中的loss。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练大概一小时以后, 我们可以做推断了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1）对输入句子进行编码并获取初始解码器状态\n",
    "- 2）以该初始状态和“序列开始”标记作为目标运行解码器的一步。输出将是下一个目标字符。\n",
    "- 3）附加预测的目标字符并重复。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "decoder_outputs, state_h, state_c = decoder_lstm(\n",
    "    decoder_inputs, initial_state=decoder_states_inputs)\n",
    "decoder_states = [state_h, state_c]\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + decoder_states_inputs,\n",
    "    [decoder_outputs] + decoder_states)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们用它来实现一个生成句子的循环:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "\n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "    # Populate the first character of target sequence with the start character.\n",
    "    target_seq[0, 0, target_token_index['\\t']] = 1.\n",
    "\n",
    "    # Sampling loop for a batch of sequences\n",
    "    # (to simplify, here we assume a batch of size 1).\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "        output_tokens, h, c = decoder_model.predict(\n",
    "            [target_seq] + states_value)\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        decoded_sentence += sampled_char\n",
    "\n",
    "        # Exit condition: either hit max length\n",
    "        # or find stop character.\n",
    "        if (sampled_char == '\\n' or\n",
    "           len(decoded_sentence) > max_decoder_seq_length):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "        target_seq[0, 0, sampled_token_index] = 1.\n",
    "\n",
    "        # Update states\n",
    "        states_value = [h, c]\n",
    "\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 结果\n",
    "\n",
    "我们的结果似乎出奇的好, 但是这并不让我吃惊, 因为我们的测试数据就是训练数据!\n",
    "\n",
    "```\n",
    "Input sentence: Be nice.\n",
    "Decoded sentence: Soyez gentil !\n",
    "-\n",
    "Input sentence: Drop it!\n",
    "Decoded sentence: Laissez tomber !\n",
    "-\n",
    "Input sentence: Get out!\n",
    "Decoded sentence: Sortez !\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 彩蛋\n",
    "\n",
    "#### 如果我想用GRU层而不是LSTM怎么办?\n",
    "\n",
    "使用GRU更简单, 因为LSTM有两个状态向量, 而GRU只有一个。\n",
    "下面是修改的代码:\n",
    "\n",
    "```python\n",
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "encoder = GRU(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h = encoder(encoder_inputs)\n",
    "\n",
    "decoder_inputs = Input(shape=(None, num_decoder_tokens))\n",
    "decoder_gru = GRU(latent_dim, return_sequences=True)\n",
    "decoder_outputs = decoder_gru(decoder_inputs, initial_state=state_h)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 如果用词而不是字母作为输入单元, 用数字代表这些词, 该怎么做\n",
    "\n",
    "如果你的输入时数字序列(用数字代表词)你可以使用keras的Embedding 层, 下面是具体做法:\n",
    "\n",
    "```python\n",
    "# Define an input sequence and process it.\n",
    "encoder_inputs = Input(shape=(None,))\n",
    "x = Embedding(num_encoder_tokens, latent_dim)(encoder_inputs)\n",
    "x, state_h, state_c = LSTM(latent_dim,\n",
    "                           return_state=True)(x)\n",
    "encoder_states = [state_h, state_c]\n",
    "\n",
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "x = Embedding(num_decoder_tokens, latent_dim)(decoder_inputs)\n",
    "x = LSTM(latent_dim, return_sequences=True)(x, initial_state=encoder_states)\n",
    "decoder_outputs = Dense(num_decoder_tokens, activation='softmax')(x)\n",
    "\n",
    "# Define the model that will turn\n",
    "# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "\n",
    "# Compile & run training\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
    "# Note that `decoder_target_data` needs to be one-hot encoded,\n",
    "# rather than sequences of integers like `decoder_input_data`!\n",
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_split=0.2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 如果我不想使用'teacher forcing'来训练\n",
    "\n",
    "在某些特殊情况下，可能无法使用'teacher forcing'，因为你无法访问完整的目标序列，例如：如果你正在对很长的序列进行在线培训，那么缓冲完整的输入目标对是不可能的。在这种情况下，你可能希望通过将解码器的输出重新注入解码器的输入来进行训练，就像我们在推理阶段做的那样。\n",
    "\n",
    "你可以通过构建硬编码输出重新注入循环的模型来实现此目的：\n",
    "\n",
    "```python\n",
    "from keras.layers import Lambda\n",
    "from keras import backend as K\n",
    "\n",
    "# The first part is unchanged\n",
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_inputs)\n",
    "states = [state_h, state_c]\n",
    "\n",
    "# Set up the decoder, which will only process one timestep at a time.\n",
    "decoder_inputs = Input(shape=(1, num_decoder_tokens))\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "\n",
    "all_outputs = []\n",
    "inputs = decoder_inputs\n",
    "for _ in range(max_decoder_seq_length):\n",
    "    # Run the decoder on one timestep\n",
    "    outputs, state_h, state_c = decoder_lstm(inputs,\n",
    "                                             initial_state=states)\n",
    "    outputs = decoder_dense(outputs)\n",
    "    # Store the current prediction (we will concatenate all predictions later)\n",
    "    all_outputs.append(outputs)\n",
    "    # Reinject the outputs as inputs for the next loop iteration\n",
    "    # as well as update the states\n",
    "    inputs = outputs\n",
    "    states = [state_h, state_c]\n",
    "\n",
    "# Concatenate all predictions\n",
    "decoder_outputs = Lambda(lambda x: K.concatenate(x, axis=1))(all_outputs)\n",
    "\n",
    "# Define and compile model as previously\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
    "\n",
    "# Prepare decoder input data that just contains the start character\n",
    "# Note that we could have made it a constant hard-coded in the model\n",
    "decoder_input_data = np.zeros((num_samples, 1, num_decoder_tokens))\n",
    "decoder_input_data[:, 0, target_token_index['\\t']] = 1.\n",
    "\n",
    "# Train model as previously\n",
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_split=0.2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 本教程的全部代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}