Python71-gensim做主题模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba\n",
    "from gensim import corpora,models,similarities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 语料\n",
    "这里手工设置了一下，自己可以更具需要处理成下面这种格式就行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences=['我的语文成绩在年级得了第一',\n",
    "           '吹泡泡是一件很有趣的事',\n",
    "           '猴子们欢蹦乱跳地在树林里玩耍着']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "jieba分词，这里其实可以做一些去除停用词和去除仅出现一次的词，这里语料简单不做处理，如有需求，请参考https://blog.csdn.net/questionfish/article/details/46739207"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['我', '的', '语文', '成绩', '在', '年级', '得', '了', '第一'], ['吹泡泡', '是', '一件', '很', '有趣', '的', '事'], ['猴子', '们', '欢蹦乱跳', '地', '在', '树林', '里', '玩耍', '着']]\n"
     ]
    }
   ],
   "source": [
    "words=[]\n",
    "for doc in sentences:\n",
    "    words.append(list(jieba.cut(doc)))\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构造词典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(23 unique tokens: ['了', '在', '年级', '得', '成绩']...)\n",
      "{'了': 0, '在': 1, '年级': 2, '得': 3, '成绩': 4, '我': 5, '的': 6, '第一': 7, '语文': 8, '一件': 9, '事': 10, '吹泡泡': 11, '很': 12, '是': 13, '有趣': 14, '们': 15, '地': 16, '树林': 17, '欢蹦乱跳': 18, '猴子': 19, '玩耍': 20, '着': 21, '里': 22}\n"
     ]
    }
   ],
   "source": [
    "dictionary=corpora.Dictionary(words)\n",
    "print(dictionary) # 一个有23个不同的词\n",
    "print(dictionary.token2id) # 每个词对应的编号"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了方便查看，我们输出下每个词对应的编号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "了--编号为:0\n",
      "在--编号为:1\n",
      "年级--编号为:2\n",
      "得--编号为:3\n",
      "成绩--编号为:4\n",
      "我--编号为:5\n",
      "的--编号为:6\n",
      "第一--编号为:7\n",
      "语文--编号为:8\n",
      "一件--编号为:9\n",
      "事--编号为:10\n",
      "吹泡泡--编号为:11\n",
      "很--编号为:12\n",
      "是--编号为:13\n",
      "有趣--编号为:14\n",
      "们--编号为:15\n",
      "地--编号为:16\n",
      "树林--编号为:17\n",
      "欢蹦乱跳--编号为:18\n",
      "猴子--编号为:19\n",
      "玩耍--编号为:20\n",
      "着--编号为:21\n",
      "里--编号为:22\n"
     ]
    }
   ],
   "source": [
    "for word,index in dictionary.token2id.items():\n",
    "    print(\"{}--编号为:{}\".format(word,index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 生成语料库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1)], [(6, 1), (9, 1), (10, 1), (11, 1), (12, 1), (13, 1), (14, 1)], [(1, 1), (15, 1), (16, 1), (17, 1), (18, 1), (19, 1), (20, 1), (21, 1), (22, 1)]]\n"
     ]
    }
   ],
   "source": [
    "corpus=[dictionary.doc2bow(text) for text in words]\n",
    "print(corpus) # 向量空间模型格式的语料库 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF-IDF变换\n",
    "将用词频向量表示一句话 变换成为用 词的重要性向量表示一句话\n",
    "\n",
    "（TF-IDF变换：评估一字词对于一个文件集或一个语料库中的其中一份文件的重要程度。字词的重要性随着它在文件中出现的次数成正比增加，但同时会随着它在语料库中出现的频率成反比下降。）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.7071067811865475), (4, 0.7071067811865475)]\n"
     ]
    }
   ],
   "source": [
    "tf_idf=models.TfidfModel(corpus)\n",
    "vec=[(0,1),(4,1)]  # 计算 “了”，“成绩”的重要性\n",
    "print(tf_idf[vec])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.37081761280562664), (1, 0.1368577477382165), (2, 0.37081761280562664), (3, 0.37081761280562664), (4, 0.37081761280562664), (5, 0.37081761280562664), (6, 0.1368577477382165), (7, 0.37081761280562664), (8, 0.37081761280562664)]\n",
      "[(6, 0.1489905855640844), (9, 0.40369167389095173), (10, 0.40369167389095173), (11, 0.40369167389095173), (12, 0.40369167389095173), (13, 0.40369167389095173), (14, 0.40369167389095173)]\n",
      "[(1, 0.1293891572290254), (15, 0.35058138249049303), (16, 0.35058138249049303), (17, 0.35058138249049303), (18, 0.35058138249049303), (19, 0.35058138249049303), (20, 0.35058138249049303), (21, 0.35058138249049303), (22, 0.35058138249049303)]\n"
     ]
    }
   ],
   "source": [
    "corpus_tfidf=tf_idf[corpus]\n",
    "for doc in corpus_tfidf:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 比较句子之间的相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.52441525), (1, 0.0), (2, 0.0)]\n"
     ]
    }
   ],
   "source": [
    "index=similarities.SparseMatrixSimilarity(tf_idf[corpus],num_features=23)\n",
    "sims=index[tf_idf[vec]]\n",
    "print(list(enumerate(sims)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里表示 “了 成绩”与第一句的相似度为52.44%，与其他两句话的相似度为0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSI变换"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, '0.259*\"成绩\" + 0.259*\"语文\" + 0.259*\"我\" + 0.259*\"了\" + 0.259*\"第一\" + 0.259*\"得\" + 0.259*\"年级\" + 0.213*\"是\" + 0.213*\"有趣\" + 0.213*\"事\"')\n",
      "(1, '0.265*\"欢蹦乱跳\" + 0.265*\"着\" + 0.265*\"玩耍\" + 0.265*\"猴子\" + 0.265*\"里\" + 0.265*\"树林\" + 0.265*\"地\" + 0.265*\"们\" + -0.265*\"很\" + -0.265*\"有趣\"')\n"
     ]
    }
   ],
   "source": [
    "lsi=models.LsiModel(corpus_tfidf,id2word=dictionary,num_topics=2)\n",
    "lsi_out=lsi.print_topics(2) # 基于SVD建立的两个主题模型内容\n",
    "print(lsi_out[0])\n",
    "print(lsi_out[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.7165913599306)]\n",
      "[(0, 0.5410455418833824), (1, -0.6556941552835392)]\n",
      "[(0, 0.469864766433178), (1, 0.7550266053107043)]\n"
     ]
    }
   ],
   "source": [
    "corpus_lsi = lsi[corpus_tfidf]\n",
    "for doc in corpus_lsi:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "说明 第一句话和第二句话和主题1相似，第三句话和主题2相似"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LDA变换"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, '0.052*\"里\" + 0.052*\"猴子\" + 0.052*\"地\" + 0.051*\"欢蹦乱跳\" + 0.051*\"玩耍\" + 0.051*\"着\" + 0.050*\"树林\" + 0.050*\"们\" + 0.044*\"一件\" + 0.044*\"有趣\"')\n",
      "(1, '0.050*\"了\" + 0.050*\"我\" + 0.050*\"成绩\" + 0.050*\"很\" + 0.050*\"是\" + 0.050*\"语文\" + 0.049*\"第一\" + 0.049*\"年级\" + 0.049*\"得\" + 0.047*\"吹泡泡\"')\n",
      "[(0, 0.16434348), (1, 0.8356565)]\n",
      "[(0, 0.21110593), (1, 0.78889406)]\n",
      "[(0, 0.84332335), (1, 0.15667662)]\n"
     ]
    }
   ],
   "source": [
    "lda = models.LdaModel(corpus_tfidf, id2word=dictionary, num_topics=2)\n",
    "lda_out=lda.print_topics(2)\n",
    "print(lda_out[0]) \n",
    "print(lda_out[1]) \n",
    "corpus_lda = lda[corpus_tfidf]\n",
    "for doc in corpus_lda:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一二句和主题二相似，第三句和主题一相似"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 判断一句话属于哪个主题模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[(19, 1)]\n",
      "[(0, 0.16039417742386808), (1, 0.26469827110693045)]\n"
     ]
    }
   ],
   "source": [
    "index = similarities.MatrixSimilarity(lsi[corpus])\n",
    "query = \"猴子\"\n",
    "print()\n",
    "query_bow = dictionary.doc2bow(list(jieba.cut(query)))\n",
    "print(query_bow) \n",
    "query_lsi = lsi[query_bow]\n",
    "print(query_lsi) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与第二个主题更相似"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 比较和第几句话相似\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0.5182331), (1, -0.31047052), (2, 0.99952173)]\n",
      "[(2, 0.99952173), (0, 0.5182331), (1, -0.31047052)]\n"
     ]
    }
   ],
   "source": [
    "sims = index[query_lsi]\n",
    "print(list(enumerate(sims))) \n",
    "sort_sims = sorted(enumerate(sims), key=lambda item: -item[1])\n",
    "print(sort_sims)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "说明“猴子”和第三句话相似度很高"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文：https://blog.csdn.net/whzhcahzxh/article/details/17528261"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}