UrIII_name-authority.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Notebook attempts to isolate names (personal names, place names, royal names, divine names) in a collection of texts downloaded from [BDTNS](http://bdtns.filol.csic.es/). The names are formatted to correspond to current [ORACC](http://oracc.org) conventions. The resulting list (name instances with normalized names) will need hand editing.\n",
    "\n",
    "The original Notebook took all names from the entire [BDTNS](http://bdtns.filol.csic.es/) database. The current version selects names from texts that are cataloged (in [BDTNS](http://bdtns.filol.csic.es/)) as coming from Puzriš-Dagan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create list of Puzriš-Dagan texts only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('data/query_cat_17_09_6-194802.txt') as f:\n",
    "#    puzd_df = pd.read_csv(f, sep=\"\\t\", header=None, names = [\"BDTNS\", \"CDLI\"])\n",
    "#puzd_df\n",
    "#puzd_l = list(puzd_df[\"BDTNS\"])\n",
    "#puzd_l = [str(no).zfill(6) for no in puzd_l]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/query_text_19_05_15-020951.txt', mode = 'r', encoding = 'utf8') as f:\n",
    "    lines = f.read().splitlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `select_puzd()` sets a flag to distinguish between texts from Puzriš-Dagan and other texts. If the 6-digit code at the beginning of the header (the [BDTNS](http://bdtns.filol.csic.es/) text number) corresponds to a number in the list `puzd_l` the flag is set to `True`, otherwise to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#flag = False\n",
    "#puzd_lines = []\n",
    "#for line in linelist:\n",
    "#    if line[:6].isdigit():\n",
    "#        if line[:line.find(\"\\t\")] in puzd_l:\n",
    "#            flag = True\n",
    "#        else:\n",
    "#            flag = False\n",
    "#    if flag:\n",
    "#        puzd_lines.append(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#len(puzd_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#linelist = puzd_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean\n",
    "Now remove all flags etc., including brackets and half brackets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean(data, oldlist, new=\"\"):\n",
    "    for i in range(len(oldlist)):\n",
    "        data = data.replace(oldlist[i],new)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "remlist = [\"`\", \"´\", \"/\", \"]\", \"[\", \"!\", \"?\", \"<\", \">\", \"(\", \")\", \"⸢\", \"⸣\", \"⌈\", \"⌉\"]\n",
    "lines = [clean(line, remlist) for line in lines]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove Editorial Lines and Headers\n",
    "Eliminate empty lines, header lines, and editorial remarks. Header lines begin with a digit. Editorial remarks begin with a line indicator (like data lines), followed by one or more TABs, followed by #. It seems that in the current version of the [BDTNS] output TABs are not used this way, but # still marks the beginning of an editorial comment. Everything after # is discarded. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = [line for line in lines if len(line) > 0]\n",
    "lines = [line for line in lines if not line[0].isdigit()]\n",
    "lines = [line for line in lines if not \"\\t#\" in line]\n",
    "lines = [line for line in lines if not \"=====\" in line]\n",
    "lines = [line.split(\"#\")[0] for line in lines]  # remove everything after #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resolve x-values\n",
    "First, use a regular expression to replace capital X by subscript ₓ. This will skip instances like KA×X, because × (the \"times\" character) is not in the list of possible preceding characters.\n",
    "\n",
    "Second, use a regular expression to resolve instances such as \n",
    "> mu-kuₓ lugal-la (=DU)\n",
    "\n",
    "and change those into\n",
    "> mu-kuₓ(DU) lugal-la\n",
    "\n",
    "Round brackets, \"(\"and \")\" have been removed in a preceding cell - they are put back for such ₓ-value specifiers. The regex in the second line uses a lookahead to check that the expression is followed either by the end of the string ($) or by a white space (\\s). Running regex replacements on tens of thousands of lines takes a little while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = [re.sub('([abdegŋhḫijklmnpqrstuwzšṣṭABDEGŊHḪIJKLMNPQRSTUWZŠṢṬ])(X)', '\\\\1ₓ', line) for line in lines]\n",
    "lines = [re.sub('([ₓ])(.*?)(=)(.*?)(?:\\s|$)', '\\\\1(\\\\4)\\\\2', line) for line in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = [re.sub('(ₓ\\()(.*?)([\\.×])(.*?)(\\))', '\\\\1|\\\\2\\\\3\\\\4|\\\\5', line) for line in lines]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect results. Note that this code fails for a line like\n",
    "\n",
    "> 18 gin2 nagga mu-kuX gibil (=AN.NA) (=DU)\n",
    "\n",
    "which will yield\n",
    "\n",
    "> 18 gin2 nagga mu-kuₓ(|AN.NA|) gibil =DU\n",
    "\n",
    "The notation (=AN.NA) in this case refers to nagga, whereas (=DU) refers to kuX - but there is no way the script can detect that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>o. 2     gi ziₓ(SIG7)-a 12 sar-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>o. 3     4 guruš gi ziₓ(SIG7)-a 12 sar-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>o. 6     la2-i3 su-ga Lugal-ušurₓ(|LAL2.TUG2|) sipa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>r. 2     1 Ur-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>222</th>\n",
       "      <td>o. 4     1 Lu2-Urubₓ(|URU×KAR2|){ki}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>427</th>\n",
       "      <td>r. 5     Umma{ki}-a mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>481</th>\n",
       "      <td>o. 5     kišib Ur-nigarₓ{gar} x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>487</th>\n",
       "      <td>s. 1     Ur-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>r. 4     ša3 Tum-alₓ(TUR3){ki}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>548</th>\n",
       "      <td>r. 14     mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>567</th>\n",
       "      <td>o. 7     24 sar tug2-sagₓ(|ŠE.KIN|) x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>o. 3     mu-kuₓ(DU) {d}Inanna Zabalam4{ki} =MUŠ3.ZA.UNUG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1073</th>\n",
       "      <td>r. 31     šu-nigin2 0.0.4 munu4 al-nagₓ(GAZ)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1119</th>\n",
       "      <td>o. 7     +4.0.0 še apin-la2 a-ša3 illu Nigarₓ{gar}-ki-du10 ki Lugal-nig2-lagar-e-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1130</th>\n",
       "      <td>r. 17     Ur-nigarₓ{gar} x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1133</th>\n",
       "      <td>r. 20     2.0.0 še apin-la2 a-ša3 illu Nigarₓ{gar}-ki-du10 ki Lugal-nig2-lagar-e-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1209</th>\n",
       "      <td>r. 19     mu-kuₓ(DU) libir-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1214</th>\n",
       "      <td>r. 24     18 gin2 nagga mu-kuₓ(|AN.NA|) gibil =DU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1266</th>\n",
       "      <td>r. 24     mu-kuₓ(DU) dingir e3-a x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1268</th>\n",
       "      <td>r. 26     mu-kuₓ(DU) ša3 e2-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1277</th>\n",
       "      <td>r. 33     mu-kuₓ(DU) {d}Alim-mah</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1775</th>\n",
       "      <td>r.ii 7     mu-kuₓ(DU) {d}Inanna</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1797</th>\n",
       "      <td>r.ii 28     mu-kuₓ(DU) {d}Nin-šubur</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1819</th>\n",
       "      <td>r.iii 18     mu-kuₓ(DU) {d}Na-na-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1820</th>\n",
       "      <td>r.iii 19     mu-kuₓ(DU) {d}Inanna Zabalam3{ki} =MUŠ3.UNUG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1828</th>\n",
       "      <td>s.iii 3     dumu Ur-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1864</th>\n",
       "      <td>o.i 8     3 Geme2-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003</th>\n",
       "      <td>r. 43     mu-kuₓ(DU) {d}Šara2 Giš-gi-gal-ka</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2087</th>\n",
       "      <td>o. 2     Ur-nigarₓ{gar} engar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2134</th>\n",
       "      <td>o. 1     8 gi ziₓ(SIG7)-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137100</th>\n",
       "      <td>o. 9     ba-ri-ri mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137188</th>\n",
       "      <td>r. 14     iti buruₓ(GAN2)-maš</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137846</th>\n",
       "      <td>o. 6     mu-kuₓ(DU) lugal-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137877</th>\n",
       "      <td>r. 8     mu-kuₓ(DU) Na-sa6 i3-dab5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137983</th>\n",
       "      <td>r. 10     mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1137996</th>\n",
       "      <td>r. 1     mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138040</th>\n",
       "      <td>r. 16     mu-kuₓ(DU) kilib3-ba</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138044</th>\n",
       "      <td>r. 20     iti zahₓ(ŠEŠ)-da-gu7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138222</th>\n",
       "      <td>o. 4     mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138694</th>\n",
       "      <td>o. 2     1 sila4 Ur-nigarₓ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138699</th>\n",
       "      <td>r. 1     mu-kuₓ(DU) lugal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138785</th>\n",
       "      <td>r. 2     Ur-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138834</th>\n",
       "      <td>o. 7     mu-kuₓ(DU)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138908</th>\n",
       "      <td>r. 13'     a-ša3 {d}Amar-{d}Suen-{d}Šara2-ki-ag2 še gurₓ(|ŠE.KIN|)-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138915</th>\n",
       "      <td>o.iii 2'     giri3 Lu2-tur-tur dumu  Lugal-ušurₓ(|LAL2.TUG2|)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1138998</th>\n",
       "      <td>r.ii 19     ša3 Tum-alₓ(TUR3){ki}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139097</th>\n",
       "      <td>o.iii 26'     2 gin2 Ur-nigarₓ{gar} dumu  Puzur4-E2-a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139142</th>\n",
       "      <td>r.i 13     dam Lu2-du10-ga dumu  Nigarₓ{gar}-ki-du10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139236</th>\n",
       "      <td>s.ii 1     kurunₓ(|DIN.KAŠ|)-a-gal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139326</th>\n",
       "      <td>o.ii 11     Ur-nigarₓ{gar} dumu Ur-{d}Ig-alim</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139559</th>\n",
       "      <td>o.i 5     iti buruₓ(GAN2)-maš</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139625</th>\n",
       "      <td>r.ii 6'     kišib Ur-nigarₓ{gar} dumu Ur-{d}Ig-alim</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139629</th>\n",
       "      <td>r.ii 10'     iti buruₓ(GAN2)-maš-ta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139753</th>\n",
       "      <td>o. 16     13 0.0.3 3 Lu2-Urubₓ(|URU×KAR2|){ki}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139869</th>\n",
       "      <td>o.iv 9     dumu geme2-Urubₓ(|URU×KAR2|){ki}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139940</th>\n",
       "      <td>r.i 17     dumu geme2-Urubₓ(|URU×KAR2|){ki}-me</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139942</th>\n",
       "      <td>r.i 19     dumu Geme2-nigarₓ{gar}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1140478</th>\n",
       "      <td>r.iv 34     Ur-{d}Ša-uₓ(LU)-ša</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1141210</th>\n",
       "      <td>o.iv 14     Nigarₓ{gar}-ki-du10 engar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1141895</th>\n",
       "      <td>r. 4     iti zahₓ(ŠEŠ)-da-gu7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18683 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                           0\n",
       "87       o. 2     gi ziₓ(SIG7)-a 12 sar-ta                                                  \n",
       "98       o. 3     4 guruš gi ziₓ(SIG7)-a 12 sar-ta                                          \n",
       "115      o. 6     la2-i3 su-ga Lugal-ušurₓ(|LAL2.TUG2|) sipa                                \n",
       "195      r. 2     1 Ur-nigarₓ{gar}                                                          \n",
       "222      o. 4     1 Lu2-Urubₓ(|URU×KAR2|){ki}                                               \n",
       "427      r. 5     Umma{ki}-a mu-kuₓ(DU)                                                     \n",
       "481      o. 5     kišib Ur-nigarₓ{gar} x                                                    \n",
       "487      s. 1     Ur-nigarₓ{gar}                                                            \n",
       "506      r. 4     ša3 Tum-alₓ(TUR3){ki}                                                     \n",
       "548      r. 14     mu-kuₓ(DU)                                                               \n",
       "567      o. 7     24 sar tug2-sagₓ(|ŠE.KIN|) x                                              \n",
       "851      o. 3     mu-kuₓ(DU) {d}Inanna Zabalam4{ki} =MUŠ3.ZA.UNUG                           \n",
       "1073     r. 31     šu-nigin2 0.0.4 munu4 al-nagₓ(GAZ)                                       \n",
       "1119     o. 7     +4.0.0 še apin-la2 a-ša3 illu Nigarₓ{gar}-ki-du10 ki Lugal-nig2-lagar-e-ta\n",
       "1130     r. 17     Ur-nigarₓ{gar} x                                                         \n",
       "1133     r. 20     2.0.0 še apin-la2 a-ša3 illu Nigarₓ{gar}-ki-du10 ki Lugal-nig2-lagar-e-ta\n",
       "1209     r. 19     mu-kuₓ(DU) libir-a                                                       \n",
       "1214     r. 24     18 gin2 nagga mu-kuₓ(|AN.NA|) gibil =DU                                  \n",
       "1266     r. 24     mu-kuₓ(DU) dingir e3-a x                                                 \n",
       "1268     r. 26     mu-kuₓ(DU) ša3 e2-a                                                      \n",
       "1277     r. 33     mu-kuₓ(DU) {d}Alim-mah                                                   \n",
       "1775     r.ii 7     mu-kuₓ(DU) {d}Inanna                                                    \n",
       "1797     r.ii 28     mu-kuₓ(DU) {d}Nin-šubur                                                \n",
       "1819     r.iii 18     mu-kuₓ(DU) {d}Na-na-a                                                 \n",
       "1820     r.iii 19     mu-kuₓ(DU) {d}Inanna Zabalam3{ki} =MUŠ3.UNUG                          \n",
       "1828     s.iii 3     dumu Ur-nigarₓ{gar}                                                    \n",
       "1864     o.i 8     3 Geme2-nigarₓ{gar}                                                      \n",
       "2003     r. 43     mu-kuₓ(DU) {d}Šara2 Giš-gi-gal-ka                                        \n",
       "2087     o. 2     Ur-nigarₓ{gar} engar                                                      \n",
       "2134     o. 1     8 gi ziₓ(SIG7)-a                                                          \n",
       "...                             ...                                                         \n",
       "1137100  o. 9     ba-ri-ri mu-kuₓ(DU)                                                       \n",
       "1137188  r. 14     iti buruₓ(GAN2)-maš                                                      \n",
       "1137846  o. 6     mu-kuₓ(DU) lugal-ta                                                       \n",
       "1137877  r. 8     mu-kuₓ(DU) Na-sa6 i3-dab5                                                 \n",
       "1137983  r. 10     mu-kuₓ(DU)                                                               \n",
       "1137996  r. 1     mu-kuₓ(DU)                                                                \n",
       "1138040  r. 16     mu-kuₓ(DU) kilib3-ba                                                     \n",
       "1138044  r. 20     iti zahₓ(ŠEŠ)-da-gu7                                                     \n",
       "1138222  o. 4     mu-kuₓ(DU)                                                                \n",
       "1138694  o. 2     1 sila4 Ur-nigarₓ                                                         \n",
       "1138699  r. 1     mu-kuₓ(DU) lugal                                                          \n",
       "1138785  r. 2     Ur-nigarₓ{gar}                                                            \n",
       "1138834  o. 7     mu-kuₓ(DU)                                                                \n",
       "1138908  r. 13'     a-ša3 {d}Amar-{d}Suen-{d}Šara2-ki-ag2 še gurₓ(|ŠE.KIN|)-a               \n",
       "1138915  o.iii 2'     giri3 Lu2-tur-tur dumu  Lugal-ušurₓ(|LAL2.TUG2|)                      \n",
       "1138998  r.ii 19     ša3 Tum-alₓ(TUR3){ki}                                                  \n",
       "1139097  o.iii 26'     2 gin2 Ur-nigarₓ{gar} dumu  Puzur4-E2-a                              \n",
       "1139142  r.i 13     dam Lu2-du10-ga dumu  Nigarₓ{gar}-ki-du10                               \n",
       "1139236  s.ii 1     kurunₓ(|DIN.KAŠ|)-a-gal                                                 \n",
       "1139326  o.ii 11     Ur-nigarₓ{gar} dumu Ur-{d}Ig-alim                                      \n",
       "1139559  o.i 5     iti buruₓ(GAN2)-maš                                                      \n",
       "1139625  r.ii 6'     kišib Ur-nigarₓ{gar} dumu Ur-{d}Ig-alim                                \n",
       "1139629  r.ii 10'     iti buruₓ(GAN2)-maš-ta                                                \n",
       "1139753  o. 16     13 0.0.3 3 Lu2-Urubₓ(|URU×KAR2|){ki}                                     \n",
       "1139869  o.iv 9     dumu geme2-Urubₓ(|URU×KAR2|){ki}                                        \n",
       "1139940  r.i 17     dumu geme2-Urubₓ(|URU×KAR2|){ki}-me                                     \n",
       "1139942  r.i 19     dumu Geme2-nigarₓ{gar}                                                  \n",
       "1140478  r.iv 34     Ur-{d}Ša-uₓ(LU)-ša                                                     \n",
       "1141210  o.iv 14     Nigarₓ{gar}-ki-du10 engar                                              \n",
       "1141895  r. 4     iti zahₓ(ŠEŠ)-da-gu7                                                      \n",
       "\n",
       "[18683 rows x 1 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1)\n",
    "lines_df = pd.DataFrame(lines)\n",
    "lines_df[lines_df[0].str.contains('ₓ')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract Names\n",
    "In BDTNS names start with a capital or with {d}. The list comprehension iterates through the list of lines. It iterates through each line by splitting the line into words, testing whether the word begins with a capital or with {d}. The result is a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = [word for line in lines for word in line.split() if word[0].isupper() or word.startswith('{d}')]\n",
    "len(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove All Upper Case\n",
    "Remove words that are entirely in upper case (for instance LU2.SU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names =[word for word in names if not word.isupper()]\n",
    "len(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove Incomplete Entries\n",
    "Remove names that have ellipsis (damage) or illegible signs.\n",
    "\n",
    "Note that the x in KA×A is not a x, but the \"times\" sign. The X in x-values (such as NigarX) has been replaced in a previous cell by index x (Nigarₓ). The X in KA×X, however, is a real X and names including that sign will be eliminated as partially illegible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = [word for word in names if not '...' in word]\n",
    "names = [word for word in names if not '-x' in word.lower() \n",
    "         and not 'x-' in word.lower() \n",
    "         and not '.x' in word.lower() \n",
    "         and not 'x.' in word.lower()\n",
    "        and not '}x' in word.lower()\n",
    "        and not 'x{' in word.lower()]\n",
    "len(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove duplicates\n",
    "Reduce the list to a unique set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_names = len(names)\n",
    "#keep the total number of names for statistics\n",
    "names = set(names)\n",
    "len(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sort Alphabetically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = sorted(names)\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create DataFrame\n",
    "The DataFrame has two columns: column 1 has the original transliteration (as in BDTNS); column 2 starts out with the same data, but this data is transformed into ORACC compatible lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = ['Transliteration']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = df['Transliteration']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shin, Emphatic T, and Emphatic S\n",
    "Replace c by š, C by Š, ty by ṭ, etc.\n",
    "\n",
    "This is no longer necessary - but also doesn't hurt. Kept in case someone uses an old BDTN output file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "signs = {'c': 'š',\n",
    "        'C': 'Š',\n",
    "        'ty': 'ṭ',\n",
    "        'TY': 'Ṭ',\n",
    "        'sy': 'ṣ',\n",
    "        'SY': 'Ṣ'}\n",
    "for key in signs:\n",
    "    df['Normalized'] = [word.replace(key, signs[key]) for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sign Reading Substitution\n",
    "The conventions for reading signs in BDTNS differs from ORACC: BDTNS does not distinguish between G and nasal G (ŋ), and BDTNS uses short readings (`ku3`) where ORACC uses long readings (`kug`).\n",
    "\n",
    "The following function (`signreplace()`) replaces a BDTNS reading with an ORACC reading. The regular expression uses `\\\\b` (before and after the sign) to indicate word boundaries, so that replacing `sag` by `saŋ` does not find `sag2` etc. Word boundaries (as defined by the `regex` module) include `-`, `.`, `{`, and `}`.\n",
    "\n",
    "Since names are capitalized (as in `Dingir-nu-me-a`) each sign-replacement is run twice: once in lower case (`dingir`, replaced by `diŋir`) and once capitalized (`Dingir`, replaced by `Diŋir`).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def signreplace(old, new, data):\n",
    "    old_cap = old.capitalize()\n",
    "    new_cap = new.capitalize()\n",
    "    data = re.sub('\\\\b'+old_cap+'\\\\b', new_cap, data)\n",
    "    data = re.sub('\\\\b'+old+'\\\\b', new, data)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dictionary of signs with canonical ORACC reading\n",
    "Preliminary list of \"short\" vs. \"long\" sign readings (`du11` vs. `dug4`) and sign readings with nasal G (ŋ). The list does *not* include `mu` : `ŋu10`, because that is valid *only* at the end of a word. It is necessary to first remove morphological suffixes such as -ta, -še3, etc., which happens in the next phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bdtns_oracc = {'ag2': 'aŋ2',\n",
    "               'balag': 'balaŋ', \n",
    "               'dagal': 'daŋal', \n",
    "               'dingir': 'diŋir',\n",
    "               'eridu': 'eridug',\n",
    "               'ga2': 'ŋa2', \n",
    "               'gar': 'ŋar',\n",
    "               'geštin': 'ŋeštin',\n",
    "               'gir2': 'ŋir2',\n",
    "               'gir3': 'ŋiri3',\n",
    "               'giri3': 'ŋiri3',\n",
    "               'giš': 'ŋeš',\n",
    "               'giškim': 'ŋiškim',\n",
    "               'gišnimbar' : 'ŋešnimbar',\n",
    "               'hun': 'huŋ',\n",
    "               'kin': 'kiŋ2', \n",
    "               'nig2': 'niŋ2',\n",
    "               'nigin': 'niŋin',\n",
    "               'nigin2': 'niŋin2',\n",
    "               'pisan': 'bisaŋ',\n",
    "               'pirig': 'piriŋ',\n",
    "               'sag': 'saŋ',\n",
    "               'sanga': 'saŋŋa',\n",
    "               'šeg3': 'šeŋ3', \n",
    "               'šeg6': 'šeŋ6',\n",
    "               'umbisag': 'umbisaŋ',\n",
    "               'uri2': 'urim2',\n",
    "               'uri5': 'urim5',\n",
    "               'uru': 'iri',\n",
    "               \n",
    "               'bara2' : 'barag',\n",
    "               'du10' : 'dug3',\n",
    "               'du11' : 'dug4',\n",
    "               'gu4' : 'gud',\n",
    "               'kala' : 'kalag',\n",
    "               'ku3' : 'kug',\n",
    "               'ku5': 'kud',\n",
    "              # 'lu5' : 'lul',\n",
    "               'sa6' : 'sag9',\n",
    "               'ša6' : 'sag9',\n",
    "               'za3' : 'zag'\n",
    "           } "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in bdtns_oracc:\n",
    "    df['Normalized'] = [signreplace(key, bdtns_oracc[key], word) for word in df['Normalized']]\n",
    "#normalized = [signreplace(key, bdtns_oracc[key], word) for key in bdtns_oracc for word in df['Normalized']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Capitalize god names\n",
    "God names (as part of personal names) are not consistently capitalized (as in `{d}utu-ki-ag2`). The first character after `{d}` must be a capital. The `regex` for doing so was found [here](http://stackoverflow.com/questions/8934477/making-letters-uppercase-using-re-sub-in-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [re.sub('{d}([a-zšŋ])', lambda match: '{d}'+'{}'.format(match.group(1).upper()), word) \n",
    "                    for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove Morphology\n",
    "The following lines remove morphology that can (almost) unambigously be identified, namely `-ta` (ablative); `ke4` (genitive + ergative) and `-še3`. The genitive element of `-ke4` (`-k`) is kept when it immediately follows a vowel because in such cases it usually belongs to the name. Note that `-ra` (dative) is ambiguous, since it may be part of a name ending in /r/ like `Šul-gi-ra` (in the genitive). After removing these morphemes, word-final `-mu` is replaced by `-ŋu10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [word[:-3] if word.endswith('-ta') else word for word in df['Normalized']]\n",
    "df['Normalized'] = [word[:-4] if word.endswith('-še3') else word for word in df['Normalized']]\n",
    "df['Normalized'] = [word[:-2] if word.endswith(('a-ke4', 'e-ke4', 'i-ke4', 'u-ke4')) else word for word in df['Normalized']]\n",
    "df['Normalized'] = [word[:-4] if word.endswith('-ke4') else word for word in df['Normalized']]\n",
    "df['Normalized'] = [word[:-2]+'ŋu10' if word.endswith('-mu') else word for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[1000:1050]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names with Genitive -k\n",
    "Names such as `Nin-ŋir2-su` contain a (hidden) genitive morpheme `-(a)k` that only appears when followed by a vowel, as in `Nin-ŋir2-su-ke4`. In the preceding `Nin-ŋir2-su-ke4` has been shortened to `Nin-ŋir2-su-k` (removing the ergative morpheme). The presence of `Nin-ŋir2-su-k` in the list proves that `Nin-ŋir2-su` has a hidden geneitive and should be normalized `Ninŋirsuk`. The following cell tests for the existence of such instances, if yes, the `-k` is added to the normalized form of the name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [word + '-k' if word + '-k' in df.Normalized.values else word for word in df['Normalized']]\n",
    "df.loc[df.Transliteration.str.contains(\"Nin-gir2-su\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Replace a-a with aya\n",
    "Replace a-a and A-a with aya and Aya, but only between word boundaries. For this we can use the function `signreplace()` defined above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [signreplace('a-a', 'aya', word) for word in df['Normalized']]\n",
    "df[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove dashes and sign index numbers\n",
    "In order to produce a normalized form of the name, sign separators (dashes) and sign index numbers are removed. In first instance this is done *only* if there are no uppercase characters further on in the name (uppercase may indicate a logogram). Secondarily we will consider instances where the uppercase letter is dues to a god name (as in `A-bu-um-{d}Dumu-zi`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', ':']\n",
    "df['Normalized'] = [clean(word, remove) if word[1:].islower() \n",
    "                    or (word.startswith('{d}') and word[4:].islower()) else word for word in df['Normalized']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theophoric Names\n",
    "Theophoric names, if the god name is not at the beginning of the word, will have a capital in the middle of the word (as in `A-ba-{d}Dumu-zi-gen7`) and are thus ignored by the previous cell (no removal of dashes etc.). The following cell tests for the presence of `{d}` in the middle of the word (not at position 0) and splits that name into two. Each half may start with a capital, but should be lower case otherwise. If that is the case the dashes and index numbers are removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [clean(word, remove) if ('{d}' in word[1:] \n",
    "              and word.split('{d}')[0][1:].islower() and word.split('{d}')[1][1:].islower()) else word \n",
    "              for word in df['Normalized']]\n",
    "df[df['Normalized'].str.contains('{d}Dumu')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theophoric Names with {d} twice\n",
    "Names of the pattern `{d}Amar-{d}Suen` are still not normalized, because of the capital in position 3 (in `Amar`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [clean(word, remove) if ('{d}' in word[1:] and word.startswith('{d}')\n",
    "              and word.split('{d}')[1][1:].islower() and word.split('{d}')[2][1:].islower()) else word \n",
    "              for word in df['Normalized']]\n",
    "df[df['Transliteration'].str.contains('Suen')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theophoric Names without {d}\n",
    "God names such as `I-šum` and `E2-a` (and others?) are usually not preceded by `{d}`. We can submit those to a similar test, splitting the name at a dash when followed by `I-šum` and `E2-a`, followed by a word boundary `\\\\b` (using a positive lookahead regex). If the two halves of the split are both lower case after the initial character, remove dashes and index numbers. If other gods are found that are usually not preceded by `{d}` they can be added to the list `gods_no_d`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gods_no_d = ['I-šum', 'E2-a', 'Er3-ra', 'A-šur5']\n",
    "for god in gods_no_d:\n",
    "    df['Normalized'] = [clean(word, remove) if (god in word[2:] \n",
    "              and re.split('-(?=' + god + '\\\\b)', word)[0][1:].islower() \n",
    "              and re.split('-(?=' + god + '\\\\b)', word)[1][1:].islower()) else word \n",
    "              for word in df['Normalized']]\n",
    "df[df['Transliteration'].str.contains('E2-a')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Replace uu by u, etc.\n",
    "In the normalization replace double vowels and double consonants with single ones (replace `Abuum` with `Abum` and `Abasagga` with `Abasaga`), but not at the beginning or the end of a word.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [re.sub('\\\\B([a-z])\\\\1\\\\B', '\\\\1', word) for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Alef between Vowels\n",
    "Put alef between lowercase vowels. Note: for some reason the Alef is represented as an Ayin (`ʾ`) in the code cell. It does. however, correctly produce an Alef (`ʿ`) in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vowel_combis = {\"ae\":\"aʾe\",\n",
    "          \"ai\": \"aʾi\",\n",
    "          \"au\": \"aʾu\",\n",
    "          \"ea\": \"eʾa\",\n",
    "          \"ei\": \"eʾi\",\n",
    "          \"eu\": \"eʾu\",\n",
    "          \"ia\": \"iʾa\",\n",
    "          \"ie\": \"iʾe\",\n",
    "          \"iu\": \"iʾu\",\n",
    "          \"ua\": \"uʾa\",\n",
    "          \"ue\": \"uʾe\",\n",
    "          \"ui\": \"uʾi\"\n",
    "         }\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in vowel_combis:\n",
    "    df['Normalized'] = [word.replace(key, vowel_combis[key]) for word in df['Normalized']]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assign Proper Noun Classes\n",
    "Proper noun classes include:\n",
    "    - RN       Royal Name\n",
    "    - DN       Divine Name\n",
    "    - PN       Personal Name\n",
    "    - SN       Settlement Name\n",
    "    - GN       Geographical Name (larger Geographical units such as states)\n",
    "    - TN       Temple Name\n",
    "    - ON       Object Name (such as divine vessels and chariots)\n",
    "    - FN       Field Name\n",
    "\n",
    "The class is indicated after square brackets after the name (as in `Utu[]DN`).\n",
    "\n",
    "There are only 5 royal names: `UrNammak`, `Šulgir`, `AmarSuen`, `ŠuSuen`, and `IbbiSuen`.\n",
    "\n",
    "Settlement names are followed by the determinative {ki}.\n",
    "\n",
    "Divine names are preceded by the determinative {d}.\n",
    "\n",
    "The rest are considered to be Personal Names. Field names, Temple names, and Geographical names etc. can usually not  be recognized unambiguously.\n",
    "\n",
    "The first line of the code in the cell below adds `[]PN` to each entry in the Normalization, turning every entry into a Personal Name. Subsequent lines replace `PN` with `SN`, `DN`, or `RN` where appropriate.\n",
    "\n",
    "A certain amount of error is unavoidable. Personal names are often preceded by `{d}` because they contain a divine name. Similarly, god names may contain place names (as in `{d}Nin-Urim5{ki}`: Lady of Ur)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [word+'[]PN' for word in df['Normalized']]\n",
    "# Settlement names\n",
    "df['Normalized'] = [word[:-2]+'SN' if '{ki}' in word else word for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divine names\n",
    "df['Normalized'] = [word[:-2]+'DN' if word.startswith('{d}') and not '{d}' in word[4:] \n",
    "                    else word for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Royal names\n",
    "Shulgi = ['Šulgira', 'Šulgi', '{d}Šulgira', '{d}Šulgi']\n",
    "AmarSuen = ['Amar{d}Suʾenra', 'Amar{d}Suʾen', 'Amar{d}Suʾenka', '{d}Amar{d}Suʾenra', \n",
    "            '{d}Amar{d}Suʾen', '{d}Amar{d}Suʾenka']\n",
    "ShuSuen = ['Šu{d}Suʾen', 'Šu{d}Suʾenra', 'Šu{d}Suʾenka',\n",
    "          '{d}Šu{d}Suʾen', '{d}Šu{d}Suʾenra', '{d}Šu{d}Suʾenka']\n",
    "IbbiSuen = ['Ibi{d}Suʾen', 'Ibi{d}Suʾenra', 'Ibi{d}Suʾenka',\n",
    "          '{d}Ibi{d}Suʾen', '{d}Ibi{d}Suʾenra', '{d}Ibi{d}Suʾenka']\n",
    "df['Normalized'] = ['Šulgir[]RN' if word[:-4] in Shulgi else word for word in df['Normalized']]\n",
    "df['Normalized'] = ['AmarSuʾen[]RN' if word[:-4] in AmarSuen else word for word in df['Normalized']]\n",
    "df['Normalized'] = ['ŠuSuʾen[]RN' if word[:-4] in ShuSuen else word for word in df['Normalized']]\n",
    "df['Normalized'] = ['IbbiSuʾen[]RN' if word[:-4] in IbbiSuen else word for word in df['Normalized']]\n",
    "df['Normalized'] = ['UrNammak[]RN' if 'Ur{d}Nama' in word else word for word in df['Normalized']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['Normalized'].str.contains('[]RN', regex=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remove Determinatives\n",
    "Remove determinatives, but only from those names that have been successfully normalized (do not contain `-` or `.` anymore)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Normalized'] = [re.sub('{.+?}', '', word) if not '-' in word and not '.' in word \n",
    "                    else word for word in df['Normalized']]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Replace numbers by Index numbers\n",
    "In names that could not be normalized automatically, replace numbers by index numbers and dashes (`-`) by dots (`.`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers_index = {'0':'₀',\n",
    "               '1': '₁',\n",
    "               '2':'₂',\n",
    "               '3':'₃',\n",
    "               '4':'₄',\n",
    "               '5':'₅',\n",
    "               '6':'₆',\n",
    "               '7':'₇',\n",
    "               '8':'₈',\n",
    "               '9':'₉',\n",
    "                '-': '.'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in numbers_index:\n",
    "    df['Normalized'] = [word.replace(key, numbers_index[key]) for word in df['Normalized']]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistics\n",
    "In the current set, how many name forms and how many names? How many name forms could not be normalized?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_norm_df = df[df['Normalized'].str.contains('.', regex=False)]\n",
    "norm_df = df[~df['Normalized'].str.contains('.', regex=False)]\n",
    "not_normalized = len(not_norm_df)\n",
    "norm = len(norm_df)\n",
    "norm_set = len(set(norm_df['Normalized']))\n",
    "names_forms = len(df)\n",
    "print('Name Instances ' + str(total_names))\n",
    "print('Name forms: ' + str(names_forms))\n",
    "print('Name forms Normalized: ' + str(norm) + \"; representing \" + str(norm_set) + \" different names.\")\n",
    "print('Name forms not normalized: ' + str(not_normalized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Utamišaram\n",
    "The Ur III name `Utamišaram` appears in many different spellings and name forms. How did our script do with him?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "miša = df[df['Transliteration'].str.contains('mi-ša')]\n",
    "Utamišaram = miša[miša['Transliteration'].str[0]=='U']\n",
    "Utamišaram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out that 10 different spellings/forms are correctly identified with `Utamišaram`; others are normalized as `Udamišaram` or `Utamišarum`, or, in one case `U₂.ta₂.mi.šar.MI.ra.am` (apparently an ancient spelling mistake). Such entries need to be corrected by hand; further research shows that in one case `U2-ta-mi-car-um-ta` is a modern mistake (= `Utamišaram`); the other cases of `Utamišarum` are correctly transliterated and represent a variant form of the same name refering to the same person (an official in the Drehem administration) as `Utamišaram`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save to File\n",
    "Separator is a `TAB` (instead of comma)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('output/UrIII-Names.csv', sep = '\\t', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/query_text_19_05_15-020951.txt', mode = 'r', encoding = 'utf8') as f:\n",
    "    lines = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(lines)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}