notebooks/Demo-restructuring-documents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac9ef6af",
   "metadata": {},
   "source": [
    "# Restructuring PageXML Documents\n",
    "\n",
    "It can be useful to restructure the content of a PageXML document. That is, grouping `TextRegion`s or `TextLine`s in new `TextRegion`s of `Column`s. Layout analysis typically only contains information on which text lines belong together in a text region, but if you know more about the kinds of scans in a dataset and how they are structured, you may want to use that knowledge to make the PageXML document reflect that structure. \n",
    "\n",
    "There can be many reasons to restructure the text content of a PageXML document:\n",
    "\n",
    "- Many scans are based on openings of books, in which cases they represent two pages. Splitting a scan into pages allows you to analyse, classify and process individual pages instead of combinations of pages.\n",
    "- Text regions that are horizontally aligned can be grouped into columns, which allows you to process a document column by column.\n",
    "- A scan can have main text elements and elements that are part of the margin, e.g. headers, footers, marginalia, etc. Separating them allows focusing the analysis on only the main text or only the margins. Moreover, when making running text over multiple scans or pages, having access to the main text elements as columns makes it easier to make paragraphs.\n",
    "\n",
    "As an example of a text archive, this tutorial uses a dataset provided by the [National Archives of the Netherlands](https://www.nationaalarchief.nl/en) (NA) via their HTR repository on [Zenodo](https://zenodo.org/): https://zenodo.org/record/6414086#.Y8Elk-zMIUo. The repository contains many other HTR PageXML datasets that NA made available.\n",
    "\n",
    "The dataset contains HTR output in [PageXML](https://www.primaresearch.org/tools/PAGELibraries) format of scans from the following archive: \n",
    "- (medium) _Verspreide West-Indische stukken, 1614-1875, 1.05.06, 1-1413_ ([EAD](https://www.nationaalarchief.nl/onderzoeken/archief/1.05.06/invnr/%401?query=1.05.06&search-type=inventory)). This is an archive maintained by the [Nationaal Archief](https://www.nationaalarchief.nl/en). \n",
    "\n",
    "You can download the datasets via the following URLs:\n",
    "- https://zenodo.org/record/6414086/files/HTR%20results%201.05.06%20PAGE.zip?download=1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28955c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84d5e74d",
   "metadata": {},
   "source": [
    "## Extracting PageXML files from Zip/Tar files\n",
    "\n",
    "The example zip files contains of smaller zip files, that each have a number of PageXML files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ba14c6ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'lines': 33, 'words': 165, 'text_regions': 1, 'columns': 0, 'extra': 0, 'pages': 0}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from pagexml.analysis.stats import get_doc_stats\n",
    "from pagexml.helper.file_helper import read_page_archive_files\n",
    "from pagexml.parser import parse_pagexml_files_from_archive\n",
    "\n",
    "da_archive_file = '../data/HTR results DA 0114.11 PAGE.zip'\n",
    "na_archive_file = '../data/HTR results 1.05.06 PAGE.zip'\n",
    "\n",
    "for scan in parse_pagexml_files_from_archive(na_archive_file):\n",
    "    print(scan.stats)\n",
    "    break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91f63f0d",
   "metadata": {},
   "source": [
    "The `pagexml.column_parser` module has a function called `split_lines_on_column_gaps`, which takes all lines from a `TextRegion` object and sorts them into lists of horizontally overlapping lines. Then, it measures the horizontal gap (in number of pixels) between each adjacent pair of lists. If the pixel gap surpasses the threshold, the two lists are considered to be part of different columns. If not, the lists are part of the same column. \n",
    "\n",
    "Below, the first 10 scans of the archive are analysed and split into column. As you can see, in these first scans, most of the larger text regions (those containing more than one line) represent columns. Probably, these are scans of double pages with one main text column each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3c4a045b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scan: NL-HaNA_1.05.06_1_0001.jpg\ttext regions:    1\tcolumns:    1\n",
      "\ttext region: r1\t{'lines': 33, 'words': 165, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_1_0001.jpg-column-109-136-2356-3546\t{'lines': 33, 'words': 165, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_1_0002.jpg\ttext regions:    2\tcolumns:    2\n",
      "\ttext region: r1\t{'lines': 29, 'words': 140, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 30, 'words': 146, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_1_0002.jpg-column-588-318-1853-3380\t{'lines': 29, 'words': 140, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_1_0002.jpg-column-2975-349-1841-3271\t{'lines': 30, 'words': 146, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_1_0003.jpg\ttext regions:    2\tcolumns:    2\n",
      "\ttext region: r1\t{'lines': 30, 'words': 148, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 24, 'words': 115, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_1_0003.jpg-column-600-340-1829-3329\t{'lines': 30, 'words': 148, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_1_0003.jpg-column-3003-352-1816-2557\t{'lines': 24, 'words': 115, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_1_0004.jpg\ttext regions:    0\tcolumns:    0\n",
      "\n",
      "scan: NL-HaNA_1.05.06_1_0005.jpg\ttext regions:    0\tcolumns:    0\n",
      "\n",
      "scan: NL-HaNA_1.05.06_10_0001.jpg\ttext regions:    2\tcolumns:    1\n",
      "\ttext region: r1\t{'lines': 1, 'words': 0, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 33, 'words': 254, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0001.jpg-column-580-122-1979-3414\t{'lines': 34, 'words': 254, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_10_0002.jpg\ttext regions:    3\tcolumns:    2\n",
      "\ttext region: r1\t{'lines': 1, 'words': 0, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 33, 'words': 243, 'text_regions': 0}\n",
      "\ttext region: r3\t{'lines': 33, 'words': 276, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0002.jpg-column-512-263-2009-3295\t{'lines': 34, 'words': 243, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0002.jpg-column-3030-119-2003-3371\t{'lines': 33, 'words': 276, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_10_0003.jpg\ttext regions:    4\tcolumns:    3\n",
      "\ttext region: r1\t{'lines': 1, 'words': 1, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 32, 'words': 267, 'text_regions': 0}\n",
      "\ttext region: r3\t{'lines': 1, 'words': 1, 'text_regions': 0}\n",
      "\ttext region: r4\t{'lines': 36, 'words': 277, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0003.jpg-column-127-102-109-126\t{'lines': 1, 'words': 1, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0003.jpg-column-529-272-2009-3230\t{'lines': 32, 'words': 267, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0003.jpg-column-3056-149-1993-3390\t{'lines': 37, 'words': 278, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_10_0004.jpg\ttext regions:    2\tcolumns:    2\n",
      "\ttext region: r1\t{'lines': 34, 'words': 278, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 36, 'words': 281, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0004.jpg-column-594-321-1954-3210\t{'lines': 34, 'words': 278, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0004.jpg-column-3014-89-2035-3449\t{'lines': 36, 'words': 281, 'text_regions': 0}\n",
      "\n",
      "scan: NL-HaNA_1.05.06_10_0005.jpg\ttext regions:    3\tcolumns:    3\n",
      "\ttext region: r1\t{'lines': 1, 'words': 1, 'text_regions': 0}\n",
      "\ttext region: r2\t{'lines': 34, 'words': 256, 'text_regions': 0}\n",
      "\ttext region: r3\t{'lines': 36, 'words': 291, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0005.jpg-column-178-91-91-142\t{'lines': 1, 'words': 1, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0005.jpg-column-619-266-1953-3237\t{'lines': 34, 'words': 256, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_10_0005.jpg-column-3107-111-1976-3435\t{'lines': 36, 'words': 291, 'text_regions': 0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pagexml import column_parser\n",
    "\n",
    "for si, scan in enumerate(parse_pagexml_files_from_archive(na_archive_file)):\n",
    "    columns = column_parser.split_lines_on_column_gaps(scan, gap_threshold=10)\n",
    "    print(f\"scan: {scan.id}\\ttext regions: {scan.stats['text_regions']: >4}\\tcolumns: {len(columns): >4}\")\n",
    "    for tr in scan.text_regions:\n",
    "        print(f\"\\ttext region: {tr.id}\\t{tr.stats}\")\n",
    "    for column in columns:\n",
    "        print(f\"\\tcolumn: {column.id}\\t{column.stats}\")\n",
    "    print()\n",
    "    if (si+1) >= 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212cc16a",
   "metadata": {},
   "source": [
    "One type of page where identifying columns is potentially very useful is a page containing a table or multiple lists. Especially where the scans consists of a large number of small text regions.\n",
    "\n",
    "We can look for scans with at least 10 text regions and see what columns split does with this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dc8a618c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scan: NL-HaNA_1.05.06_103_0001.jpg\ttext regions:  161\tcolumns:   10\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-45-223-3131-6267\t{'lines': 159, 'words': 153, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-3192-467-310-4879\t{'lines': 6, 'words': 5, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-3556-423-656-5220\t{'lines': 40, 'words': 26, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-4239-241-689-6195\t{'lines': 31, 'words': 22, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-4942-392-480-5160\t{'lines': 16, 'words': 12, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-5467-306-914-5279\t{'lines': 44, 'words': 26, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-6398-343-790-5152\t{'lines': 42, 'words': 30, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-7281-249-357-3875\t{'lines': 10, 'words': 4, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-7866-386-362-5207\t{'lines': 39, 'words': 33, 'text_regions': 0}\n",
      "\tcolumn: NL-HaNA_1.05.06_103_0001.jpg-column-8270-1209-261-4589\t{'lines': 23, 'words': 18, 'text_regions': 0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pagexml import column_parser\n",
    "\n",
    "for si, scan in enumerate(parse_pagexml_files_from_archive(na_archive_file)):\n",
    "    if scan.stats['text_regions'] < 10:\n",
    "        continue\n",
    "    columns = column_parser.split_lines_on_column_gaps(scan, gap_threshold=10)\n",
    "    print(f\"scan: {scan.id}\\ttext regions: {scan.stats['text_regions']: >4}\\tcolumns: {len(columns): >4}\")\n",
    "\n",
    "    for column in columns:\n",
    "        print(f\"\\tcolumn: {column.id}\\t{column.stats}\")\n",
    "    print()\n",
    "    if (si+1) >= 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4ce97f6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           N                 1      H\n",
      "                                                       5\n",
      "                                         ee\n",
      "                                                       400\n",
      "                       e    8                        2\n",
      "                                   130                  5\n",
      "                                 dC              N\n",
      "                                         9 d —\n",
      "                  d\n",
      "                                 PpC v9      o5 o65\n",
      "         E                  235\n",
      "                                   152\n",
      "                          5          7 15   13 5     5\n",
      "  Kaassen in Poort    1                                  3\n",
      "                                                          N\n",
      "                                          1\n",
      "  Rogie Meet.       1    d       1—       E     D.\n",
      "                     7\n",
      "  Jarwe J Meel.\n",
      "                       D\n",
      "                           1                          2\n",
      "                     N\n",
      "   Zout.\n",
      "  Stokvis in Soort                                 „400\n",
      "  Booter            P\n",
      "                                                    5\n",
      "   Boonen in Soort.             DD\n",
      "                        —\n",
      "   Haver                              5\n",
      "\n",
      "  Blauw EErten.\n",
      "                                             1\n",
      "                                                          17\n",
      "   Grauw EErwten.\n",
      "                                             6\n",
      "                           1\n",
      "                       17          5\n",
      "    Eventen\n",
      "\n",
      "    Lozynen Prumen &amp;                                —\n",
      "                                    —\n",
      "               —.\n",
      "  Gemeene Gort                  2          2\n",
      "                       —\n",
      "    Borger Gort.\n",
      "                                    3             1\n",
      "   Kaas in Soort\n",
      "                                            5\n",
      "   Hammen.                                         5.\n",
      "                                            5\n",
      "    Gerookt Vleesch.                          3           3\n",
      "                      8\n",
      "                     3\n",
      "   Raap Olij.\n",
      "                    L„    1\n",
      "    Lyn &am Olij.                                          P\n",
      "                     „\n",
      "    Azyn in soort.\n",
      "   Witte Wijn.\n",
      "                                            No\n",
      "   Genever Brandewijn\n",
      "                                            5\n",
      "                                            —2\n",
      "   &amp; Licqueurs.                     5 1\n",
      "  Bier.            3\n",
      "                                            3\n",
      "   Vleesch                E\n",
      "                       —         5 E       E        5\n",
      "                                         —\n",
      "  Zoeten Olij                             20\n",
      "                    5\n",
      "Zunnaamsche delrr\n",
      "\n",
      "                                  2                     2\n",
      "Schoppen\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "VE\n",
      "46\n",
      " C\n",
      "P\n",
      "\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "   15\n",
      "    R\n",
      " 5\n",
      "5        5\n",
      "           5\n",
      "           _\n",
      "       5\n",
      "            5\n",
      "        —\n",
      " 1        —\n",
      "        2\n",
      "1         17\n",
      "\n",
      "   1\n",
      "       —\n",
      "       5\n",
      "          E\n",
      "          5\n",
      "           D\n",
      "          —\n",
      "\n",
      "          3\n",
      "  20\n",
      "       5\n",
      "           5\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "           s\n",
      "   15\n",
      " „\n",
      "    9\n",
      "ve        3\n",
      "Do\n",
      "5  —    5\n",
      "       2\n",
      "       2\n",
      "        —\n",
      "        57\n",
      "\n",
      "       7\n",
      "       N\n",
      "       3\n",
      "       5\n",
      "     1\n",
      "       5\n",
      "    2\n",
      "          5\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "rw\n",
      "5\n",
      "    —   5\n",
      "     n\n",
      "7\n",
      "2\n",
      "    1\n",
      "       1\n",
      "        1\n",
      "   E E\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "       0\n",
      "          15\n",
      "   15\n",
      "      13\n",
      "         1\n",
      "                 55\n",
      "   vo\n",
      "5 5 7    5  50\n",
      "       5\n",
      "3\n",
      "\n",
      "             —\n",
      "    5\n",
      "   5\n",
      "  8\n",
      "\n",
      "   5\n",
      "   —\n",
      "   5\n",
      "\n",
      "  5\n",
      "  P\n",
      "         8\n",
      "            — 5\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "        PN\n",
      "   5            „\n",
      "        _\n",
      "     _\n",
      "                1\n",
      "           N\n",
      "                d:\n",
      "--\n",
      "\n",
      "6      5 32\n",
      "S\n",
      "2     15  15\n",
      "   5\n",
      "     7     F\n",
      "   x\n",
      "\n",
      "   8         do\n",
      "   2\n",
      "\n",
      "               5\n",
      "    57\n",
      "   5\n",
      "   —\n",
      "   9\n",
      "    11\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "_\n",
      "1\n",
      "     D\n",
      "\n",
      " 3\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      "   5\n",
      "L\n",
      "\n",
      "N\n",
      "\n",
      "„\n",
      "_\n",
      "N„\n",
      "\n",
      "   „\n",
      "\n",
      "   —\n",
      "   8\n",
      "3\n",
      "_o5\n",
      "\n",
      "P 5\n",
      "\n",
      "   5\n",
      "\n",
      "9—\n",
      "\n",
      "d\n",
      "\n",
      "    5\n",
      "5\n",
      "tuks\n",
      "\n",
      "   —\n",
      "\n",
      "   F\n",
      "\n",
      " —- - - -\n",
      "\n",
      "  D\n",
      "  8\n",
      "\n",
      "  5\n",
      " „\n",
      "\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n",
      " 5\n",
      " 8\n",
      " —\n",
      "—\n",
      "\n",
      "—\n",
      "17\n",
      "\n",
      "v\n",
      " 5\n",
      "9\n",
      "5\n",
      "—\n",
      "\n",
      " 8\n",
      "5\n",
      "55\n",
      "\n",
      " 81.\n",
      "3\n",
      "\n",
      "2\n",
      "5\n",
      "\n",
      "\n",
      "\n",
      "------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pagexml.helper.pagexml_helper import pretty_print_textregion\n",
    "\n",
    "for column in columns:\n",
    "    pretty_print_textregion(column)\n",
    "    print('\\n------------------------------------\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}