01-inventory-and-fixity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01. Inventory and metadata\n",
    "\n",
    "This file demonstrates how to use Python and the bagger module \n",
    "to create metadata about a group of files, inventory the files, \n",
    "create fixity information for each of the original files, and \n",
    "then to create a BagIt object for the files. \n",
    "\n",
    "**Note that this operation has already been performed in this git repository.\n",
    "There is _no need to perform these steps again_, since that would create another layer\n",
    "of bag metadata.**\n",
    "\n",
    "See the reference for the BagIt Python tool at https://github.com/LibraryOfCongress/bagit-python.\n",
    "\n",
    "To begin, you'll need to ensure that the bagit module for Python is installed on your system:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you don't have bagit installed, install following instructions at https://github.com/LibraryOfCongress/bagit-python\n",
    "# Alternatively, you can use the magic command on the line below by removing the hashtag and running the cell.\n",
    "# (When the command below runs, you will see response output appear below this cell as the program downloads and installs.)\n",
    "#!pip install bagit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the bagit module is ready, import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bagit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag the files\n",
    "\n",
    "The purpose of the \"bag\" is to create information about the file structure, basic information that can demonstrate that the information has not changed, and to provide basic context (information about where the files came from, who filed them, and what they are). It is an open specification, so there are few requirements about how the files are structured. In this case, I am taking all of the files within a specific folder, using the Python bagit library to generate the fixity information, and explaining each step throughout the rest of this notebook:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A well-formed bag should include as much information on the \"tag\" as possible, \n",
    "since this is where we can include information about the source and provenance of the \n",
    "data. This \"BagInfo\" information can be added using arguments in the functions that \n",
    "create bags. This example creates a dictionary of the bag information called `my_BagInfo`,\n",
    "which will be inserted as an argument during bag creation. If you use this code, \n",
    "replace information below with you the information appropriate to the project you're working on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bag Info:\n",
      " {'Contact-Name': 'Jesse Johnston', 'Contact-Email': 'morskyjezek@gmail.com', 'External-Description': 'NEH Grant data files downloaded from NEH in December 2020.', 'Source-Organization': 'National Endowment for the Humanities (NEH)', 'Source-URL': 'https://securegrants.neh.gov/open/data/', 'Collected-Date': '2020-12-14'}\n"
     ]
    }
   ],
   "source": [
    "# create baginfo data\n",
    "my_BagInfo = {\n",
    "    'Contact-Name' : 'Jesse Johnston',\n",
    "    'Contact-Email': 'morskyjezek@gmail.com',\n",
    "    'External-Description': 'NEH Grant data files downloaded from NEH in December 2020.',\n",
    "    'Source-Organization': 'National Endowment for the Humanities (NEH)',\n",
    "    'Source-URL': 'https://securegrants.neh.gov/open/data/',\n",
    "    'Collected-Date': '2020-12-14'\n",
    "}\n",
    "\n",
    "print('Bag Info:\\n',my_BagInfo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the tool working and created basic metadata for the bag, we can move on to \"bag\" the files. In this case, the files that we wanted to bag were in a directory named `neh-grants-data-2012`. We use the `make_bag()` function to make the bag, and we pass in as arguments the location of the files that we want to bag, the bag info (`my_BagInfo` dictionary), and in this case designated the `utf-8` text encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the bag\n",
    "bag = bagit.make_bag('neh-grants-data-202012', bag_info = my_BagInfo, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the `is_valid()` function to see if the bag object is ready, and if it is indeed an object that we can validate is a well-formed BagIt object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yay :)\n"
     ]
    }
   ],
   "source": [
    "# check to see if the bag is valid\n",
    "if bag.is_valid():\n",
    "    print(\"yay :)\")\n",
    "else:\n",
    "    print(\"boo :(\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. path:data/NEH_Grants1960s.csv sha256:20b521a035307e01fdfe00288806eb80498c9e9d19f02ffa3b5da5a4bfbaab41\n",
      "2. path:data/NEH_Grants1960s.zip sha256:8ae10fac88a052c0b1c7f9f35028dd2fe58950a43038c6000f9de4ed20d3e3e9\n",
      "3. path:data/NEH_Grants1960s_Flat.zip sha256:7c3c3464769e9f0918ac6afc271dd4c7cfb7d6098ea6cd510b9cd6ec674cf1b9\n",
      "4. path:data/NEH_Grants1970s.csv sha256:f1bc38c2124e6dc03fef8663a07b4496131aeba4c50ccd94266403571e097064\n",
      "5. path:data/NEH_Grants1970s.zip sha256:8fae9380ae095414bb7a2722deac19ac25b250255e4a5ad8ebf50f20ee909777\n",
      "6. path:data/NEH_Grants1970s_Flat.zip sha256:f671023d9552b95724bfe2b66d643cb7f4a7041180642f1cad971d776194d7a5\n",
      "7. path:data/NEH_Grants1980s.csv sha256:46c78195f14be0c99ca605550bd0389d9fcd6a9d698ea38f019fab639c16766c\n",
      "8. path:data/NEH_Grants1980s.zip sha256:a6739f1d981895915debacb65a86ee972d70c16ab3ecd1aeb340e9e64664173d\n",
      "9. path:data/NEH_Grants1980s_Flat.zip sha256:60d63627583b0c175535a0bb6d6044256b721d3f31ba214db825edebc29ac498\n",
      "10. path:data/NEH_Grants1990s.csv sha256:59483caaa328bd1e1f5883e6811180fd5fb50a7ffa4bc866873f42c231f5fab8\n",
      "11. path:data/NEH_Grants1990s.zip sha256:5cf59671595ab7480205276f911ccf173084b23473d1cb17015ee98414dbdb58\n",
      "12. path:data/NEH_Grants1990s_Flat.zip sha256:c756220a57f47f0fe1e975462627ec7afd6431bf738768c0d52a1c57f023f70d\n",
      "13. path:data/NEH_Grants2000s.csv sha256:e32cd052f276161ba01d8cc2ad5dab8b00a6827b4ff078b1be4266804286e28c\n",
      "14. path:data/NEH_Grants2000s.zip sha256:249a38c8514ba650b33e5d81cff805f591b33997ffc649bc519f2426145e9c65\n",
      "15. path:data/NEH_Grants2000s_Flat.zip sha256:8c15ce6bef5acf45eb8973113189ee1bff7f5f5ea526244dfcdc5238b46cc2bd\n",
      "16. path:data/NEH_Grants2010s.csv sha256:8ffc259b4b86b23b76624573ee1751e0e4573620d50d0772199542faa2435bf3\n",
      "17. path:data/NEH_Grants2010s.zip sha256:80fd5164ab027515fa4420ed1bcbfa4a913ee1bc4d4f80f14bb98dfc560dad83\n",
      "18. path:data/NEH_Grants2010s_Flat.zip sha256:d56b828a94a0c17c93b6ffd625bd56104d9df0ae3ebb4d653c8c67be75f4b76f\n",
      "19. path:data/NEH_Grants2020s.csv sha256:e85c45f1d326b6bcd0bbc044d8a224c1bac2c07c6247169b94328d0132242a62\n",
      "20. path:data/NEH_Grants2020s.zip sha256:b1cd8f3640a6ad4ed8b99b8ebaad5fab2927026a8feaf319680a5992def6c375\n",
      "21. path:data/NEH_Grants2020s_Flat.zip sha256:c756b3781a61fb82a06455478c386d8290eb6ddaeb925c6d7770ff652f7df67b\n",
      "22. path:data/NEH_GrantsDictionary.pdf sha256:8453313299588cc516f7a8233e3b04d5f6bb94bf80bcc5ebc0da7d38452b778f\n",
      "23. path:data/NEH_GrantsReference.pdf sha256:9df1d6593740b97c555fbe686b441be97d97a72cd3d051a20380eecdf69f8ddc\n",
      "24. path:data/README.md sha256:021683e6f0fb988a3226603c419339a058d8f0acf17f9c20201ad1fcb92d0897\n",
      "25. path:bagit.txt sha256:e91f941be5973ff71f1dccbdd1a32d598881893a7f21be516aca743da38b1689\n",
      "26. path:bag-info.txt sha256:1d637aa8471cfe8135be09dfc6d9fc2abe99d27bd57007822b6e3c3c195dc75b\n",
      "27. path:manifest-sha512.txt sha256:c8219eb061032b1732c525064f2d27a042144c92fc4f306b75acb9f55309e359\n",
      "28. path:manifest-sha256.txt sha256:ad6abefb8dbe8bbdd8ccb7addc1c3eb76ef7062456f199ef44adf5ca67fa8f50\n"
     ]
    }
   ],
   "source": [
    "# to get an idea what's in the bag, display a list of the files and fixity information\n",
    "line_count = 0\n",
    "for path, fixity in bag.entries.items():\n",
    "    line_count += 1\n",
    "    print(\"%s. path:%s sha256:%s\" % (line_count, path, fixity['sha256']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have created basic metadata for this group of files, two types of fixity signatures (sha256 and sha512), and an inventory of the files (see `neh-grants-data-202012/manifest-sha256.txt`). This will serve as a basic description of the original files. In the next activities, we will continue to work with this information, but the original files\n",
    "will remain unchanged and available for further consultation or work beyond this project. All of our work will be to extract, transform, and clean the data that we pull from these files, which will be the basis for further computation, visualization, or other analysis. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}