01-using-bagit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "381dcf7a-f191-4d46-b3a3-3b3ff7013f8c",
   "metadata": {},
   "source": [
    "# Using Bagit\n",
    "\n",
    "This notebook contains all of the steps to use the python `bagit` library\n",
    "to create a valid BagIt bag."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50b360f-f0c2-431f-80b1-647bedaae34c",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Now let's look into how we can create a BagIt object for some sample files. \n",
    "This notebook will demonstrate how \n",
    "to do that using a Python module called `bagit`, from files on your computer. If you want to follow this notebook,\n",
    "the instructions explain the process, step by step, for a folder of sample files in this Git repository. \n",
    "\n",
    "If you don't already have the bagit library installed, you may need to get it. You can run the \n",
    "following cell to install it with pip, by uncommenting the last line (remove the `#`) and then running the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be9cc7a2-8d27-450b-b292-72482756d411",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you don't have bagit installed, install following instructions at https://github.com/LibraryOfCongress/bagit-python\n",
    "# Alternatively, you can use the magic command on the line below by removing the hashtag and running the cell.\n",
    "# (When the command below runs, you will see response output appear below this cell as the program downloads and installs.)\n",
    "#!pip install bagit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7da8d0d5-bacc-45f4-809f-0976cfdd33c7",
   "metadata": {},
   "source": [
    "To begin this activity, set up by importing the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ad276a-4d04-4c76-9ced-930333dd38a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bagit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa33aeed-35f7-4e32-a871-545751838de3",
   "metadata": {},
   "source": [
    "In this exercise, we will also use the system to automatically generate some data about the files.\n",
    "For the most part, this is all done by the `bagit` library, but this activity will also demonstrate\n",
    "generating date information from the system. In practice, this might be done through manual entry,\n",
    "using a date picker tool in a software, or by generating date information from the system,\n",
    "as shown here. To do this, use the `date` functions of Python's `datetime` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c5a722f-3232-4384-b468-f893f2204329",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3924635-605b-452c-a9ff-404fec58cf30",
   "metadata": {},
   "source": [
    "## Bag the Files\n",
    "\n",
    "The purpose of the “bag” is to create information about the file structure, basic information that can demonstrate that the information has not changed, and to provide basic context (information about where the files came from, who filed them, and what they are). It is an open specification, so there are few requirements about how the files are structured. In this case, I am taking all of the files within a specific folder, using the Python bagit library to generate the fixity information, and explaining each step throughout the rest of this notebook.\n",
    "\n",
    "For demonstration, let's bag the files in the directory `sample-files`. To see how the directory looks now,\n",
    "use a magic command (`!`) and the `ls`, which lists the contents of the directory. The `-F` flag adds a slash \n",
    "at the end of any contents that are a directory, which is a helpful visual indicator. This is like using a \n",
    "shell command from inside the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26134c6c-f8b4-42e0-89a2-66a7853392b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls sample-files/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66c81d9-d612-4f64-ba4a-058646675102",
   "metadata": {},
   "source": [
    "- You should see five folders and one csv file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968b0ba0-36f5-4f93-8c6f-7541da4a8bac",
   "metadata": {},
   "source": [
    "### Create BagInfo data\n",
    "\n",
    "Using the Python bagit library, we can create “BagInfo” information by using a Python dictionary. This example creates a dictionary of the bag information called `my_BagInfo`, which will be inserted as an argument during bag creation. If you use this code, replace information below with you the information appropriate to the project you’re working on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71126ff5-e874-46f1-8903-1b578e2e7132",
   "metadata": {},
   "source": [
    "### Small bonus: automate date info\n",
    "\n",
    "The `date` functions (imported earlier) will suffice to create date information. If you run this, the following block should return the current system date from your system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d37c00eb-720c-4390-b84b-d2bce3266c70",
   "metadata": {},
   "outputs": [],
   "source": [
    "dateStamp = date.today()\n",
    "\n",
    "print(dateStamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "531a358f-b697-4f2b-b8cc-a7341d1f0b2a",
   "metadata": {},
   "source": [
    "Note that the above is a Python datetime object, so for purposes of our BagIt activity, that must be converted to a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "534e07c4-632a-42dd-8bfa-97c705ac5a87",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(dateStamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6d3963e-d046-48bf-b6bb-39193ef01535",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the date stamp to a string, which should be formatted as YYYY-MM-DD by default\n",
    "str(dateStamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef91b97-94fd-4952-98e3-571800c182a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create baginfo data\n",
    "\n",
    "my_BagInfo = {\n",
    "    'Source-Organization': 'Data Curation Training Pros, via Library of Congress (LC)',\n",
    "    'Contact-Name' : 'Anonymous', # <- type your name here\n",
    "    'Contact-Email': 'hello@some.email', # <- type your email here\n",
    "    'External-Description': 'These are sample files from the Library of Congress Web Archives that we wanted to structure in BagIt for practice.',\n",
    "    'External-Identifier': 'myfiles:documents/test/files/1234', # <- this would be something like a call number or collection ID, if the content corresponds to a catalog description or digitized item\n",
    "    'Source-URL': 'https://www.loc.gov/programs/web-archiving/about-this-program/', #this is a reference URL for the collection, in this case doesn't point to each individual file\n",
    "    'Collected-Date': '2021-10-12',\n",
    "    'Demonstration-Date': str(dateStamp) #string of date formatted following ISO date standard format YYYY-MM-DD\n",
    "}\n",
    "\n",
    "print('Bag Info:\\n\\n',\n",
    "      my_BagInfo,\n",
    "     '\\n\\nDatatype: ',type(my_BagInfo))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df20e533-3c0e-45d2-8ad5-fa5264feac23",
   "metadata": {},
   "source": [
    "### Bagging the files: make_bag()\n",
    "\n",
    "Now that we have created the basic metadata for the bag (the \"Bag Info\"), we can move on to “bag” the files. In this case, the files that we wanted to bag are in a directory named `sample-files`. To make a bag, the `bagit` library has a function called `make_bag()`. We can use `help()` to get information about the `make_bag()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea35a6f4-a3e7-4f37-b769-20d577ddb09a",
   "metadata": {},
   "outputs": [],
   "source": [
    "help(bagit.make_bag)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2de33fb2-bd1c-4287-926c-52a1123599c4",
   "metadata": {},
   "source": [
    "This displays what arguments the function takes. The only required information is the location of the files that you want to bag (ie, `bag_dir`), which can be provided as a file path. As a default, no `bag_info` is provided, but we will provide the information created above. As the help function describes it, running the function will \"convert a given directory into a bag,\" which is the next step.\n",
    "\n",
    "So: use the `make_bag()` function to make the bag, and we pass in as arguments the location of the files that we want to bag (`sample-files`) and the bag info (`my_BagInfo` dictionary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f8666ed-4caa-4798-b851-610b4bf45c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the bag\n",
    "\n",
    "try:\n",
    "    my_bag = bagit.make_bag('sample-files', bag_info = my_BagInfo)\n",
    "    print('Success!')\n",
    "except:\n",
    "    print('No bag was created :(')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043fe2df-4f89-4362-9981-ea20f71fde3c",
   "metadata": {},
   "source": [
    "If the cell runs and you don't see the error message, this created a bag,\n",
    "which is accessible as a python object in the `my_bag` variable. \n",
    "(More about this later!)\n",
    "But before we move on, think about the structure of the BagIt object we previously discussed. \n",
    "If you created a bag out of the `sample-files` directory, how do you think it has changed? \n",
    "\n",
    "- What files would you expect to see in the directory now?\n",
    "- What additional folder or directory might you expect to see?\n",
    "- Where would you expect to find the files that were bagged?\n",
    "\n",
    "Now, take a look at the `sample-files` directory. If the above cell ran correctly and did not return any errors, you should see changes in the `sample-files` directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9871cb7-a5ea-45ea-984e-86ac6b5d5317",
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the contents of sample-files directory\n",
    "!ls sample-files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4b3c2e7-3071-487b-ae21-544d977eef6e",
   "metadata": {},
   "source": [
    "- What changes do you see? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9e98d91-f41c-43cf-af7e-389fa9e3d8a8",
   "metadata": {},
   "source": [
    "### What's in the Bag?\n",
    "\n",
    "To get an idea what's in the bag, you can explore the `bag` object and its data. Use the shell list command (`ls`) to see if the required bagit structure and files have been created:  \n",
    "\n",
    "- Note: run shell commands from the notebook by putting an exclamation point character at the beginning of the line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bac1593b-84fc-41e0-ad9d-079525c5aeb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check to see, is this bagit? Display the contents of the sample-files directory:\n",
    "!ls sample-files/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "145e926d-023c-4250-acb8-4cf19a9eb5a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check to see, is this bagit? First test is whether or not there's a bagit declaraction. do you see bagit.txt?\n",
    "!cat sample-files/bagit.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e47a766d-5b88-4126-990d-223494b09de3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# is this bagit? are there bag tags, specified in the bag-info.txt file? do they appear to be valid key:value combinations?\n",
    "!cat sample-files/bag-info.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d4066cf-d5cf-48ab-85c5-1ff107d73a79",
   "metadata": {},
   "source": [
    "- Is this the same information that you put in the bag info dictionary?\n",
    "- What information is here that you wasn't in the `my_baginfo` dictionary?\n",
    "\n",
    "You can also read the file contents of the `sample-files/manifest-sha256.txt`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a05c0bf-34e4-4a81-b872-8c55e9cd37bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# is this bagit? is there a manifest that lists checksums and files? \n",
    "!cat sample-files/manifest-sha256.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c35244cd-21fc-470c-91ab-65a00954a965",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check to see, is this bagit? Is there a data directory? (aka \"payload\" in the BagIt docs)\n",
    "!ls sample-files/data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c6f40d-1c48-4d42-bf03-ee20ee2d8b7b",
   "metadata": {},
   "source": [
    "- the `data` directory should include the contents of the directory, which was previously named `sample-files`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0921d09a-e954-49e9-bb22-eacf6eb33a16",
   "metadata": {},
   "source": [
    "- for further description of methods for python bagit objects, see the module documentation at https://github.com/LibraryOfCongress/bagit-python  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cfdc58e-66a2-4e06-85f8-e51685b85fb6",
   "metadata": {},
   "source": [
    "A more extensive lesson on this topic would include further explanation of tools\n",
    "within `bagit` that a digital curator may use to check bags, how to research\n",
    "errors that may occur, and how to update bag manifests when content is changed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "280e6d17-6dc4-4c64-bbd6-1a1b17005f6e",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The above activity showed the steps in creating fixity and basic descriptive information - **metadata** - for this group of files. Using an agreed-upon file packaging specification, like BagIt, allows digital curators \n",
    "to create information packages that contain basic information about the contents, and can \n",
    "help organizations exchanging content to ensure that the content that was sent was the content that was received.\n",
    "Moreover, keeping this information together can allow a repository, its maintainers, and its users, to \n",
    "be able to have some assurance that information received now is the same as that originally received.\n",
    "\n",
    "### Strengths\n",
    "\n",
    "- Adopted and used within many digital collection workflows in major libraries, including the Library of Congress, California Digital Library, large research libraries, and some state and government archives.\n",
    "- Can be easily opened and understood by standard operating systems and software on the Web, on desktop computers, laptops, and networked systems.\n",
    "- Relatively simple to add files to update, add, and delete information prior to transfer\n",
    "- Does not require or rely upon file encoding, compression, or proprietary software to create or to open\n",
    "- Reliably create and confirm the fixity of information and the completeness of digital objects in a group, according to accompanying information.\n",
    "- The file structure helps to group descriptive metadata with the content it describes, using standard filesystem tools available on most operating systems (Windows, Mac, Linux). \n",
    "- Content transmitted and received can easily be opened with standard tools, which could be useful for providing information to patrons and users. \n",
    "\n",
    "### Weaknesses\n",
    "\n",
    "- Does not work well in a production or processing environment. If you are actively creating files, or adding them\n",
    "to a folder, you don't want to create and update BagIt information every time you make a change. \n",
    "- The information in bag-info.txt is uncontrolled. While the specification does recommend some labels,\n",
    "there is a large room for ambiguity or confusion to be created when different units or organizations\n",
    "are creating metadata or bags in various workflows. \n",
    "- For robust projects with highly structured and consistent\n",
    "packages (like the Chronicling America project), bag validation must be accompanied by additional quality \n",
    "checks to understand if content is, indeed, \"complete\" (beyond just matching what was sent). In other words, if you \n",
    "are sent incomplete or faulty data, BagIt is not a tool that will help you solve that problem, though it might\n",
    "(or might not) be one that can help you to identify the problem.\n",
    "- If used in a digital repository workflow, additional documentation and strategies must accompany the process (see below for how some libraries have managed this sort of distributed environment).\n",
    "\n",
    "![Data Integrity guidance at Library of Congress, screenshot from January 2022, which illustrates data integrity monitoring process that relies on BagIt, see https://www.loc.gov/programs/digital-collections-management/inventory-and-custody/data-integrity-management/](images/loc-data-integrity.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ab3e06-b777-4d65-aba6-ba81f8049ec9",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "See these additional resources for more detailed information:\n",
    "* B. Lazorchak, [\"From There to Here, from Here to There, Digital Content is Everywhere!\"](https://blogs.loc.gov/thesignal/2012/01/from-there-to-here-from-here-to-there-digital-content-is-everywhere/), _The Signal_ (3 January 2012).\n",
    "* State Archives of North Carolina, \"[Bagger GUI User Guide](https://files.nc.gov/dncr-archives/documents/files/using_bagger.pdf)\" (Updated 2012, v. 1.5), available as of March 2018.\n",
    "* M. Phillips, [\"What do we put in our BagIt bag-info.txt files?\"](https://vphill.com/journal/post/4142/) (2015).\n",
    "* UNT Libraries, UNT OAIS Information Package Specification (2015), https://www.library.unt.edu/sites/default/files/documents/digital-libraries-uploads/Appendix_M_UNT_Libraries_OAIS_Information_Package_Specification.pdf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}