harvard_art_museum_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with APIs in Python\n",
    "\n",
    "Making API requests in Python can be really simple. There's a low-level module called urllib that can also make the kinds of web requests that we want, but it's not as friendly as the `requests` module, which we'll be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication\n",
    "\n",
    "You'll have to authenticate each request to the Harvard Art Museum API with an API key. Other APIs may require different kinds of authentication (sometimes very complicated auth! Look for libraries at that point), but HAM has some pretty simple authentication, which makes things easy for us. You can sign up for a key [here](https://www.harvardartmuseums.org/collections/api)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "APIKEY = \"b0cde630-ce66-11e8-951c-b3d75228cc98\" # Enter your API key here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic request\n",
    "\n",
    "We're going to start off with a basic request to the API. This API, like many others, has a variety of endpoints, each with their own url, slightly modified from a base url. We'll worry about the general case in a bit, for now let's look at a basic API request.\n",
    "\n",
    "In this example, we'll re-create the first example in the [Object endpoint documentation](https://github.com/harvardartmuseums/api-docs/blob/master/object.md), which will give each of you the records for 10 objects that have never been viewed online in the museum's collections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = \"https://api.harvardartmuseums.org/object\"\n",
    "parameters = {\n",
    "    \"q\":\"totalpageviews:0\",\n",
    "    \"size\":10,\n",
    "    \"apikey\":APIKEY\n",
    "}\n",
    "R = requests.get(url,params=parameters)\n",
    "R.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Refresher on Dictionaries\n",
    "\n",
    "Python dictionaries are sets of key / value pairs, where a value can be accessed by its key. You're essentially naming a value in a container, so you can easily call it up later.\n",
    "\n",
    "Dictionaries have very fast lookups, so you can get a value from its key very quickly, no matter how large the dictionary is. However, they are also unordered, so if you iterate through all of the key / value pairs in the dictionary, there's no guarantee that they'll be in the same order.\n",
    "\n",
    "We're just going to be looking up data in dictionaries, so here's a quick refresher on the syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters['q']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters['apikey'] # This also works when we've set the value to another variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters['q'] = \"totalpageviews:1\" # You can also set the value of a key like you would a variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a Request\n",
    "\n",
    "The request syntax is so simple, you might have missed it. Let's query again for objects with only one pageview, and take a closer look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R = requests.get(url,params=parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formatted parameters\n",
    "\n",
    "That request has created a request object, which contains not only the data that we get from the Harvard Art Museums, but information on the request we sent, like the URL that it used. Notice that requests has turned our query parameter dictionary into a GET request at the end of our URL.\n",
    "\n",
    "If you've been working with API requests or web scraping before, you might be used to seeing URLs get constructed like this:\n",
    "\n",
    "```python\n",
    "url = \"https://api.harvardartmuseums.org/object?q=\" + query + \"&apikey=\" + apikey\n",
    "```\n",
    "\n",
    "If you have, I'm sure you'll appreciate how much simpler this is, especially when dealing with more query parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Taking a look at the results\n",
    "\n",
    "Request objects have a built-in method, `.json()`, which converts a JSON file received as a response to a request from a string of text that happens to be in this data format into Python native data structures, like lists, dictionaries, numbers and strings. We can use this method to see a dictionary representation of what we've gotten from the API request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing our request\n",
    "\n",
    "Let's say we're not interested in the most obscure parts of the collection (pot sherds, apparently), but rather in the most popular parts of the collection. There are a few ways we might go about doing this. One way might be to sort our search results by `totalpageviews`, and see what the top 10 are.\n",
    "\n",
    "To do that, we can go back to the [API documentation](https://github.com/harvardartmuseums/api-docs/blob/master/object.md) and look for hints about what we might be able to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    \"size\":10,\n",
    "    \"apikey\":APIKEY,\n",
    "    \"sort\": \"totalpageviews\",\n",
    "    \"sortorder\": \"desc\"\n",
    "}\n",
    "R = requests.get(url,params=parameters)\n",
    "R.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looking at the results\n",
    "\n",
    "Often, you'll want to look at some specific aspect of the data you're getting. Since the API returns everything, you'll have to format the output in some friendly, readable format. In this next cell, we'll iterate through the results and print them  out in a nicer format.\n",
    "\n",
    "We're being pretty low-level with the text formatting here, and one important key to understanding this bit is that \"\\t\" means \"tab\", so that you can insert that character, which normally does something else.\n",
    "\n",
    "Feel free to play around with this cell to format it more to your liking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "records = R.json()['records']\n",
    "print(\"views\\tartwork\")\n",
    "print()\n",
    "for record in records:\n",
    "    print(\"{}\\t{}\".format(record['totalpageviews'],record['title']))\n",
    "    # `.format` puts its arguments sequentially in the string calling it wherever there are {} pairs\n",
    "    # It does a lot more than that, with more advanced documentation here: \n",
    "    # https://docs.python.org/3.4/library/string.html#id1\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top result from this query is a Van Gogh painted titled \"Self-Portrait Dedicated to Paul Gauguin.\" You can grab just the first object by accessing the records list (which is indexed from 0):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topResult = R.json()['records'][0]\n",
    "topResult"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily access properties from the image record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topResult['title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try adding an additional code field to the notebook below to access information about Van Gogh. If that's easy, try displaying all HAM works by Van Gogh and filtering to only records with an image associated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More endpoints to love\n",
    "\n",
    "You might notice, looking at the documentation, that we've only been accessing the \"objects\" API endpoint, when there are many other endpoints that we could ask for information.\n",
    "\n",
    "A note on terminology, an API endpoint is a one place that you can go to ask specific questions about a certain part of a dataset or service. Many APIs, especially commercial APIs, contain many, many endpoints, to facilitate all sorts of different activity on a platform.\n",
    "\n",
    "For example, you can take a look at the [reddit API documentation](https://www.reddit.com/dev/api/) (which we won't be using, this is just an example), to see all of the different endpoints that an application might need to serve as an alternative front end for reddit. \n",
    "\n",
    "Endpoints on the same API are likely to behave similarly, but they will all serve different purposes. Looking at our HAM endpoints, it looks like they all follow the same basic formulation: https://api.harvardartmuseums.org/RESOURCE_TYPE . We can use this to our advantage, and create a function to query any endpoint easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ham_query(apikey, endpoint, **kwargs):\n",
    "    \"\"\"Sends kwargs to the specified endpoint, using apikey for authentication\"\"\"\n",
    "    params = kwargs\n",
    "    params['apikey'] = apikey\n",
    "    url = \"https://api.harvardartmuseums.org/{}\".format(endpoint)\n",
    "    R = requests.get(url,params=params)\n",
    "    return R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "response = ham_query(APIKEY, \"gallery\", floor=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boy, that's convenient!\n",
    "\n",
    "That function works because Python has this neat ability to take arbitrary arguments in functions, if you tell it to. Essentially, there are two special arguments in function definitions: `*args` and `**kwargs`. These make available `args` and `kwargs` objects, respectively, in your function. `args` is a list, and `kwargs` is a dictionary. This makes it so that you don't have to specify all of the arguments your function can take, you can just give it general rules for lists or key pairs of data as input.\n",
    "\n",
    "You might be wondering why you wouldn't just use a dictionary or list instead of those arguments. In our case, it's mostly a stylistic choice, and one that saves us a couple of key strokes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Try out some other endpoints!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here's an example: all of the current exhibits with their begin and end dates\n",
    "response = ham_query(APIKEY, \"exhibition\", status=\"current\")\n",
    "current_exhibits = response.json()['records']\n",
    "current_exhibits\n",
    "print()\n",
    "for exhibit in current_exhibits:\n",
    "    print(\"{} ({} to {})\".format(exhibit['title'],exhibit['begindate'],exhibit['enddate'])) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Individual Objects and IIIF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HAM object API can provide more information (such as `exhibition`, `citation`, `publication`, and `marks`) if you ask for a specific object by its objectid. For some records that have been extensively annotated (often those with `verificationlevel` == 4) the lists for these properties can contain hundreds of entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "objectid = topResult['objectid']\n",
    "objectid\n",
    "parameters = {\n",
    "    \"apikey\": APIKEY\n",
    "}\n",
    "objectUrl = url + \"/\" + str(objectid)\n",
    "R = requests.get(objectUrl, parameters)\n",
    "topResultFull = R.json()\n",
    "print(topResultFull['verificationlevel'] == 4)\n",
    "topResultFull"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we printed the 10 most popular records above (under Looking at the Results), you may have noticed a sharp dropoff after the first few records. Our Van Gogh painting is particularly popular, with ~5000 more views than the second most popular record and almost 4x as many as the tenth most popular. This particular Art Museum record is used as the default image asset for the demo installation of [Project Mirador](http://projectmirador.org/demo/), an image viewer for [IIIF (International Image Interoperability Framework)](https://iiif.io/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're not going to go deep into IIIF in this workshop, but want to mention that IIIF is both a community of developers and a collection of APIs and API-compliant tools that you can use to share, manipulate, and display visual materials. The [Image API]() and [Presentation API]() are the most used outputs as of now, though there are also APIs for Authentication, Search, and beta versions for other media (video and VR)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Explanation of Image and Presi APIs (manifest, canvas especially) - images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within our topResultFull object, there is an images list, which contains IIIF baseurls as well as Image Delivery Service URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "images = topResultFull['images']\n",
    "images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This particular record has 6 images associated with it. Try opening some of the urls:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for index, image in enumerate(images, start=1):\n",
    "    print(\"image{} baseimageurl: {}\\nimage{} iiifbaseuri: {}\".format(index,image['baseimageurl'],index,image['iiifbaseuri']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the `baseimageurls` use Harvard's Name Resolution service, which redirects to an Image Delivery Service URL. We're more interested in the `iiifbaseuris` because we can manipulate IIIF resources using the Image API. Try opening one of those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IIIF Image API spec requires that we pass not just a baseurl, but a well-formed IIIF-compliant URI to get an image. Let's check out that [documentation](https://iiif.io/api/image/2.1/) and see what else we need to construct one of those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the docs:\n",
    "\n",
    ">The IIIF Image API URI for requesting an image must conform to the following URI Template:\n",
    ">\n",
    ">`{scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format}`\n",
    ">\n",
    ">For example:\n",
    ">\n",
    ">`http://www.example.org/image-service/abcd1234/full/full/0/default.jpg`\n",
    ">\n",
    "The parameters of the Image Request URI include region, size, rotation, quality and format, which define the characteristics of the returned image. These are described in detail in Image Request Parameters.\n",
    "\n",
    "The `iiifbaseuri`s include up through the `{identifier}`, but we need to include additional parameters to get the server to actually render the image for us. These parameters are passed within the URI itself, rather than in a query string appended after a delimiter (usually `?`), which is what we've been using `requests` to do. Let's write a function that can generate IIIF URIs for us. Because all of the parameters we want to insert are required, we won't use `**kwargs` - instead we'll set default params which you can override by passing in new ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def iiif_query(baseuri, region=\"full\", size=\"full\", rotation=0, quality=\"default\", format=\"jpg\", info=False):\n",
    "    \"\"\"Creates a valid IIIF URL, with the option to request image information\"\"\"\n",
    "    if baseuri[-1:] != \"/\":\n",
    "        baseuri += \"/\"\n",
    "    if info == True:\n",
    "        return baseuri+\"info.json\"\n",
    "    else:\n",
    "        url = baseuri+\"{}/{}/{}/{}.{}\".format(region, size, rotation, quality, format)\n",
    "        return url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for image in images:\n",
    "    print(iiif_query(image['iiifbaseuri']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have some valid image URLs that you can open in your browser!\n",
    "\n",
    "This is nice, but the Image API lets us do a lot more by just by passing in some parameters. Maybe we want to generate some square, grayscale images for a gallery:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for image in images:\n",
    "    print(iiif_query(image['iiifbaseuri'], quality=\"gray\", region=\"square\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try requesting only the right half of an image, in black and white, and getting back a PNG:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(iiif_query([image][0]['iiifbaseuri'], quality='bitonal', region=\"pct:50,0,100,100\", format=\"png\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to try to manipulate the images in other ways as well! That's it for our quick introduction to the Image API.\n",
    "\n",
    "Later in the session we'll consume these resources using [Mirador](http://projectmirador.org/), an image viewer which uses the IIIF Image and Presentation APIs. If you're interested in the Presi API, check out [this documentation](https://iiif.io/api/presentation/2.1/) to learn how IIIF manifests structure sequences of canvases which image viewers then present to end users. You can find an HAM Object's manifest in the `seeAlso` property, or by by appending the object ID to a baseurl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(topResult['seeAlso'])\n",
    "print('https://iiif.harvardartmuseums.org/manifests/object/{}'.format(topResult['id']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More stuff!\n",
    "\n",
    "So far, we've only been getting limited sets of object data. But what if there were a big query we wanted to make? Let's try it out on \"Unidentified culture\" materials in the museum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unknown = ham_query(APIKEY, \"object\", culture=\"Unidentified culture\", size=100).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at our previous queries, it looks like we've got some information about our query in the \"info\" section. Let's take a look at that..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unknown['info']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterating through pages\n",
    "\n",
    "It looks like we have 7 pages of data to get, and our response gives us a \"next\" url for easy iteration. Nice!\n",
    "\n",
    "However, let's look at how we would iterate even without this convenience factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unknown.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like we have two components to our response, info and records. Since `info` is request specific, we're just after `records`, and we'll want to combine them all. \n",
    "\n",
    "We could set this up in a regular loop, which would query the API as fast as our processors can go, which can produce many queries per second, and is usually limited more by network speed than by processor speed. However, this can put a strain on the API endpoint, so it can be good practice to build in timers when making many requests. Sometimes an API will specify a number of requests/second that you're allowed to make, sometimes not. Putting even a fraction of a second delay in your code will help make sure that you don't accidentally get yourself banned from the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unknown_records = []\n",
    "keepGoing = True\n",
    "page = 1\n",
    "\n",
    "while keepGoing:\n",
    "    R = ham_query(APIKEY, \"object\", culture=\"Unidentified culture\", size=100, page=page)\n",
    "    time.sleep(0.5)\n",
    "    response = R.json()\n",
    "    unknown_records.extend(response['records'])\n",
    "    if response['info']['pages'] == page:\n",
    "        keepGoing = False\n",
    "    else:\n",
    "        page += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(unknown_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting\n",
    "\n",
    "Now we have some cool data, but maybe we want to do something with it outside of Python. It's common to see CSV data traded around, since it's just a plain text spreadsheet file, so most things can parse it. Let's make one of those! We could use the relatively low level `csv` library, but instead, let's use a higher level library, `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd # Common invocation of pandas. Gotta save those 4 keystrokes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Be a dataframe!\" - us\n",
    "\n",
    "Pandas thinks of things in terms of dataframes, which will be familiar if you work in R. Basically, they're really efficient arrays of data. They also translate really well to a tabular format.\n",
    "\n",
    "To make an iterable object into a dataframe, sometimes you can just get away with shouting \"Hey you! Be a dataframe!\" at it (in code). Since we have a list of dictionaries with consistent keys, there's a good chance this process will do something smart for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(unknown_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you know! It worked. But let's take a look at a more hands on approach to the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame.from_dict(unknown_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pd.DataFrame.from_dict` gives you more control over the conversion process, so you can provide more options if things don't look how you expect them to.\n",
    "\n",
    "As a side note, we do have some data structures in here that don't make a lot of sense in a tabular format. Look at `worktypes` at the very end. That's a list, and each cell has list data in it. We won't be able to do much with that in Excel or some other tabular data processing tool, but it also won't break anything for us. It just looks weird. Within the dataframe, they still work like lists though, so you can access the data while you're still in Python if you're clever about it.\n",
    "\n",
    "### Exporting\n",
    "\n",
    "From here, our export process is really easy. We just say \"Hey you! Be a CSV file now!\", and so it shall be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(unknown_records)\n",
    "df.to_csv(\"unknown_ham_records.csv\",index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data collected!\n",
    "\n",
    "Now we've got some interesting data, and we can throw it into a program like Tableau or Excel to visualize it or further explore it. We could also continue to explore it in Python, but that's a workshop for another day."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reverse the flow!\n",
    "\n",
    "Now that we've got some information from the Harvard Art Museum API, let's look at how we can send that information somewhere else to add content to another site. \n",
    "\n",
    "Here, we'll use the API for Omeka. Omeka is a content management system, like WordPress, but focused on making the collections of libraries, archives, and museums more easily accessible on the web. It's built around the concept of items, and focuses on describing those items, collecting them sensibly, and incorporating them into online narratives.\n",
    "\n",
    "The site we'll be using is the site that we use for testing our Omeka service here: http://demo.omeka-dev.fas.harvard.edu/\n",
    "\n",
    "You'll find documentation for the API here: http://omeka.readthedocs.io/en/latest/Reference/api/index.html\n",
    "\n",
    "Our goal for this portion will be to use the documentation and what we've already learned to create items in Omeka representing each of the places in our dataframe.\n",
    "\n",
    "Before we get started on that, we'll want to see how the API represents items, so we can copy that when creating new ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "omeka_api_key = '11db6a2b70226f1c55b63a6df75e8093e9bcd01a' # We'll give you a key to use for the site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R = requests.get('http://demo.omeka.fas.harvard.edu/api/items/378', params={'key':omeka_api_key})\n",
    "demo_item = R.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "demo_item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Well that looks weird\n",
    "\n",
    "There are a lot of different fields, and some of them have a few different layers. It's a lot less flat than our HAM response. Let's take a look at it in a different way..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for k,v in demo_item.items():\n",
    "    print(\"{0: >18}:\\t{1} type\".format(k, type(v)))\n",
    "    # I just went a bit wild with string formatting in the line above, documentation here:\n",
    "    # https://docs.python.org/2/library/string.html#formatstrings\n",
    "    # Basically, I right justified the first format target with spaces to a width of 18 characters\n",
    "    # You don't need to know or ever use this, but I think it's cool and handy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, this thing has some basic properties, and something that sounds interesting: \"element_texts\". Each piece of metadata about an Item in Omeka is referred to as an \"element\", so this sounds promising. Since it's a list, let's look at the first item in that list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "demo_item['element_texts'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Element stuff\n",
    "\n",
    "So there are some different parts of each element text, including the element, element set, text, and what looks like a boolean flag for whether or not this text should be rendered as HTML. That's kind of complicated and annoying. Let's think about this real hard for a little bit so we don't have to think about it again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New function!\n",
    "\n",
    "Since making items isn't exactly intuitive, let's make a quick function to construct items from dictionaries. We're relying on some things specific to this site, namely the IDs of each element, and for a more general solution we'd want to do something more nuanced than hard coding those IDs into our workflow. For now, though, this is a workable solution.\n",
    "\n",
    "Also, you'll notice we're only specifying the element ID, because it turns out that's all you actually need when creating a new Item in Omeka."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_item(element_texts):\n",
    "    \"\"\"\n",
    "    Takes a dictionary with format {element_id:element_text, ...}\n",
    "    \"\"\"\n",
    "    base_item = {\n",
    "        'element_texts':[],\n",
    "        'featured': False,\n",
    "        'public': True,\n",
    "    }\n",
    "    for _id, text in element_texts.items():\n",
    "        element = {\n",
    "            'element': { 'id': int(_id) },\n",
    "            'text': text,\n",
    "            'html': True\n",
    "        }\n",
    "        base_item['element_texts'].append(element)\n",
    "    return base_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test = {\n",
    "    50: 'A Test Item',\n",
    "    41: \"The description of the test item. It might be a bit longer, which is fine since it won't be used as a page title or anything.\"\n",
    "}\n",
    "test_item = make_item(test)\n",
    "print(test_item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## POST new data\n",
    "\n",
    "Now that we have content to upload, let's take a look at how we'll do that. We're using a different method of sending data to the url, you'll notice. We're POSTing data, which usually means we're adding something new. We can still use our `params` argument, but our data is in our `json` argument.\n",
    "\n",
    "The `requests` module has this as a convenient parameter, so you don't have to turn your dictionary into a string to use it as a data payload. Since this is such a common task, `requests` has built it into this method call so we can just use the dictionary object we've created.\n",
    "\n",
    "We'll still get a response, but in this case, we'll get a representation of the item that we just created, as long as it was created successfully. We know this from the documentation, which tells us what response to expect from each kind of query we can send to the items API endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R = requests.post('http://demo.omeka.fas.harvard.edu/api/items',json=test_item, params={'key':omeka_api_key})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "R.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions in functions\n",
    "\n",
    "We can make another function to take a dictionary that represents our item in a pretty convenient way and add that directly to Omeka. We're using the function that we made to create an item within this function, so we don't have to add that functionality to this function too. We might want to keep these functions separate, in case we want to use the `make_item` function on its own for some other purpose, like creating several items and then adding them all at once.\n",
    "\n",
    "Each of the functions has a short string right after the definition, surrounded by triple quotes. This lets you have a multi-line string, and is almost always used the way we're using it here, to define a docstring for a function. This is basically a standard for the helper text you'll see with shift+tab on your functions. Try it when we use the functions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_item_to_omeka(element_texts):\n",
    "    \"\"\"Add an item to our Omeka site\"\"\"\n",
    "    item = make_item(element_texts)\n",
    "    R = requests.post('http://demo.omeka.fas.harvard.edu/api/items',json=item, params={'key':omeka_api_key})\n",
    "    return R.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_manifest(record):\n",
    "    \"\"\"Get a manifest from a HAM object record\"\"\"\n",
    "    try:\n",
    "        for see in record['seeAlso']:\n",
    "            if see['type'] == \"IIIF Manifest\":\n",
    "                manifest = requests.get(see['id']).json()\n",
    "                return manifest\n",
    "    except:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_canvas(manifest):\n",
    "    \"\"\"Get the first canvas from a IIIF manifest\"\"\"\n",
    "    try:\n",
    "        return manifest['sequences'][0]['canvases'][0]\n",
    "    except:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need another library for the next function, so we're importing it here. Normally, it's a good practice to move any new libraries you find you'll need to the top of your script or notebook, but we're breaking rules* today!\n",
    "\n",
    "**really more of best practices*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def record_to_omeka(record):\n",
    "    \"\"\"Take a Harvard Art Museum object record and add it to our Omeka site\"\"\"\n",
    "    \n",
    "    # Set up info we will add to the item\n",
    "    title = record['title']\n",
    "    description = record['description']\n",
    "    manifest = get_manifest(record)\n",
    "    canvas = get_canvas(manifest)\n",
    "    \n",
    "    # Set up element texts, using element IDs figured out from other item JSON representations\n",
    "    element_texts = {\n",
    "        '50': title,              # DC:Title\n",
    "        '41': description,        # DC:Description\n",
    "        '39': 'Your name here!',  # DC:Creator\n",
    "        '56': json.dumps(canvas)  # IIIF:Json Data\n",
    "    }\n",
    "    \n",
    "    # Add the item to Omeka\n",
    "    added_item = add_item_to_omeka(element_texts)\n",
    "    \n",
    "    # If there's a manifest, there's a thumbnail to add to enable the image viewer\n",
    "    if manifest != None:\n",
    "        \n",
    "        # get the thumbnail from the manifest as a raw file object\n",
    "        thumbnail = requests.get(manifest['thumbnail']['@id']).content\n",
    "        \n",
    "        # Set up file data\n",
    "        files = {\n",
    "            \"file\":(manifest['thumbnail']['@id'], thumbnail, \"application/octet-stream\"),\n",
    "            \"data\":(None, json.dumps({'item':{'id':added_item['id']}}))\n",
    "        }\n",
    "        \n",
    "        # Post data to file endpoint of the API\n",
    "        R = requests.post(\n",
    "            \"https://demo.omeka.fas.harvard.edu/api/files\", \n",
    "            params={\"key\":omeka_api_key}, \n",
    "            files=files\n",
    "        )\n",
    "    \n",
    "    # Return the JSON representation of the new item\n",
    "    return added_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "added_item = record_to_omeka(unknown_records[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mischief Managed!\n",
    "\n",
    "Now we have the tools at our disposal to take content from the Harvard Art Museums and put it into an Omeka site, or export it so we can analyze it somewhere else. We're ready for our big data heist!*\n",
    "\n",
    "**Not a real heist, we are using freely available data that the museum has generously made available. Please do not steal any physical art.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}