galaxy_api_histories.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interacting with histories in Galaxy API\n",
    "========================================\n",
    "\n",
    "We are going to use the [requests](http://python-requests.org/) Python library to communicate via HTTP with the Galaxy server. To start, let's define the connection parameters.\n",
    "\n",
    "**You need to insert the API key for your Galaxy server in the cell below**: \n",
    "1. Open the Galaxy server in another browser tab\n",
    "2. Click on \"User\" on the top menu, then \"Preferences\"\n",
    "3. Click on \"Manage API key\"\n",
    "4. Generate an API key if needed, then copy the alphanumeric string and paste it as the value of the `api_key` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://usegalaxy.eu/api'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "import json\n",
    "\n",
    "import requests\n",
    "from six.moves.urllib.parse import urljoin\n",
    "\n",
    "server = 'https://usegalaxy.eu/'\n",
    "api_key = ''\n",
    "base_url = urljoin(server, 'api')\n",
    "base_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now make a GET request to retrieve all histories owned by a user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"url\": \"/api/histories/effec70bec8ba12c\", \"published\": false, \"model_class\": \"History\", \"name\": \"New history\", \"purged\": false, \"annotation\": null, \"deleted\": false, \"tags\": [], \"id\": \"effec70bec8ba12c\"}, {\"url\": \"/api/histories/49e446c3d6585583\", \"published\": false, \"model_class\": \"History\", \"name\": \"Unnamed history\", \"purged\": false, \"annotation\": null, \"deleted\": false, \"tags\": [], \"id\": \"49e446c3d6585583\"}]\n",
      "[{'annotation': None,\n",
      "  'deleted': False,\n",
      "  'id': 'effec70bec8ba12c',\n",
      "  'model_class': 'History',\n",
      "  'name': 'New history',\n",
      "  'published': False,\n",
      "  'purged': False,\n",
      "  'tags': [],\n",
      "  'url': '/api/histories/effec70bec8ba12c'},\n",
      " {'annotation': None,\n",
      "  'deleted': False,\n",
      "  'id': '49e446c3d6585583',\n",
      "  'model_class': 'History',\n",
      "  'name': 'Unnamed history',\n",
      "  'published': False,\n",
      "  'purged': False,\n",
      "  'tags': [],\n",
      "  'url': '/api/histories/49e446c3d6585583'}]\n"
     ]
    }
   ],
   "source": [
    "params = {'key': api_key}\n",
    "r = requests.get(base_url + '/histories', params)\n",
    "print(r.text)\n",
    "hists = r.json()\n",
    "pprint(hists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, GET requests in Galaxy API return JSON strings, which need to be **deserialized** into Python data structures. In particular, GETting a resource collection returns a list of dictionaries.\n",
    "\n",
    "Each dictionary returned when GETting a resource collection gives basic info about a resource, e.g. for a history you have:\n",
    "- `id`: the unique **identifier** of the history, needed for all specific requests about this resource\n",
    "- `name`: the name of this history as given by the user\n",
    "- `deleted`: whether the history has been deleted\n",
    "- `url`: the relative URL to get all info about this resource.\n",
    "\n",
    "There is no readily-available filtering capability, but it's not difficult to filter histories **by name**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'annotation': None,\n",
      "  'deleted': False,\n",
      "  'id': '49e446c3d6585583',\n",
      "  'model_class': 'History',\n",
      "  'name': 'Unnamed history',\n",
      "  'published': False,\n",
      "  'purged': False,\n",
      "  'tags': [],\n",
      "  'url': '/api/histories/49e446c3d6585583'}]\n"
     ]
    }
   ],
   "source": [
    "pprint([_ for _ in hists if _['name'] == 'Unnamed history'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are interested in more **details** about a given resource, you just need to append its `id` to the previous collection request, e.g. to the get more info for a history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "effec70bec8ba12c\n",
      "{'annotation': None,\n",
      " 'contents_url': '/api/histories/effec70bec8ba12c/contents',\n",
      " 'create_time': '2015-07-02T11:04:17.100787',\n",
      " 'deleted': False,\n",
      " 'empty': False,\n",
      " 'genome_build': 'hg38',\n",
      " 'id': 'effec70bec8ba12c',\n",
      " 'importable': False,\n",
      " 'model_class': 'History',\n",
      " 'name': 'New history',\n",
      " 'published': False,\n",
      " 'purged': False,\n",
      " 'size': 54092,\n",
      " 'slug': None,\n",
      " 'state': 'ok',\n",
      " 'state_details': {'discarded': 0,\n",
      "                   'empty': 0,\n",
      "                   'error': 0,\n",
      "                   'failed_metadata': 0,\n",
      "                   'new': 0,\n",
      "                   'ok': 3,\n",
      "                   'paused': 0,\n",
      "                   'queued': 0,\n",
      "                   'running': 0,\n",
      "                   'setting_metadata': 0,\n",
      "                   'upload': 0},\n",
      " 'state_ids': {'discarded': [],\n",
      "               'empty': [],\n",
      "               'error': [],\n",
      "               'failed_metadata': [],\n",
      "               'new': [],\n",
      "               'ok': ['bbd44e69cb8906b550f62a0227a2da04',\n",
      "                      'bbd44e69cb8906b544479115d78d7a93',\n",
      "                      'bbd44e69cb8906b50e45a3912d4c471e'],\n",
      "               'paused': [],\n",
      "               'queued': [],\n",
      "               'running': [],\n",
      "               'setting_metadata': [],\n",
      "               'upload': []},\n",
      " 'tags': [],\n",
      " 'update_time': '2020-07-15T23:43:47.043706',\n",
      " 'url': '/api/histories/effec70bec8ba12c',\n",
      " 'user_id': '1c510fef372551ec',\n",
      " 'username_and_slug': None}\n"
     ]
    }
   ],
   "source": [
    "hist0_id = hists[0]['id']\n",
    "print(hist0_id)\n",
    "params = {'key': api_key}\n",
    "r = requests.get(base_url + '/histories/' + hist0_id, params)\n",
    "pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are much more entries in the returned dictionary, e.g.:\n",
    "- `create_time`\n",
    "- `size`: total disk space used by the history\n",
    "- `state_ids`: ids of history datasets for each possible state.\n",
    "\n",
    "To get the list of **datasets contained** in a history, simply append `/contents` to the previous resource request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'create_time': '2015-07-02T11:08:56.653404',\n",
      "  'dataset_id': 'bbd44e69cb8906b5ab999dca04b8ece0',\n",
      "  'deleted': False,\n",
      "  'extension': 'txt',\n",
      "  'hid': 1,\n",
      "  'history_content_type': 'dataset',\n",
      "  'history_id': 'effec70bec8ba12c',\n",
      "  'id': 'bbd44e69cb8906b550f62a0227a2da04',\n",
      "  'name': '1.txt',\n",
      "  'purged': False,\n",
      "  'state': 'ok',\n",
      "  'tags': [],\n",
      "  'type': 'file',\n",
      "  'type_id': 'dataset-bbd44e69cb8906b550f62a0227a2da04',\n",
      "  'update_time': '2015-07-02T11:09:26.632285',\n",
      "  'url': '/api/histories/effec70bec8ba12c/contents/bbd44e69cb8906b550f62a0227a2da04',\n",
      "  'visible': True},\n",
      " {'create_time': '2015-07-02T11:10:57.626465',\n",
      "  'dataset_id': 'bbd44e69cb8906b5333417f7e7c6ca91',\n",
      "  'deleted': False,\n",
      "  'extension': 'txt',\n",
      "  'hid': 2,\n",
      "  'history_content_type': 'dataset',\n",
      "  'history_id': 'effec70bec8ba12c',\n",
      "  'id': 'bbd44e69cb8906b544479115d78d7a93',\n",
      "  'name': '1.txt',\n",
      "  'purged': False,\n",
      "  'state': 'ok',\n",
      "  'tags': [],\n",
      "  'type': 'file',\n",
      "  'type_id': 'dataset-bbd44e69cb8906b544479115d78d7a93',\n",
      "  'update_time': '2015-07-02T11:11:39.923731',\n",
      "  'url': '/api/histories/effec70bec8ba12c/contents/bbd44e69cb8906b544479115d78d7a93',\n",
      "  'visible': True},\n",
      " {'create_time': '2015-07-02T15:35:38.073851',\n",
      "  'dataset_id': 'bbd44e69cb8906b522410530cb31d9b0',\n",
      "  'deleted': False,\n",
      "  'extension': 'tabular',\n",
      "  'hid': 3,\n",
      "  'history_content_type': 'dataset',\n",
      "  'history_id': 'effec70bec8ba12c',\n",
      "  'id': 'bbd44e69cb8906b50e45a3912d4c471e',\n",
      "  'name': 'Convert on data 2',\n",
      "  'purged': False,\n",
      "  'state': 'ok',\n",
      "  'tags': [],\n",
      "  'type': 'file',\n",
      "  'type_id': 'dataset-bbd44e69cb8906b50e45a3912d4c471e',\n",
      "  'update_time': '2015-07-02T15:36:13.366689',\n",
      "  'url': '/api/histories/effec70bec8ba12c/contents/bbd44e69cb8906b50e45a3912d4c471e',\n",
      "  'visible': True}]\n"
     ]
    }
   ],
   "source": [
    "params = {'key': api_key}\n",
    "r = requests.get(base_url + '/histories/' + hist0_id + '/contents', params)\n",
    "hdas = r.json()\n",
    "pprint(hdas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dictionaries returned when GETting the history content give basic info about each dataset, e.g.: `id`, `name`, `deleted`, `state`, `url`...\n",
    "\n",
    "To get the details about a specific dataset, you can use the `datasets` controller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bbd44e69cb8906b550f62a0227a2da04\n",
      "{'accessible': True,\n",
      " 'annotation': None,\n",
      " 'api_type': 'file',\n",
      " 'create_time': '2015-07-02T11:08:56.653404',\n",
      " 'created_from_basename': None,\n",
      " 'creating_job': 'faa39c69e6841f30',\n",
      " 'data_type': 'galaxy.datatypes.data.Text',\n",
      " 'dataset_id': 'bbd44e69cb8906b5ab999dca04b8ece0',\n",
      " 'deleted': False,\n",
      " 'display_apps': [],\n",
      " 'display_types': [],\n",
      " 'download_url': '/api/histories/effec70bec8ba12c/contents/bbd44e69cb8906b550f62a0227a2da04/display',\n",
      " 'extension': 'txt',\n",
      " 'file_ext': 'txt',\n",
      " 'file_size': 16,\n",
      " 'genome_build': '?',\n",
      " 'hda_ldda': 'hda',\n",
      " 'hid': 1,\n",
      " 'history_content_type': 'dataset',\n",
      " 'history_id': 'effec70bec8ba12c',\n",
      " 'id': 'bbd44e69cb8906b550f62a0227a2da04',\n",
      " 'meta_files': [],\n",
      " 'metadata_data_lines': 4,\n",
      " 'metadata_dbkey': '?',\n",
      " 'misc_blurb': '4 lines',\n",
      " 'misc_info': 'uploaded txt file',\n",
      " 'model_class': 'HistoryDatasetAssociation',\n",
      " 'name': '1.txt',\n",
      " 'peek': '<table cellspacing=\"0\" cellpadding=\"3\"><tr><td>1 '\n",
      "         'a</td></tr><tr><td>2 b</td></tr><tr><td>3 c</td></tr><tr><td>4 '\n",
      "         'd</td></tr></table>',\n",
      " 'purged': False,\n",
      " 'rerunnable': False,\n",
      " 'resubmitted': False,\n",
      " 'state': 'ok',\n",
      " 'tags': [],\n",
      " 'type': 'file',\n",
      " 'type_id': 'dataset-bbd44e69cb8906b550f62a0227a2da04',\n",
      " 'update_time': '2015-07-02T11:09:26.632285',\n",
      " 'url': '/api/histories/effec70bec8ba12c/contents/bbd44e69cb8906b550f62a0227a2da04',\n",
      " 'uuid': '0c728ee0-5d3c-4076-9dc5-02c45ff4c1c7',\n",
      " 'validated_state': 'unknown',\n",
      " 'validated_state_message': None,\n",
      " 'visible': True,\n",
      " 'visualizations': [{'description': 'Manually edit text',\n",
      "                     'embeddable': False,\n",
      "                     'entry_point': {'attr': {},\n",
      "                                     'file': 'editor.mako',\n",
      "                                     'type': 'mako'},\n",
      "                     'groups': None,\n",
      "                     'href': '/plugins/visualizations/editor/show',\n",
      "                     'html': 'Editor',\n",
      "                     'logo': None,\n",
      "                     'name': 'editor',\n",
      "                     'settings': None,\n",
      "                     'specs': None,\n",
      "                     'target': 'galaxy_main',\n",
      "                     'title': None},\n",
      "                    {'description': 'Cytoscape-based graph visualization '\n",
      "                                    'plugin.',\n",
      "                     'embeddable': False,\n",
      "                     'entry_point': {'attr': {},\n",
      "                                     'file': 'graphviz.mako',\n",
      "                                     'type': 'mako'},\n",
      "                     'groups': None,\n",
      "                     'href': '/plugins/visualizations/graphviz/show',\n",
      "                     'html': 'Graph Visualization',\n",
      "                     'logo': None,\n",
      "                     'name': 'graphviz',\n",
      "                     'settings': None,\n",
      "                     'specs': None,\n",
      "                     'target': 'galaxy_main',\n",
      "                     'title': None},\n",
      "                    {'description': None,\n",
      "                     'embeddable': False,\n",
      "                     'entry_point': {'attr': {},\n",
      "                                     'file': 'jupyter.mako',\n",
      "                                     'type': 'mako'},\n",
      "                     'groups': None,\n",
      "                     'href': '/plugins/interactive_environments/jupyter/show',\n",
      "                     'html': 'Jupyter',\n",
      "                     'logo': None,\n",
      "                     'name': 'jupyter',\n",
      "                     'settings': None,\n",
      "                     'specs': None,\n",
      "                     'target': 'galaxy_main',\n",
      "                     'title': None},\n",
      "                    {'description': 'The MSA viewer is a modular, reusable '\n",
      "                                    'component to visualize large MSAs '\n",
      "                                    'interactively on the web.',\n",
      "                     'embeddable': False,\n",
      "                     'entry_point': {'attr': {'src': 'script.js'},\n",
      "                                     'file': None,\n",
      "                                     'type': 'chart'},\n",
      "                     'groups': None,\n",
      "                     'href': '/plugins/visualizations/msa/show',\n",
      "                     'html': 'Multiple Sequence Alignment',\n",
      "                     'logo': './static/plugins/visualizations/msa/static/logo.png',\n",
      "                     'name': 'msa',\n",
      "                     'settings': [{'help': 'Do you want to display a bar '\n",
      "                                           'diagram indicating sequence '\n",
      "                                           'conservation?',\n",
      "                                   'label': 'Show conservation',\n",
      "                                   'name': 'conserv',\n",
      "                                   'type': 'boolean',\n",
      "                                   'value': 'true'},\n",
      "                                  {'help': 'Do you want to display the '\n",
      "                                           'overview box below the sequence '\n",
      "                                           'alignments?',\n",
      "                                   'label': 'Show overview box',\n",
      "                                   'name': 'overviewbox',\n",
      "                                   'type': 'boolean',\n",
      "                                   'value': 'true'},\n",
      "                                  {'help': 'Do you want to show a menu for '\n",
      "                                           'interactive configuration?',\n",
      "                                   'label': 'Show interactive menu',\n",
      "                                   'name': 'menu',\n",
      "                                   'type': 'boolean',\n",
      "                                   'value': 'true'}],\n",
      "                     'specs': None,\n",
      "                     'target': 'galaxy_main',\n",
      "                     'title': None}]}\n"
     ]
    }
   ],
   "source": [
    "hda0_id = hdas[0]['id']\n",
    "print(hda0_id)\n",
    "params = {'key': api_key}\n",
    "r = requests.get(base_url + '/datasets/' + hda0_id, params)\n",
    "pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the interesting additional dictionary entries are:\n",
    "- `create_time`\n",
    "- `creating job`: id of the job which created this dataset\n",
    "- `download_url`: URL to download the dataset\n",
    "- `file_ext`: the Galaxy data type of this dataset\n",
    "- `file_size`\n",
    "- `genome_build`: the genome build (dbkey) associated to this dataset.\n",
    "\n",
    "**New resources** are created with POST requests. The uploaded **data needs to be serialized** in a JSON string. For example, to create a new history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'annotation': None,\n",
      " 'contents_url': '/api/histories/1d45bdeb654111cb/contents',\n",
      " 'create_time': '2020-07-15T23:23:00.692409',\n",
      " 'deleted': False,\n",
      " 'empty': True,\n",
      " 'genome_build': None,\n",
      " 'id': '1d45bdeb654111cb',\n",
      " 'importable': False,\n",
      " 'model_class': 'History',\n",
      " 'name': 'New history',\n",
      " 'published': False,\n",
      " 'purged': False,\n",
      " 'size': 0,\n",
      " 'slug': None,\n",
      " 'state': 'new',\n",
      " 'state_details': {'discarded': 0,\n",
      "                   'empty': 0,\n",
      "                   'error': 0,\n",
      "                   'failed_metadata': 0,\n",
      "                   'new': 0,\n",
      "                   'ok': 0,\n",
      "                   'paused': 0,\n",
      "                   'queued': 0,\n",
      "                   'running': 0,\n",
      "                   'setting_metadata': 0,\n",
      "                   'upload': 0},\n",
      " 'state_ids': {'discarded': [],\n",
      "               'empty': [],\n",
      "               'error': [],\n",
      "               'failed_metadata': [],\n",
      "               'new': [],\n",
      "               'ok': [],\n",
      "               'paused': [],\n",
      "               'queued': [],\n",
      "               'running': [],\n",
      "               'setting_metadata': [],\n",
      "               'upload': []},\n",
      " 'tags': [],\n",
      " 'update_time': '2020-07-15T23:23:00.692469',\n",
      " 'url': '/api/histories/1d45bdeb654111cb',\n",
      " 'user_id': '1c510fef372551ec',\n",
      " 'username_and_slug': None}\n"
     ]
    }
   ],
   "source": [
    "params = {'key': api_key}\n",
    "data = {'name': 'New history'}\n",
    "r = requests.post(base_url + '/histories', data=json.dumps(data), params=params, headers={'Content-Type': 'application/json'})\n",
    "new_hist = r.json()\n",
    "pprint(new_hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The return value of a POST request is a dictionary with detailed info about the created resource.\n",
    "\n",
    "To **update** a resource, make a PUT request, e.g. to change the history name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "{'annotation': None,\n",
      " 'contents_url': '/api/histories/1d45bdeb654111cb/contents',\n",
      " 'create_time': '2020-07-15T23:23:00.692409',\n",
      " 'deleted': False,\n",
      " 'empty': True,\n",
      " 'genome_build': None,\n",
      " 'id': '1d45bdeb654111cb',\n",
      " 'importable': False,\n",
      " 'model_class': 'History',\n",
      " 'name': 'Updated history',\n",
      " 'published': False,\n",
      " 'purged': False,\n",
      " 'size': 0,\n",
      " 'slug': None,\n",
      " 'state': 'new',\n",
      " 'state_details': {'discarded': 0,\n",
      "                   'empty': 0,\n",
      "                   'error': 0,\n",
      "                   'failed_metadata': 0,\n",
      "                   'new': 0,\n",
      "                   'ok': 0,\n",
      "                   'paused': 0,\n",
      "                   'queued': 0,\n",
      "                   'running': 0,\n",
      "                   'setting_metadata': 0,\n",
      "                   'upload': 0},\n",
      " 'state_ids': {'discarded': [],\n",
      "               'empty': [],\n",
      "               'error': [],\n",
      "               'failed_metadata': [],\n",
      "               'new': [],\n",
      "               'ok': [],\n",
      "               'paused': [],\n",
      "               'queued': [],\n",
      "               'running': [],\n",
      "               'setting_metadata': [],\n",
      "               'upload': []},\n",
      " 'tags': [],\n",
      " 'update_time': '2020-07-15T23:23:24.332031',\n",
      " 'url': '/api/histories/1d45bdeb654111cb',\n",
      " 'user_id': '1c510fef372551ec',\n",
      " 'username_and_slug': None}\n"
     ]
    }
   ],
   "source": [
    "params = {'key': api_key}\n",
    "data = {'name': 'Updated history'}\n",
    "r = requests.put(base_url + '/histories/' + new_hist['id'], json.dumps(data), params=params, headers={'Content-Type': 'application/json'})\n",
    "print(r.status_code)\n",
    "pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The return value of a PUT request is usually a dictionary with detailed info about the updated resource.\n",
    "\n",
    "Finally to **delete** a resource, make a DELETE request, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "params = {'key': api_key}\n",
    "r = requests.delete(base_url + '/histories/' + new_hist['id'], params=params)\n",
    "print(r.status_code)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}