TCIA_Jupyter_Learning_Lab_2023.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "7YGlC_ajggC8",
        "nj6P7YCmll4S",
        "aZvpm8BiUAnx",
        "elX2coKjUK4R",
        "rOVuIo2Xqozd",
        "BJEBAyTyqcU5",
        "YMIPh4vzqfN5"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_Jupyter_Learning_Lab_2023.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Jupyter_Learning_Lab_2023.ipynb)"
      ],
      "metadata": {
        "id": "DUmcjg7W2Pz8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Course Description\n",
        "\n",
        "Access to large, high quality data is essential for researchers to understand disease and precision medicine pathways, especially in cancer. However HIPAA constraints make sharing medical images outside an individual institution a complex process. [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) TCIA is a public service funded by the National Cancer Institute which addresses this challenge by providing hosting and de-identification services to take major burdens of data sharing off researchers.\n",
        "\n",
        "TCIA has published over 200 unique data collections containing more than 60 million images. Recognizing that images alone are not enough to conduct meaningful research, most collections are linked to rich supporting data including patient outcomes, treatment information, genomic / proteomic analyses, and expert image analyses (segmentations, annotations, and radiomic / radiogenomic features). **In this course we will address a variety of use cases for identifying TCIA datasets of interest and downloading them via Jupyter Notebooks.**\n",
        "\n",
        "# Learning Objectives\n",
        "\n",
        "* Learn how TCIA makes data sharing easier for researchers, and hear a summary of existing datsets that are freely available for download\n",
        "* Practice utilizing TCIA for data exploration, cohort definition, and downloading of data\n",
        "* Learn how to access public and restricted access datasets using TCIA's REST APIs and other command line tools via Google Colab"
      ],
      "metadata": {
        "id": "KmXfYFZtja2F"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AruUGe3lmjkh"
      },
      "source": [
        "# 1 Learn about Available Collections on the TCIA Website\n",
        "\n",
        "[Browsing Collections](https://www.cancerimagingarchive.net/collections) and viewing [Analysis Results](https://www.cancerimagingarchive.net/tcia-analysis-results/) of datasets on TCIA are the easiest ways to become familiar with what is available. These pages will help you quickly identify datasets of interest, find valuable supporting data that are not available via our APIs (e.g. clinical spreadsheets and non-DICOM segmentation data), and answer the most common questions you might have about the datasets.  "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2 Downloading with the NBIA Data Retriever\n",
        "\n",
        "TCIA uses software called NBIA to manage its DICOM data.  One way to download TCIA data is to install the NBIA Data Retriever.  This tool provides a number of useful features such as auto-retry if there are any problems, saving data in an organized hierarchy on your hard drive (Collection > Patient > Study > Series > Images), and providing a CSV file containing key DICOM metadata about the images you've downloaded."
      ],
      "metadata": {
        "id": "eRvQa63-kJIY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.1 Install the NBIA Data Retriever\n",
        "There are versions of this tool for Windows, Mac and Linux.  If you're working from a system with a GUI you can follow the [instructions](https://wiki.cancerimagingarchive.net/display/NBIA/Downloading+TCIA+Images) to install Data Retriever on your computer.\n",
        "\n",
        "There is also a [command-line version of the NBIA Data Retriever](https://wiki.cancerimagingarchive.net/x/2QKPBQ) which can be installed via the steps below if you're running this notebook in a **Linux** environment.  "
      ],
      "metadata": {
        "id": "RtLE_18NoaJ8"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B714bOkDk8kd"
      },
      "source": [
        "# Install NBIA Data Retriever CLI software for downloading images later in this notebook.\n",
        "\n",
        "!mkdir /usr/share/desktop-directories/\n",
        "!wget -P /content/NBIA-Data-Retriever https://cbiit-download.nci.nih.gov/nbia/releases/ForTCIA/NBIADataRetriever_4.4/nbia-data-retriever-4.4.2.deb\n",
        "!dpkg -i /content/NBIA-Data-Retriever/nbia-data-retriever-4.4.2.deb\n",
        "\n",
        "# NOTE: If you're working on a Linux OS that uses RPM packages, you can change the lines above to use\n",
        "#       https://cbiit-download.nci.nih.gov//nbia/releases/ForTCIA/NBIADataRetriever_4.4/NBIADataRetriever-4.4-2.x86_64.rpm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2uMbL1nwIJ5f"
      },
      "source": [
        "### 2.2 Download a Manifest File\n",
        "The NBIA Data Retriever software works by ingesting a \"manifest\" file that contains the DICOM Series Instance UIDs of the scans you'd like to download. How do you find these manifest files?  \n",
        "\n",
        "Starting from the TCIA homepage, choose [Access the data](https://www.cancerimagingarchive.net/access-data) and then click on [Browse collections](https://www.cancerimagingarchive.net/collections).  After reviewing the table of datasets, let's say you decided you were interested in the [RIDER Breast MRI](https://doi.org/10.7937/K9/TCIA.2015.H1SXNUXL) collection.  We can find the URL of the manifest to download the full collection by looking at the blue \"Download\" button on that page.  You can download the file to your computer and upload it to Colab manually, or you can import it directly to Colab by using **wget** with the URL of the manifest file.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://wiki.cancerimagingarchive.net/download/attachments/22512757/doiJNLP-Fo0H1NtD.tcia"
      ],
      "metadata": {
        "id": "sb5HEOgC_lYm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you look at the file you'll see some configuration information at the top, followed by a list of Series Instance UIDs that are part of the dataset.  \n",
        "\n",
        "Let's edit the manifest file to only include the first 3 UIDs in the manifest so that we can demonstrate the download process more quickly.\n"
      ],
      "metadata": {
        "id": "RhG_JOUZ-CpF"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gsorr-mi7--3"
      },
      "outputs": [],
      "source": [
        "with open('doiJNLP-Fo0H1NtD.tcia','r') as firstfile, open('RIDER-Breast-MRI-Sample.tcia','a') as secondfile:\n",
        "    count = 0\n",
        "    for line in firstfile:\n",
        "        # append content to second file\n",
        "        secondfile.write(line)\n",
        "        # Stop after header and first 3 series UIDs\n",
        "        count += 1;\n",
        "        if count == 9:\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IlPLgxkBZPMS"
      },
      "source": [
        "### 2.3 Open the Manifest File with the NBIA Data Retriever\n",
        "Next, let's open the sample manifest file with the NBIA Data Retriever to download the actual DICOM data.  To do this, we'll call the command to launch Data Retriever, specify the **--cli** flag to indicate we want to run this via command line (not with a GUI).  We also need to specify the path to the manifest file we want to open and then use **-d** to specify the path where we want to save the data.\n",
        "\n",
        "**<font color='red'>After running the following command, click in the output cell, type \"y,\" and press Enter to agree with the TCIA Data Usage Policy and start the download.</font>**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a4lklpk4Xwpc"
      },
      "outputs": [],
      "source": [
        "# download the data using NBIA Data Retriever\n",
        "\n",
        "!/opt/nbia-data-retriever/nbia-data-retriever --cli '/content/RIDER-Breast-MRI-Sample.tcia' -d /content/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgESwVXSawv_"
      },
      "source": [
        "### 2.4 NBIA Data Retriever Conclusion\n",
        "You should now find that the data have been saved to your machine in a well-organized hierarchy with some useful metadata in the accompanying CSV file and a license file detailing how it can be used.  Take a look at the data before moving on.\n",
        "\n",
        "A few other notes:\n",
        "* With the release of v4.4.1 the CLI Data Retriever now supports both \"Descriptive\" and \"Classic\" organization of the data.  In short, Descriptive naming provides more human-readable directory names whereas Classic names everything by machine-readable unique identifiers.  If you prefer machine-readable directory names simply add the **-cd** parameter to your download command.\n",
        "* In some cases, you must specifically request access to [Collections](https://www.cancerimagingarchive.net/collections/) before you can download them.  Information about how to do this can be found on the homepage for the Collection(s) you're interested in, but will always require that you first [create a TCIA user account](https://wiki.cancerimagingarchive.net/x/xgHDAg).  Once you've created an account and obtained permission to the restricted data you want to download, you can use your login/password to create the **credentials.txt** file that NBIA Data Retriever uses to verify your permissions.  The path to the credential file is specified using the **-l** parameter.\n",
        "\n",
        "You can find examples showing how to do this at [TCIA_Linux_Data_Retriever_App.ipynb](https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Linux_Data_Retriever_App.ipynb).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmeqJoR5k9z0"
      },
      "source": [
        "# 3 Downloading with the REST API\n",
        "The NBIA REST APIs are provided for the search and download functions used in the TCIA radiology portal and allow access to both public and limited access collections.\n",
        "1. The [NBIA Search REST APIs](https://wiki.cancerimagingarchive.net/x/fILTB) allow you to perform basic queries and download data from **public** collections. These APIs do not require a TCIA account.\n",
        "2. The [NBIA Search with Authentication REST APIs](https://wiki.cancerimagingarchive.net/x/X4ATBg) allow you to perform basic queries and download data from **public and limited-access** collections. These APIs require a TCIA account to create authentication tokens.\n",
        "3. The [NBIA Advanced REST APIs](https://wiki.cancerimagingarchive.net/x/YoATBg) also allow access to **public and limited-access** collections, but provide query endpoints mostly geared towards developers seeking to integrate searching and downloading TCIA data into web and desktop applications. This API requires a TCIA account to create authentication tokens.\n",
        "\n",
        "Feel free to reference this documentation if you run into issues or need clarification about anything related to the API.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqulqBEDMSS6"
      },
      "source": [
        "## 3.1 Setup\n",
        "\n",
        "Rather than working directly with the API we will be relying heavily on [**tcia_utils**](https://pypi.org/project/tcia-utils/), which contains a variety of useful functions for accessing TCIA via Jupyter/Python. Docstrings detailing how to use each function are provided in the code itself, and an extensive library of tutorial notebooks can be found at https://github.com/kirbyju/TCIA_Notebooks.\n",
        "\n",
        "By default, most functions from tcia_utils return results in JSON.  However, you can use **format = \"df\"** to return the results as a dataframe, or **format = \"csv\"** to save a CSV file in addition to returning a dataframe.\n",
        "\n",
        "Nearly all functions allow you to specify **api_url** as a query parameter.  This is generally only needed for accessing restricted collections, but is also used if you want to access the [National Lung Screening Trial (NLST)](https://doi.org/10.7937/TCIA.HMQ8-J677) collection.  NLST lives on a separate server with a different URL from the rest of our datasets due to its size (>26,000 patients!).  \n",
        "\n",
        "We'll incorporate examples using these different output formats and API URLs below.  But first, let's install the tools we need and import them so they're ready for use.\n",
        "\n",
        "**Note:** The setup steps have been updated slightly since the creation of the explainer video at https://vimeo.com/893497931 to make them more generalizable and efficient but everything else afterwards should work the same.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U7PaOf9vG77X"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# install tcia utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p3zEqnxi9rk2",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "from tcia_utils import nbia\n",
        "\n",
        "# set logging level to INFO in Google Colab (not necessary in Jupyter)\n",
        "if 'google.colab' in sys.modules:\n",
        "  import logging\n",
        "\n",
        "  for handler in logging.root.handlers[:]:\n",
        "      logging.root.removeHandler(handler)\n",
        "\n",
        "  # Set handler with level = info\n",
        "  logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',\n",
        "                      level=logging.INFO)\n",
        "\n",
        "  print(\"Google Colab Logging set to INFO\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iao1mstCf8Vl"
      },
      "source": [
        "## 3.2 Download Examples\n",
        "\n",
        "In this section we'll cover downloading data via the REST API for the following use cases:\n",
        "\n",
        "1.   Download a full TCIA collection and preview a series\n",
        "2.   Download custom results of an API query\n",
        "3.   Download custom results of an API query\n",
        "4.   Download a \"[Shared Cart](https://wiki.cancerimagingarchive.net/display/NBIA/TCIA+Radiology+Portal+User+Guide#TCIARadiologyPortalUserGuide-SharingDatainYourCart)\" that was created via https://nbia.cancerimagingarchive.net/\n",
        "5.   Download data from a TCIA manifest file\n",
        "6.   Download images with DICOM segmentations\n",
        "7.   Download images based on related clinical data\n",
        "8.   Download data from a restricted collection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwj_OZjphcWP"
      },
      "source": [
        "### 3.2.1 Download a full collection and preview a series\n",
        "\n",
        "As mentioned earlier, you can [Browse Collections](https://www.cancerimagingarchive.net/collections) on our website to figure out what you might want to download, but you can also get a list of available collections via the API as shown below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "II1o1709zq1d"
      },
      "outputs": [],
      "source": [
        "# get list of available collections as JSON\n",
        "nbia.getCollections()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jbmm5DIU4Exy"
      },
      "source": [
        "Let's say that we're interested in downloading the entire **Soft-tissue-Sarcoma** collection.  First we need to get a list of all Series Instance UIDs in that collection.  We can use **nbia.getSeries()** to save the JSON metadata about all series (scans) in this collection to a variable called **data**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eVnVh_yv7Av3"
      },
      "outputs": [],
      "source": [
        "data = nbia.getSeries(collection = \"Soft-tissue-Sarcoma\")\n",
        "print(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfkCWxiK8nQz"
      },
      "source": [
        "Then we can pass that **data** variable to the our download function.  We'll leverage the **number** parameter here to just grab the first scan as a test.  You can remove this parameter if you want to download the full collection."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2XVUkw5vgMZX"
      },
      "outputs": [],
      "source": [
        "nbia.downloadSeries(data, number = 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before moving on, take a second to review how the data are saved.  You'll note that by default the data are stored to a directory called **tciaDownload** with each directory named after the **Series Instance UID**.  While these UIDs are important to uniquely identify each scan, it can be difficult to figure out what is what unless you have additional metadata.  \n",
        "\n",
        "In order to obtain such metadata, you can set the **format** parameter to **df** to return a dataframe containing the metadata for the files you've downloaded.  Setting **format = \"csv\"** will save a spreadsheet in addition to returning a dataframe."
      ],
      "metadata": {
        "id": "UYQDcnmj2VQ_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries(data, number = 1, format = \"df\")"
      ],
      "metadata": {
        "id": "GILRfvGWbZqy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**tcia_utils** also provides a function called **viewSeries()** we can use to do a quick preview of the images here in the notebook.  The function assumes \"tciaDownload/**seriesUid**/\" as path if seriesUid is provided since this is where downloadSeries() saves data.  However you can also open a series from a custom path using the **path** parameter."
      ],
      "metadata": {
        "id": "0GdGyJ59hPJA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# example using just the series UID\n",
        "nbia.viewSeries(\"1.3.6.1.4.1.14519.5.2.1.5168.1900.104193299251798317056218297018\")\n",
        "\n",
        "# alternate example specifying a custom path\n",
        "# nbia.viewSeries(path = \"/content/tciaDownload/1.3.6.1.4.1.14519.5.2.1.5168.1900.104193299251798317056218297018\")"
      ],
      "metadata": {
        "id": "jY6NmTeNhWz3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7Pw0jvy4HO8"
      },
      "source": [
        "### 3.2.2 Download custom API query\n",
        "For this use case, let's assume that you've recently trained a model to segment tumors in breast MRIs and are looking for data to test it with.  First, let's use **getBodyPartCounts()**, which returns the number of subjects with a given value in the Body Part Examined DICOM tag."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getBodyPartCounts(modality = \"MR\")"
      ],
      "metadata": {
        "id": "4wec_DUM4VZr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that the data are returned in JSON format by default, and it appears we have ~3,000 subjects with breast MR images.  \n",
        "\n",
        "Next, let's use **getManufacturerCounts()** to see what kind of MRI scanners were used to acquire the images for these breast cancer subjects.  This time we'll also use the **format = \"df\"** parameter to make the result a little easier to read."
      ],
      "metadata": {
        "id": "hcLXCr9I4i6k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getManufacturerCounts(modality = \"MR\", bodyPart = \"BREAST\", format = \"df\")"
      ],
      "metadata": {
        "id": "LO0_3F834_oo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ok, let's say that we'd like to only focus on the MR scans acquired on GE scanners to see if our model works well on this kind of data.  We can use **getSeries()** to create a detailed inventory of these for our review before we download the data."
      ],
      "metadata": {
        "id": "ow07ay6h5j75"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Y-uSUCBE_bJ"
      },
      "outputs": [],
      "source": [
        "# getSeries with query parameters\n",
        "data = nbia.getSeries(modality = \"MR\",\n",
        "                      bodyPart = \"BREAST\",\n",
        "                      manufacturer = \"GE MEDICAL SYSTEMS\",\n",
        "                      format = \"df\")\n",
        "\n",
        "display(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mzDBsCavGuHR"
      },
      "source": [
        "Looks like we've got over 16,000 scans in our results.  As you can see from the **Series Description** column, there are a variety of different types of scans that match this criteria.  \n",
        "\n",
        "Let's say that the tool or model you're developing is aimed specifically at analyzing T2 MR data, so let's filter our dataframe to only keep the scans that mention \"T2\" in the **SeriesDescription** or **ProtocolName** columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uyaHU_f2aSSJ"
      },
      "outputs": [],
      "source": [
        "# convert the columns to lowercase to catch both upper/lower case with filter for 't2'\n",
        "t2_data = data[(data['ProtocolName'].str.lower().str.contains('t2')) |\n",
        "               (data['SeriesDescription'].str.lower().str.contains('t2'))].reset_index(drop=True)\n",
        "\n",
        "\n",
        "display(t2_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_Dk04k7c0kB"
      },
      "source": [
        "Now we can provide our **t2_data** dataframe as input to **downloadSeries()** using **input_type = \"df\"**.  By setting this as the **input_type** our download function will look in the SeriesInstanceUID column to determine which series to download."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xVIo_onkfMBn"
      },
      "outputs": [],
      "source": [
        "# download the selected series_uids\n",
        "nbia.downloadSeries(t2_data, input_type = \"df\", number = 3, format = \"df\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JEKzWV7a7Q4M"
      },
      "source": [
        "### 3.2.3 Download custom NLST API query\n",
        "Let's show a simplified example where we look for a specific modality and manufacturer within the [National Lung Screening Trial (NLST) Collection](https://doi.org/10.7937/TCIA.HMQ8-J677).  Remember that we have to set **api_url = \"nlst\"** in our functions for this to work since this data lives on a different server, but otherwise the steps are basically the same.  \n",
        "\n",
        "Let's start by checking what modalities are available in the collection."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getModalityCounts(collection = \"NLST\", format = \"df\", api_url = \"nlst\")"
      ],
      "metadata": {
        "id": "_UAVfVD-AiG7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Looks like CT is the only modality for this collection.  Let's check the manufacturers."
      ],
      "metadata": {
        "id": "njnndwF8AmIb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getManufacturerCounts(collection = \"NLST\", format = \"df\", api_url = \"nlst\")"
      ],
      "metadata": {
        "id": "5XOTJ4HZAL7s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's say we're interested in Philips scanners this time and just want to know how many scans we'll be downloading rather than doing additional filtering."
      ],
      "metadata": {
        "id": "pHpLviUOAb21"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zGlhMzna7wgS"
      },
      "outputs": [],
      "source": [
        "# getSeries with query parameters\n",
        "data = nbia.getSeries(collection = \"NLST\",\n",
        "               modality = \"CT\",\n",
        "               manufacturer = \"Philips\",\n",
        "               api_url = \"nlst\")\n",
        "\n",
        "print(len(data), 'Series returned')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Nearly 14,000 scans!  Looks like we'll have plenty of data for training this time.  Here's the final command to download the data.  We don't need to specify an **input_type** this time since we are passing the default JSON data that we stored with **getSeries()**."
      ],
      "metadata": {
        "id": "Ccon9EpzBISw"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D58Jjiqp8UVE"
      },
      "outputs": [],
      "source": [
        "# feed series_data to our downloadSampleSeries function\n",
        "df = nbia.downloadSeries(data, number = 1, api_url = \"nlst\", format = \"df\")\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AJw430rwTkJ0"
      },
      "source": [
        "### 3.2.4 Download a \"shared cart\"\n",
        "If you're collaborating with someone who prefers to use https://nbia.cancerimagingarchive.net to define their cohort rather than the API, they can create a \"[Shared Cart](https://wiki.cancerimagingarchive.net/display/NBIA/TCIA+Radiology+Portal+User+Guide#TCIARadiologyPortalUserGuide-SharingDatainYourCart)\" which includes a specific set of scans they'd like to share with others.\n",
        "\n",
        "After creating a Shared Cart the user receives a URL that looks like https://nbia.cancerimagingarchive.net/nbia-search/?saved-cart=nbia-49121659384603347 which can be shared with others to let them easily download the same data.  Try clicking the link to see what this looks like on the TCIA website.  Then use the code below to see how you can use the cart name at the end of the URL to download the related scans via the API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QIaBcHEO0DSt"
      },
      "outputs": [],
      "source": [
        "# getSharedCart metadata\n",
        "data = nbia.getSharedCart(name = \"nbia-49121659384603347\")\n",
        "print(len(data), 'Series returned')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ga_N66rMWvK"
      },
      "source": [
        "We'll skip the use of the **number** parameter this time since the full cart is only 4 series.  Let's also try **format = \"csv\"** to save a spreadsheet of the metadata in addition to returning a dataframe.  Feel free to open the resulting CSV file if you like.  It will look identical to the dataframe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XZnECiCh0DSt"
      },
      "outputs": [],
      "source": [
        "# feed series_data to our downloadSampleSeries function\n",
        "df = nbia.downloadSeries(data, format = \"csv\")\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "60vpsOC2S6ff"
      },
      "source": [
        "### 3.2.5 Download data from a TCIA manifest file\n",
        "\n",
        "When working with manifest files you can install the NBIA Data Retriever to open the manifest and download the data as discussed earlier in this notebook.  However, there may be cases where you don't have administrative rights to install software or prefer using the REST API to download an existing TCIA manifest file.  \n",
        "\n",
        "In order to demonstrate this use case, let's assume that after you [Browse Collections](https://www.cancerimagingarchive.net/collections) you are interested in the [RIDER Breast MRI](https://doi.org/10.7937/K9/TCIA.2015.H1SXNUXL) collection.  We can find the URL of the manifest to download the full collection by looking at the blue \"Download\" button on that page.  After copying and pasting that URL into the code below we can download the manifest and save it.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pAtUWEcSHiO5"
      },
      "outputs": [],
      "source": [
        "# download manifest file from RIDER Breast MRI page\n",
        "manifest = requests.get(\"https://wiki.cancerimagingarchive.net/download/attachments/22512757/doiJNLP-Fo0H1NtD.tcia?version=1&modificationDate=1534787017928&api=v2\")\n",
        "with open('RIDER_Breast_MRI.tcia', 'wb') as f:\n",
        "    f.write(manifest.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LHbh-qSHDhs3"
      },
      "source": [
        "If you open this manifest file in a text editor you'll notice that it contains several lines of download parameters that precede a list of Series Instance UIDs to download.  If we set **input_type = \"manifest\"** when calling **downloadSeries()** we can provide the path/filename to **downloadSeries()** rather than the JSON or dataframes that we have shown in other examples.  This will automatically extract the UIDs from the file and download them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tmx0Ej70WKJ4"
      },
      "outputs": [],
      "source": [
        "df = nbia.downloadSeries(\"RIDER_Breast_MRI.tcia\", input_type = \"manifest\", number = 3, format = \"df\")\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.2.6 Download images with DICOM segmentations\n",
        "Here we'll walk through some steps to identify an example DICOM segmentation file, find the corresponding reference series and visualize them together inside the notebook.  There are two DICOM modalities that are commonly used for storing segmentation data.  These are **SEG** and **RTSTRUCT**.\n",
        "\n",
        "The best way to get a broad sense of what kinds of DICOM segmentation data are available in TCIA is to review the [Browse Collections](https://www.cancerimagingarchive.net/collections) and [Browse Analysis Results](https://www.cancerimagingarchive.net/tcia-analysis-results/) pages.  You can use the filter box on both pages to limit the results in the table to **SEG** or **RTSTRUCT** on the collections page, or you could filter for **Segmentation** on the Analysis Results page.  \n",
        "\n",
        "It's worth noting that that vast majority of the Analysis Results datasets contain segmentations so that won't reduce the list by very much.  Further, we often receive segmentations in other formats (e.g. NIfTI, NRRD) which are not discoverable via the NBIA API since this system only handles DICOM data.  However, that makes the [Browse Analysis Results](https://www.cancerimagingarchive.net/tcia-analysis-results/) page a great place to look for these other kinds of segmentations!\n",
        "\n",
        "In any case, let's say that we are a researcher interested in kidney cancer so we filtered for **kidney** and **SEG** on the [Browse Collections](https://www.cancerimagingarchive.net/collections) page and discovered the [C4KC-KiTS](https://doi.org/10.7937/TCIA.2019.IX49E8NX) collection.  This dataset contains CT scans and segmentations from subjects from the training set of the [2019 Kidney and Kidney Tumor Segmentation Challenge (KiTS19)](https://kits19.grand-challenge.org/) in DICOM SEG format.\n",
        "\n",
        "Let's start by getting an inventory of all scans in the collection using **nbia.getSeries()** and then taking a look at the first patient's data."
      ],
      "metadata": {
        "id": "UINwQgzAiJcF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = nbia.getSeries(collection = \"C4KC-KiTS\", format = \"df\")\n",
        "sorted = df.sort_values([\"PatientID\", \"SeriesDescription\"])\n",
        "sorted.head(4)"
      ],
      "metadata": {
        "id": "NNfqtMZUE4xE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we can see that patient KiTS-00000 has 3 CT series and one SEG series.  How do we know which one of the CTs goes with the SEG?  \n",
        "\n",
        "The vast majority of SEG/RTSTRUCT data in TCIA has the Reference Series UID tag populated, which will tell you the Series Instance UID of the images that were used to create the segmentation.  However, in some rare cases the data was submitted to us without this information present.  We encourage all future submitters to be sure to populate this tag to make it easier on others to figure out the relationship between segmentations and their images!\n",
        "\n",
        "In any case, let's see if this C4KC-KiTS collection has this information.  First, we'll need to save the SEG Series Instance UID to a variable.\n"
      ],
      "metadata": {
        "id": "JDtkd13_Hr7i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "segSeries = sorted.loc[df['Modality'] == 'SEG', 'SeriesInstanceUID'].iloc[0]\n",
        "\n",
        "print(segSeries)"
      ],
      "metadata": {
        "id": "WFEOLIYRs33G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can pass that UID to **nbia.getSegRefSeries()** to try and determine the Reference Series Instance UID of the CT scan that goes with the segmentation."
      ],
      "metadata": {
        "id": "8NVZAb3qu_X4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "refSeries = nbia.getSegRefSeries(segSeries)\n",
        "\n",
        "print(refSeries)"
      ],
      "metadata": {
        "id": "jogkmh9kvMoW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hurray! The submitter populated the Referenced Series UID tag in their SEG data.  Now let's download these two series.  This time we'll set **input_type = \"list\"** so that we can send a simple Python list of UIDs to the function to be downloaded."
      ],
      "metadata": {
        "id": "KkMkyHoYwC2F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries([refSeries, segSeries], input_type= \"list\", format = \"df\")"
      ],
      "metadata": {
        "id": "1P-s8sZBwIIz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can look at the images and segmentation together with **viewSeriesAnnotation()**.  \n",
        "\n",
        "You can move the slider to flip through the images and toggle the segmentation layer on/off.  Once the slider is selected, sometimes it's easier to move between images using the left/right arrow keys on your keyboard than to use your mouse.\n",
        "\n"
      ],
      "metadata": {
        "id": "6HU0fGOwvxxO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.viewSeriesAnnotation(seriesUid = refSeries, annotationUid = segSeries)"
      ],
      "metadata": {
        "id": "gZb33aCNv24K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This function is only meant to be a quick and dirty way to preview the data.  There are many edge cases in which the segmentation data will not load properly.  If you are looking for more comprehensive solutions to visualize images together with segmentations please check out the [3D Slicer](https://slicer.org/) desktop application or [itkWidgets](https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_RTStruct_SEG_Visualization_with_itkWidgets.ipynb) if you're trying to do this inside of a notebook."
      ],
      "metadata": {
        "id": "WuoAOVCYoiXe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.2.7 Download images based on demographics and treatment data\n",
        "\n",
        "Basic demographic data such as patient sex, ethnicity and age at the time of the imaging study can sometimes be found within DICOM tags.  When these data are populated you can use **getStudy()** to access this information.  Here's a random example showing this with the [Prostate-MRI-US-Biopsy](https://doi.org/10.7937/TCIA.2020.A61IOC1A) collection."
      ],
      "metadata": {
        "id": "fXZmTf3diRlN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getStudy(collection = \"Prostate-MRI-US-Biopsy\", format = \"df\")"
      ],
      "metadata": {
        "id": "CPAue-mGnPEl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Many collections in TCIA also come with clinical data about the subjects that are not contained in the DICOM itself.  You can find collections which have such data by going to the [Browse Collections](https://www.cancerimagingarchive.net/browse-collections/) page and typing **clinical** into the filter box at the top of the table.  \n",
        "\n",
        "For the larger data collection initiatives sponsored by NCI/NIH such as [The Cancer Genome Atlas (TCGA), the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the Cancer Moonshot Biobank (CMB)](https://www.cancerimagingarchive.net/imaging-proteogenomics/) and the [NCI Clinical Trial Network (NCTN)](https://wiki.cancerimagingarchive.net/x/BQHDAg) these clinical data are generally stored in external databases which can be found in the **Additional Resources** section of the pages describing those datasets.  You can see an example of accessing clinical data from NCI's Genomic Data Commons website and then using that to define a custom cohort of radiology images at https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCGA/TCGA_Clinical.ipynb.\n",
        "\n",
        "However, for **community-proposed** datasets we typically host the data directly in the form of CSV files provided by the submitters.  To demonstrate this, let's assume that you're interested in looking at CT images from COVID-19 patients and are looking for additional labels that could be used to train an AI model.  \n",
        "\n",
        "First, we'll need to go to the [Browse Collections](https://www.cancerimagingarchive.net/browse-collections/) page and type both **clinical** and **covid** into the filter box.  Looking at the resulting datasets, let's say we've decided to focus on [COVID-19-NY-SBU](https://doi.org/10.7937/TCIA.BBAG-2923) since it has many more patients than the others.  \n",
        "\n",
        "After clicking the link to view the homepage for this dataset we notice that there are two spreadsheets available.  Let's take a quick look at them to see what we can learn by copying the URLs and reading them into a dataframe.  We'll start with the file on the page called **Clinical data**."
      ],
      "metadata": {
        "id": "7ToqauvKm-yu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clinical_data = pd.read_csv(\"https://www.cancerimagingarchive.net/wp-content/uploads/deidentified_overlap_tcia.csv.cleaned.csv_20210806.csv\")\n",
        "\n",
        "display(clinical_data)"
      ],
      "metadata": {
        "id": "DWEyopC3rUn0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Wow, this looks pretty extensive! Let's take a quick at the **Clinical data template** file now."
      ],
      "metadata": {
        "id": "v-fEU7g-sCMk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clinical_data_template = pd.read_csv(\"https://www.cancerimagingarchive.net/wp-content/uploads/deidentified_overlap_tcia.csv.cleaned.csv.template_20210806.csv\")\n",
        "\n",
        "display(clinical_data_template)"
      ],
      "metadata": {
        "id": "V4tgi_GyrJ3H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It appears the **Clinical Data Template** file is summarizing the contents of the clinical data.\n",
        "\n",
        "After reviewing the contents of these files, let's say that we're interested trying to predict **last.status** from CT scans. To achieve this, let's start by creating a list of the patient IDs in each category."
      ],
      "metadata": {
        "id": "eTlbaqG6rz6h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "deceased = clinical_data.loc[clinical_data['last.status'] == 'deceased', 'to_patient_id'].tolist()\n",
        "discharged = clinical_data.loc[clinical_data['last.status'] == 'discharged', 'to_patient_id'].tolist()\n",
        "\n",
        "print(len(deceased), \"deceased patients.\")\n",
        "print(len(discharged), \"discharged patients.\")"
      ],
      "metadata": {
        "id": "kPUEzz2AtmyD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This dataset has a number of different modalities in it, so let's get an inventory of all of the available CT scans."
      ],
      "metadata": {
        "id": "9osa5S_PveiH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ct_scans = nbia.getSeries(collection = \"COVID-19-NY-SBU\", modality = \"CT\", format = \"df\")\n",
        "\n",
        "display(ct_scans)"
      ],
      "metadata": {
        "id": "PnGTEPAWwOAS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's use the lists we created previously to add the **last.status** column to our inventory of CT scans."
      ],
      "metadata": {
        "id": "o5VLm9nUIF_m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the 'last.status' column and populate it based on the 'PatientID' values\n",
        "ct_scans['last.status'] = ct_scans['PatientID'].apply(lambda x: 'deceased' if x in deceased else 'discharged')\n"
      ],
      "metadata": {
        "id": "RCEpjdGqzP-_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "display(ct_scans)"
      ],
      "metadata": {
        "id": "h0NA79KIzvZZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we have everything we need in our dataframe, we can download the actual images.\n",
        "\n",
        "**Reminder:** We're using the **number** parameter to download 1 sample series.  Remove this to download the full dataset."
      ],
      "metadata": {
        "id": "_QloklFXIaHS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries(ct_scans, input_type = \"df\", number = 1, format = \"df\")"
      ],
      "metadata": {
        "id": "vZ6Deox20sYD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3exO0oNHtMY",
        "tags": []
      },
      "source": [
        "### 3.2.8 Download data from a restricted collection\n",
        "In some cases, you must specifically request access to collections before you can download them.  These are listed as **limited access** on the [Browse Collections](https://www.cancerimagingarchive.net/collections/) page. The steps to request access may vary depending on the collection, but will always require that you first [create a TCIA user account](https://wiki.cancerimagingarchive.net/x/xgHDAg). Once you've created an account, you can use your login/password to create an API token with the **getToken()** function from **tcia_utils** to verify your permissions.\n",
        "\n",
        "Tokens are valid for 2 hours and must be refreshed after that point.  The tcia_utils functions all check to see if you need to refresh your token when they're called and will take care of that for you automatically if necessary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZzwyJcNYIIsq"
      },
      "outputs": [],
      "source": [
        "nbia.getToken()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1NLNszIIKOS"
      },
      "source": [
        "Let's say that we're interested in the [QIN-Breast-02](https://doi.org/10.7937/TCIA.2019.4cfm06rr) collection. As you can see on the collection page, you must email help@cancerimagingarchive.net to request access to the data. Once you've recieved approval we can use **nbia.getSeries()** to get a full list of series UIDs in this restricted collection by including **api_url = \"restricted\"** as a parameter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IAnZc6kdTYRL"
      },
      "outputs": [],
      "source": [
        "# getSeries with query parameters\n",
        "data = nbia.getSeries(collection = \"QIN-Breast-02\",\n",
        "                      api_url = \"restricted\")\n",
        "\n",
        "print(len(data), 'Series returned')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vxYch98LmtC0"
      },
      "source": [
        "Now we can download those scans.  Don't forget to include **api_url = \"restricted\"** in the download functions as well!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DGaaZxx7JKK2"
      },
      "outputs": [],
      "source": [
        "# feed series_data to our downloadSampleSeries function\n",
        "df = nbia.downloadSeries(data, number = 3, api_url = \"restricted\", format = \"df\")\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Additional Resources\n",
        "The following pages on TCIA may be of special interest to deep learning researchers:\n",
        "\n",
        "1. [Finding Annotated Data for AI/ML on TCIA](https://wiki.cancerimagingarchive.net/x/TAGJAw) provides basic guidance for finding datasets that could be useful for deep learning tasks.\n",
        "2. [Challenge Competitions using TCIA data](https://wiki.cancerimagingarchive.net/x/nYIaAQ) can be useful for benchmarking your model's performance.\n",
        "3. [ACR Data Science Institute's Define AI Directory](https://www.acrdsi.org/DSI-Services/Define-AI) links clinically relevant AI use-cases to TCIA datasets that can be used to address them.\n",
        "4. [Additional TCIA Notebooks](https://github.com/kirbyju/TCIA_Notebooks) about accessing and visualizing data are available."
      ],
      "metadata": {
        "id": "rpt0pjo2s_Ho"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Acknowledgements\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/).  It is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/) and hosted by the [University of Arkansas for Medical Sciences (UAMS)](https://www.uams.edu/)\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/). If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to the following paper!\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ],
      "metadata": {
        "id": "LMnX2nBD9cEH"
      }
    }
  ]
}