TCIA_Data_Curation_Learning_Lab_SIIM_2024.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "7YGlC_ajggC8",
        "nj6P7YCmll4S",
        "aZvpm8BiUAnx",
        "elX2coKjUK4R",
        "rOVuIo2Xqozd",
        "BJEBAyTyqcU5",
        "YMIPh4vzqfN5"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_Data_Curation_Learning_Lab_SIIM_2024.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Data_Curation_Learning_Lab_SIIM_2024.ipynb)"
      ],
      "metadata": {
        "id": "DUmcjg7W2Pz8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# SIIM 2024 TCIA Data Curation Learning Lab\n",
        "\n",
        "Access to large, high-quality datasets is essential for researchers to understand disease and precision medicine pathways, especially in cancer. However, there are many challenges associated with publishing and utilizing radiological imaging data of human subjects. In this hands-on learning lab we'll teach you some solutions to these challenges, including how to properly de-identify and publish your DICOM data as well as how to access freely available datasets that have been published in online archives.\n",
        "\n",
        "You can view the full course description, requirements and objects at https://annualmeeting.siim.org/sessions/hands-on-data-curation-learning-lab/.  This notebook was developed for the course to demonstrate command-line and API-based options for accessing data from The Cancer Imaging Archive."
      ],
      "metadata": {
        "id": "KmXfYFZtja2F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Setup\n",
        "\n",
        "The following installs and imports **[tcia_utils](https://pypi.org/project/tcia-utils/)**, which contains a variety of useful functions for accessing TCIA via Python and Jupyter Notebooks."
      ],
      "metadata": {
        "id": "BtHGbUj47d4V"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zP4VRfgg-QXU"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# install tcia utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next we'll import modules to help us work with a few different TCIA APIs and change the logging settings if you're on Colab so you can see more of the INFO statements that tell you what's going on as we run our commands."
      ],
      "metadata": {
        "id": "wSFYJdY3-DeR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p3zEqnxi9rk2"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "from tcia_utils import wordpress\n",
        "from tcia_utils import nbia\n",
        "\n",
        "# set logging level to INFO in Google Colab (not necessary in Jupyter)\n",
        "if 'google.colab' in sys.modules:\n",
        "  import logging\n",
        "\n",
        "  for handler in logging.root.handlers[:]:\n",
        "      logging.root.removeHandler(handler)\n",
        "\n",
        "  # Set handler with level = info\n",
        "  logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',\n",
        "                      level=logging.INFO)\n",
        "\n",
        "  print(\"Google Colab Logging set to INFO\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Finding datasets of interest with our Wordpress API (aka Collection Manager)\n",
        "\n",
        "This API contains metadata about the datasets we host including free-text summaries, available files for download, citation requirements, related publications and versioning info. Full documentation about this API can be found at https://www.cancerimagingarchive.net/collection-manager-rest-api/, but we'll rely on the **wordpress** module in **tcia_utils** to simplify some common tasks."
      ],
      "metadata": {
        "id": "2gU9e6zA7SqL"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VTPpOFV3jiDT"
      },
      "source": [
        "# Getting collection metadata\n",
        "New image datasets are organized as “collections”. Typically these are patient cohorts related by a common disease (e.g. lung cancer), image modality or type (MRI, CT, digital histopathology, etc) or research focus. Supporting data related to the images such as patient outcomes, treatment details, genomics and image analyses are also provided when available.  These are described at https://www.cancerimagingarchive.net/browse-collections/.\n",
        "\n",
        "In order to get metadata about collections with tcia_utils we'll use the **wordpress.getCollections()** function.  The default format of the returned data is JSON, but you can set the format to \"df\" if you prefer a dataframe.  You can also specify a subset of fields to return and set a file_name if you'd like to save a CSV of the output.  Finally, since the values in some columns are HTML code there is a removeHTML parameter that can be used to convert those to plain text.  We'll demonstrate the use of all of these features in the next cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dm_mlAmuO6XS"
      },
      "outputs": [],
      "source": [
        "# select fields to retrieve\n",
        "fields = [\"id\", \"slug\", \"collection_page_accessibility\", \"link\", \"cancer_types\",\n",
        "          \"collection_doi\", \"cancer_locations\", \"collection_status\", \"species\",\n",
        "          \"versions\", \"citations\", \"collection_title\", \"version_number\",\n",
        "          \"date_updated\", \"subjects\", \"collection_short_title\", \"data_types\",\n",
        "          \"supporting_data\", \"program\", \"collection_summary\", \"collection_downloads\"]\n",
        "\n",
        "# request metadata\n",
        "collections = wordpress.getCollections(format = \"df\", fields = fields, file_name = \"tciaCollections.csv\", removeHtml = \"yes\")\n",
        "\n",
        "collections"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Getting analysis results metadata\n",
        "To enhance the value of TCIA collections we encourage the research community to publish their analyses of existing TCIA image collections. Examples of this kind of data includes radiologist or pathologist annotations, image classifications, segmentations, radiomics features, or derived/reprocessed images. These are datasets described at https://www.cancerimagingarchive.net/browse-analysis-results/.\n",
        "\n",
        "The **wordpress.getAnalyses()** function for this is nearly identical to getCollections() as shown below."
      ],
      "metadata": {
        "id": "zHbMj4FW2ICp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# select fields to retrieve\n",
        "fields = [\"id\", \"slug\", \"result_page_accessibility\", \"type\",\n",
        "          \"link\", \"cancer_types\", \"result_doi\", \"cancer_locations\",\n",
        "          \"status\", \"citations\", \"result_title\", \"version_number\",\n",
        "          \"date_updated\", \"versions\", \"subjects\", \"result_short_title\",\n",
        "          \"supporting_data\", \"program\", \"result_summary\", \"result_downloads\"]\n",
        "\n",
        "# request metadata\n",
        "analyses = wordpress.getAnalyses(format = \"df\", fields = fields, file_name = \"tciaAnalyses.csv\")\n",
        "\n",
        "analyses"
      ],
      "metadata": {
        "id": "aW-gooa_2Iwv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Filtering dataframes\n",
        "\n",
        "tcia_utils has a helper function called **searchDf()** that makes it easy to filter across an entire dataframe for keywords regardless of case.  Let's say that you were interesting in narrowing the results of your collection dataframe to datasets that mention **brain** in any column."
      ],
      "metadata": {
        "id": "oHNdb7WGHXyh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "brain = wordpress.searchDf(\"brain\", dataframe = collections)\n",
        "\n",
        "brain"
      ],
      "metadata": {
        "id": "6RmeOnrdK3xg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that in the **collection_page_accessibility** column a lot of these datasets are **Limited** access.  While not currently part of HIPAA regulations, there is some preliminary evidence of the possibility of matching 3D reconstructed MR and CT imaging of the head could be used to try and identify subjects.  As a result, data with these features are made available to investigators with a data use agreement.  \n",
        "\n",
        "For the sake of simplicity, let's reduce our results to only the fully public datasets which have been de-faced or skull stripped and do not require any special data usage agreement.  We can specify that we only want rows where **public** is in the **collection_page_accessibility** column here."
      ],
      "metadata": {
        "id": "WIHcZ1JuYiXE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "public_brain = wordpress.searchDf(\"public\", dataframe = brain, column_name = \"collection_page_accessibility\")\n",
        "\n",
        "public_brain"
      ],
      "metadata": {
        "id": "e-YO4O5OZNzH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's assume that we're only interested in datasets that also contain tumor segmentations.  This can be represented in a few different ways in the metadata.  In the **data_types** column, **RTSTRUCT** or **SEG** indicate DICOM representations of segmentations.  Sometimes we also receive data in other formats such as NIfTI which may be represented as **Segmentation** in this column.  \n",
        "\n",
        "Note: Including rows that have **Image Analyses** in the **supporting_data** column may also be of interest if you're looking for other types of analyses such as seed points, other 2D measurements or image classifications."
      ],
      "metadata": {
        "id": "fEf6r1teZipS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# note: 'Seg' in our search terms will catch 'SEG' and 'Segmentations'\n",
        "public_brain_seg = wordpress.searchDf(['RTSTRUCT', 'Seg'], dataframe = public_brain)\n",
        "\n",
        "public_brain_seg"
      ],
      "metadata": {
        "id": "BzCOH4m_aqnW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ok, now let's say that we also only want to include datasets that have supporting clinical data.  We can find these by filtering for **Clinical** in the **supporting_data** column."
      ],
      "metadata": {
        "id": "mrhKxTlArokt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# note: 'Seg' in our search terms will catch 'SEG' and 'Segmentations'\n",
        "public_brain_seg_clinical = wordpress.searchDf('clinical', dataframe = public_brain_seg, column_name = 'supporting_data')\n",
        "\n",
        "public_brain_seg_clinical"
      ],
      "metadata": {
        "id": "dk6tMMWmr0eM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Downloading DICOM data\n",
        "Let's assume that after reading the information in the **collection_summary** column and taking the other metadata into account we've decided the **ReMIND** dataset is of highest interest to us.  \n",
        "\n",
        "The IDs associated with the download record for each file we host is shown in the **collection_downloads** column above.  In the case of **ReMIND** the **ids** are [43723, 43725, 43727].  These can be requested with the **wordpress.getDownloads()** function using the **ids** parameter.  "
      ],
      "metadata": {
        "id": "mqZEUDSiPAnI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ids = [43723, 43725, 43727]\n",
        "\n",
        "fields = [\"id\", \"date_updated\", \"download_title\", \"data_license\", \"download_access\",\n",
        "          \"data_type\", \"file_type\", \"download_size\", \"download_size_unit\",\n",
        "          \"subjects\", \"study_count\", \"series_count\", \"image_count\",\n",
        "           \"download_type\", \"download_url\", \"download_file\", \"search_url\"]\n",
        "\n",
        "remind = wordpress.getDownloads(ids = ids, fields = fields, format = \"df\", file_name = \"ReMIND_downloads.csv\")\n",
        "\n",
        "remind"
      ],
      "metadata": {
        "id": "yKMzhEQouD79"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "There is a **query** parameter which is also a handy way to retrieve this metadata.  "
      ],
      "metadata": {
        "id": "I2A2cOM4uqIS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fields = [\"id\", \"date_updated\", \"download_title\", \"data_license\", \"download_access\",\n",
        "          \"data_type\", \"file_type\", \"download_size\", \"download_size_unit\",\n",
        "          \"subjects\", \"study_count\", \"series_count\", \"image_count\",\n",
        "           \"download_type\", \"download_url\", \"download_file\", \"search_url\"]\n",
        "\n",
        "remind = wordpress.getDownloads(query=\"ReMIND\", fields = fields, format = \"df\", file_name = \"ReMIND_downloads.csv\")\n",
        "\n",
        "remind"
      ],
      "metadata": {
        "id": "u8Rwhs6JtWtl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Information about where to download these data appear in the **download_url** column.  The **search_url** column conatains links where you can go and view the data before downloading it (where applicable).  In this particular case you can see that there's an option to look at the DICOM Images and Segmentations in our **NBIA** DICOM portal.\n",
        "\n",
        "In any case, let's first demonstrate how to download the DICOM and clinical data.  We'll start by downloading the **TCIA manifest file** for the DICOM images."
      ],
      "metadata": {
        "id": "xqpEAQBfu3B8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "url = 'https://www.cancerimagingarchive.net/wp-content/uploads/ReMIND-Manifest-Sept-2023.tcia'\n",
        "response = requests.get(url)\n",
        "with open('ReMIND-Manifest-Sept-2023.tcia', 'wb') as f:\n",
        "    f.write(response.content)"
      ],
      "metadata": {
        "id": "EqjsCxYaIX3-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Downloading with the Linux Command Line NBIA Data Retriever\n",
        "\n",
        "TCIA uses software called NBIA to manage its DICOM data.  One way to download TCIA data is to install the NBIA Data Retriever.  This tool provides a number of useful features such as auto-retry if there are any problems, saving data in an organized hierarchy on your hard drive (Collection > Patient > Study > Series > Images), and providing a CSV file containing key DICOM metadata about the images you've downloaded."
      ],
      "metadata": {
        "id": "eRvQa63-kJIY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install the NBIA Data Retriever\n",
        "There are versions of this tool for Windows, Mac and Linux.  If you're working from a system with a GUI you can follow the [instructions](https://wiki.cancerimagingarchive.net/display/NBIA/Downloading+TCIA+Images) to install Data Retriever on your computer.\n",
        "\n",
        "There is also a [command-line version of the NBIA Data Retriever](https://wiki.cancerimagingarchive.net/x/2QKPBQ) which can be installed via the steps below if you're running this notebook in a **Linux** environment.  "
      ],
      "metadata": {
        "id": "RtLE_18NoaJ8"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B714bOkDk8kd"
      },
      "source": [
        "# Install NBIA Data Retriever CLI software for downloading images later in this notebook.\n",
        "\n",
        "!mkdir /usr/share/desktop-directories/\n",
        "!wget -P /content/NBIA-Data-Retriever https://cbiit-download.nci.nih.gov/nbia/releases/ForTCIA/NBIADataRetriever_4.4/nbia-data-retriever-4.4.2.deb\n",
        "!dpkg -i /content/NBIA-Data-Retriever/nbia-data-retriever-4.4.2.deb\n",
        "\n",
        "# NOTE: If you're working on a Linux OS that uses RPM packages, you can change the lines above to use\n",
        "#       https://cbiit-download.nci.nih.gov//nbia/releases/ForTCIA/NBIADataRetriever_4.4/NBIADataRetriever-4.4-2.x86_64.rpm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you open the **ReMIND-Manifest-Sept-2023.tcia** file you'll see some configuration information at the top, followed by a list of Series Instance UIDs that are part of the dataset.  \n",
        "\n",
        "Let's edit the manifest file to only include the first 2 UIDs in the manifest so that we can demonstrate the download process more quickly.\n"
      ],
      "metadata": {
        "id": "RhG_JOUZ-CpF"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gsorr-mi7--3"
      },
      "outputs": [],
      "source": [
        "with open('ReMIND-Manifest-Sept-2023.tcia','r') as firstfile, open('ReMIND-Sample.tcia','a') as secondfile:\n",
        "    count = 0\n",
        "    for line in firstfile:\n",
        "        # append content to second file\n",
        "        secondfile.write(line)\n",
        "        # Stop after header and first 3 series UIDs\n",
        "        count += 1;\n",
        "        if count == 8:\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IlPLgxkBZPMS"
      },
      "source": [
        "### Open the Manifest File with the NBIA Data Retriever\n",
        "Next, let's open the sample manifest file with the NBIA Data Retriever to download the actual DICOM data.  To do this, we'll call the command to launch Data Retriever, specify the **--cli** flag to indicate we want to run this via command line (not with a GUI).  We also need to specify the path to the manifest file we want to open and then use **-d** to specify the path where we want to save the data.\n",
        "\n",
        "**<font color='red'>After running the following command, click in the output cell, type \"y,\" and press Enter to agree with the TCIA Data Usage Policy and start the download.</font>**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a4lklpk4Xwpc"
      },
      "outputs": [],
      "source": [
        "# download the data using NBIA Data Retriever\n",
        "\n",
        "!/opt/nbia-data-retriever/nbia-data-retriever --cli '/content/ReMIND-Sample.tcia' -d /content/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgESwVXSawv_"
      },
      "source": [
        "### NBIA Data Retriever Conclusion\n",
        "You should now find that the data have been saved to your machine in a well-organized hierarchy with some useful metadata in the accompanying CSV file and a license file detailing how it can be used.  Take a look at the data before moving on.\n",
        "\n",
        "A few other notes:\n",
        "* The CLI Data Retriever supports both \"Descriptive\" and \"Classic\" organization of the data.  Descriptive naming uses information from the DICOM Study/Series Description and Dates to make them easier for humans to interpret.  Classic names everything by machine-readable unique identifiers.  If you prefer machine-readable directory names simply add the **-cd** parameter to your download command.\n",
        "* In some cases, you must specifically request access to [Collections](https://www.cancerimagingarchive.net/browse-collections/) before you can download them.  Information about how to do this can be found on the homepage for the Collection(s) you're interested in, but will always require that you first [create a TCIA user account](https://wiki.cancerimagingarchive.net/x/xgHDAg).  Once you've created an account and obtained permission to the restricted data you want to download, you can use your login/password to create the **credentials.txt** file that NBIA Data Retriever uses to verify your permissions.  The path to the credential file is specified using the **-l** parameter.\n",
        "\n",
        "You can find examples for these use cases at [TCIA_Linux_Data_Retriever_App.ipynb](https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Linux_Data_Retriever_App.ipynb).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmeqJoR5k9z0"
      },
      "source": [
        "## Downloading DICOM data with the REST API\n",
        "The NBIA REST APIs are provided for the search and download functions used in the TCIA radiology portal and allow access to both public and limited access collections.\n",
        "1. The [NBIA Search REST APIs](https://wiki.cancerimagingarchive.net/x/fILTB) allow you to perform basic queries and download data from **public** collections. These APIs do not require a TCIA account.\n",
        "2. The [NBIA Search with Authentication REST APIs](https://wiki.cancerimagingarchive.net/x/X4ATBg) allow you to perform basic queries and download data from **public and limited-access** collections. These APIs require a TCIA account to create authentication tokens.\n",
        "3. The [NBIA Advanced REST APIs](https://wiki.cancerimagingarchive.net/x/YoATBg) also allow access to **public and limited-access** collections, but provide query endpoints mostly geared towards developers seeking to integrate searching and downloading TCIA data into web and desktop applications. This API requires a TCIA account to create authentication tokens.\n",
        "\n",
        "Feel free to reference this documentation if you run into issues or need clarification about anything related to the API.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqulqBEDMSS6"
      },
      "source": [
        "### tcia_utils\n",
        "\n",
        "Rather than working directly with the NBIA API we will once again be relying on [**tcia_utils**](https://pypi.org/project/tcia-utils/) to simplify things. Docstrings detailing how to use each function are provided in the code itself, and an extensive library of tutorial notebooks can be found at https://github.com/kirbyju/TCIA_Notebooks to address use cases we don't have time for today.\n",
        "\n",
        "By default, most search functions from tcia_utils return results in JSON.  However, you can use **format = \"df\"** to return the results as a dataframe, or **format = \"csv\"** to save a CSV file in addition to returning a dataframe.\n",
        "\n",
        "There are also two download functions that you can use after you've finished searching and determined what you want to download.  The first is **downloadSeries()** which is used for downloading an entire scan.  The second is **downloadImage()** which you can use to pull a single slice from a scan if needed.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwj_OZjphcWP"
      },
      "source": [
        "### Download the full collection and preview a series\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's say that we're interested in downloading the entire **ReMIND** collection via the API instead of installing the **NBIA Data Retriever** software.  One way to do this is to leverage the manifest file we obtained earlier.  If we set **input_type = \"manifest\"** when calling **downloadSeries()** we can provide the path/filename to **downloadSeries()**.  This will automatically extract the UIDs from the file and download them."
      ],
      "metadata": {
        "id": "Y2w7nNgLOanz"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tmx0Ej70WKJ4"
      },
      "outputs": [],
      "source": [
        "nbia.downloadSeries(\"/content/ReMIND-Sample.tcia\", input_type = \"manifest\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jbmm5DIU4Exy"
      },
      "source": [
        "Another way to download data from the API is to get a list of all Series Instance UIDs in that collection.  We can use **nbia.getSeries()** to save the JSON metadata about all series (scans) in this collection to a variable called **data**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eVnVh_yv7Av3"
      },
      "outputs": [],
      "source": [
        "data = nbia.getSeries(collection = \"ReMIND\")\n",
        "print(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfkCWxiK8nQz"
      },
      "source": [
        "Then we can pass that **data** variable to the our download function.  We'll leverage the **number** parameter here to just grab the first scan as a test.  You can remove this parameter if you want to download the full collection."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2XVUkw5vgMZX"
      },
      "outputs": [],
      "source": [
        "nbia.downloadSeries(data, number = 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before moving on, take a second to review how the data are saved.  You'll note that by default the data are stored to a directory called **tciaDownload** with each directory named after the **Series Instance UID**.  While these UIDs are important to uniquely identify each scan, it can be difficult to figure out what is what unless you have additional metadata.  \n",
        "\n",
        "In order to obtain such metadata, you can set the **format** parameter to **df** to return a dataframe containing the metadata for the files you've downloaded.  Setting **format = \"csv\"** will save a spreadsheet in addition to returning a dataframe."
      ],
      "metadata": {
        "id": "UYQDcnmj2VQ_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries(data, number = 1, format = \"csv\")"
      ],
      "metadata": {
        "id": "GILRfvGWbZqy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**tcia_utils** also provides a function called **viewSeries()** we can use to do a quick preview of the images here in the notebook.  The function assumes \"tciaDownload/**seriesUid**/\" as path if seriesUid is provided since this is where downloadSeries() saves data.  However you can also open a series from a custom path using the **path** parameter."
      ],
      "metadata": {
        "id": "0GdGyJ59hPJA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# example using just the series UID\n",
        "nbia.viewSeries(\"1.3.6.1.4.1.14519.5.2.1.129870686332767537746901777770022995560\")\n",
        "\n",
        "# alternate example specifying a custom path\n",
        "# nbia.viewSeries(path = \"/content/tciaDownload/1.3.6.1.4.1.14519.5.2.1.129870686332767537746901777770022995560\")"
      ],
      "metadata": {
        "id": "jY6NmTeNhWz3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7Pw0jvy4HO8"
      },
      "source": [
        "### 3.2.2 Download custom API query\n",
        "Now let's assume that you've recently trained a model to segment tumors in brain MRIs so you're not interested in the ultrasound data that ReMIND contains. First, let's get an inventory of the available scans.  Then we'll filter it to only include the MR and SEG modalities."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "remind_series = nbia.getSeries(collection = \"ReMIND\",format = \"df\")\n",
        "remind_seg_mr_series = nbia.searchDf([\"MR\", \"SEG\"], dataframe = remind_series, column_name = \"Modality\")\n",
        "\n",
        "display(remind_seg_mr_series)"
      ],
      "metadata": {
        "id": "DaFVbsT4SBWk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can pass this dataframe to **downloadSeries()** to grab only this subset of data.  We'll use the **number** parameter here for demonstration purposes, but you can remove this if you want to download the full dataset."
      ],
      "metadata": {
        "id": "LBQgby3_Ybuw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries(remind_seg_mr_series, input_type= \"df\", format = \"df\", number = 2)"
      ],
      "metadata": {
        "id": "rvOl_TcoYr8u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.2.7 Download images based on clinical data\n",
        "\n",
        "Basic demographic data such as patient sex, ethnicity and age at the time of the imaging study can sometimes be found within DICOM tags.  When these data are populated you can use **getStudy()** to access this information.  "
      ],
      "metadata": {
        "id": "fXZmTf3diRlN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getStudy(collection = \"ReMIND\", format = \"df\")"
      ],
      "metadata": {
        "id": "CPAue-mGnPEl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this particular case there is no basic demographic information such as patient age, sex, race/ethnicity that sometimes resides in the DICOM metadata.  However, many collections in TCIA also come with clinical data about the subjects that are not contained in the DICOM itself.  \n",
        "\n",
        "For the larger data collection initiatives sponsored by NCI/NIH such as [The Cancer Genome Atlas (TCGA), the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the Cancer Moonshot Biobank (CMB)](https://www.cancerimagingarchive.net/imaging-proteogenomics/) and the [NCI Clinical Trial Network (NCTN)](https://wiki.cancerimagingarchive.net/x/BQHDAg) these clinical data are generally stored in external databases which can be found in the **Additional Resources** section of the pages describing those datasets.  \n",
        "\n",
        "However, for **community-proposed** datasets we typically host the data directly in the form of CSV files provided by the submitters.  ReMIND happens to be an example of this as we learned while doing our Wordpress searches earlier.\n",
        "\n",
        "Let's first **ReMIND** ourselves where that clinical file lives (har har har)."
      ],
      "metadata": {
        "id": "7ToqauvKm-yu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "remind"
      ],
      "metadata": {
        "id": "BrndBNvXaix6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ok, now let's read the XLSX file into a dataframe."
      ],
      "metadata": {
        "id": "lBMRoZEOamIl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter rows where 'download_type' column contains 'clinical'\n",
        "clinical = remind[remind['download_type'] == 'Clinical Data']\n",
        "\n",
        "# Since there's only one URL, we can directly read it into a DataFrame\n",
        "url = clinical['download_url'].values[0]\n",
        "remind_clinical = pd.read_excel(url)\n",
        "\n",
        "remind_clinical"
      ],
      "metadata": {
        "id": "DWEyopC3rUn0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This data could now be easily merged with the image metadata in the **remind_seg_mr_series** we created earlier and used for training AI models on any of the available data.  "
      ],
      "metadata": {
        "id": "eTlbaqG6rz6h"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Downloading with Aspera ASCLI\n",
        "\n",
        "You may have also noticed there was also a separate row for some NRRD segmentation data in the **remind** dataframe.  "
      ],
      "metadata": {
        "id": "y3eFafyJJ9Lt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "remind"
      ],
      "metadata": {
        "id": "fwrp-j57lLSp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Most non-DICOM content in TCIA are provided via IBM Aspera Faspex packages which are typically accessed via a browser plugin.  However, the IBM Aspera developers also maintain an open source tool called [ascli (aspera-cli)](https://github.com/IBM/aspera-cli) that allows a client to download an Aspera Faspex package via the command line.\n",
        "\n",
        "You can tell if data is hosted in an Aspera Faspex by the **download_url**, which begins with https://faspex.cancerimagingarchive.net/.\n",
        "\n",
        "In order to pull data from this package we need to install Ruby, then the aspera-cli gem, and afterward we can use that to install ascli.  The steps to [install Ruby](https://www.ruby-lang.org/en/downloads/) vary by operating system but the gem and ascli commands should be the same as the last two lines below."
      ],
      "metadata": {
        "id": "9XuA7c-alMui"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# consult the link above for Ruby installation instructions if you're not on an OS that uses apt\n",
        "!apt install -y ruby ruby-dev rubygems ruby-json\n",
        "\n",
        "# these should work in all environments after ruby is installed\n",
        "!gem install aspera-cli\n",
        "!ascli conf ascp install"
      ],
      "metadata": {
        "id": "Cbfvmj_FbEnS",
        "collapsed": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, we need to know the package **--url** where the NRRD files reside.  Then you can use the **browse** parameter to look around inside the package before downloading things.  Running the command without any path specified will show you the root folder of the package.  "
      ],
      "metadata": {
        "id": "XP6unlXT1Dzu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# First let's pull the id associated with the Aspera package to get the url\n",
        "url = remind.loc[remind['id'] == 43725, 'download_url'].values[0]\n",
        "\n",
        "# Now you can use this url in your shell command\n",
        "command = f'ascli faspex5 packages browse --url=\"{url}\"'\n",
        "\n",
        "# run the ascli command\n",
        "get_ipython().system(command)\n",
        "\n"
      ],
      "metadata": {
        "id": "8rfgg90bko8K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you want to look at the contents of one of the directories you just add the directory path to the end of this command."
      ],
      "metadata": {
        "id": "P2yGf6w2Pg1m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "command = f'ascli faspex5 packages browse --url=\"{url}\" ReMIND_NRRD_Seg_Sep_2023'\n",
        "\n",
        "# run the ascli command\n",
        "get_ipython().system(command)"
      ],
      "metadata": {
        "id": "OxsvzgCrPna0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When you decide what you want to download, you can once again use the **recieve** command.  Let's say that we're interested in grabbing a single subject's data to assess it before we download the full package.  You can do that by simply appending the path of the directory to the end of the **receive** command."
      ],
      "metadata": {
        "id": "w4mDYN3Xfdkz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "command = f'ascli faspex5 packages receive --url=\"{url}\" ReMIND_NRRD_Seg_Sep_2023/ReMIND-001'\n",
        "\n",
        "# run the ascli command\n",
        "get_ipython().system(command)"
      ],
      "metadata": {
        "id": "8DssIjsEfeV6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Take a second to look at the files downloaded.  Congratulations! You've now seen how to search and download all data types programatically without a browser for this dataset!"
      ],
      "metadata": {
        "id": "sx-8UJay-CBw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Additional Resources\n",
        "The following pages on TCIA may be of special interest to deep learning researchers:\n",
        "\n",
        "1. [Finding Annotated Data for AI/ML on TCIA](https://wiki.cancerimagingarchive.net/x/TAGJAw) provides basic guidance for finding datasets that could be useful for deep learning tasks.\n",
        "2. [Challenge Competitions using TCIA data](https://wiki.cancerimagingarchive.net/x/nYIaAQ) can be useful for benchmarking your model's performance.\n",
        "3. [ACR Data Science Institute's Define AI Directory](https://www.acrdsi.org/DSI-Services/Define-AI) links clinically relevant AI use-cases to TCIA datasets that can be used to address them.\n",
        "4. [Additional TCIA Notebooks](https://github.com/kirbyju/TCIA_Notebooks) about accessing and visualizing data are available."
      ],
      "metadata": {
        "id": "rpt0pjo2s_Ho"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Acknowledgements\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/).  It is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/) and hosted by the [University of Arkansas for Medical Sciences (UAMS)](https://www.uams.edu/)\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/). If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to the following paper!\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ],
      "metadata": {
        "id": "LMnX2nBD9cEH"
      }
    }
  ]
}