TCIA_NCTN_Annotations.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "SmeqJoR5k9z0",
        "CSTkcuXMgCDd",
        "7YGlC_ajggC8",
        "nj6P7YCmll4S",
        "I9_O-nq1lr7c",
        "JkCZMxa_ua6P",
        "BJEBAyTyqcU5",
        "YMIPh4vzqfN5",
        "9Zp_hWmXsKZ9"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_NCTN_Annotations.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_NCTN_Annotations.ipynb)"
      ],
      "metadata": {
        "id": "zhJ1PywgTDmh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Summary\n",
        "\n",
        "Access to large, high quality data is essential for researchers to understand disease and precision medicine pathways, especially in cancer. However HIPAA constraints make sharing medical images outside an individual institution a complex process. [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) TCIA is a public service funded by the National Cancer Institute which addresses this challenge by providing hosting and de-identification services to take major burdens of data sharing off researchers.\n",
        "\n",
        "TCIA has published over 200 unique data collections containing more than 60 million images. Recognizing that images alone are not enough to conduct meaningful research, most collections are linked to rich supporting data including patient outcomes, treatment information, genomic / proteomic analyses, and expert image analyses (segmentations, annotations, and radiomic / radiogenomic features).\n",
        "\n",
        "**This notebook is focused on accessing NCI Clinical Trial Network (NCTN) collections via TCIA REST APIs which contain radiologist-generated tumor annotations (3d segmentation labels and seed points) which can be used for creating automated methods to detect and segment tumors.**  More information about this activity can be found on the [Imaging Clinical Trials](https://wiki.cancerimagingarchive.net/x/BQHDAg) page on TCIA.\n"
      ],
      "metadata": {
        "id": "KmXfYFZtja2F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1 Setup\n",
        "\n",
        "The following installs **[tcia_utils](https://pypi.org/project/tcia-utils/)**, which contains a variety of useful functions for accessing TCIA data via Python.  It also installs **[simpleDicomViewer](https://pypi.org/project/simpleDicomViewer/)** as a very basic way to visualize example images.  There are a few conditional steps that will execute if you're using Google Colab to adjust log settings."
      ],
      "metadata": {
        "id": "bqulqBEDMSS6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "\n",
        "# install tcia utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils\n",
        "\n",
        "# install simpleDicomViewer and forked pydicom-seg dependency\n",
        "!{sys.executable} -m pip install --upgrade -q git+https://github.com/kirbyju/pydicom-seg.git@master\n",
        "!{sys.executable} -m pip install --upgrade -q simpleDicomViewer"
      ],
      "metadata": {
        "id": "zP4VRfgg-QXU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "from tcia_utils import nbia\n",
        "from simpleDicomViewer import dicomViewer\n",
        "\n",
        "# set logging level to INFO in Google Colab (not necessary in Jupyter)\n",
        "if 'google.colab' in sys.modules:\n",
        "  import logging\n",
        "\n",
        "  for handler in logging.root.handlers[:]:\n",
        "      logging.root.removeHandler(handler)\n",
        "\n",
        "  # Set handler with level = info\n",
        "  logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',\n",
        "                      level=logging.INFO)\n",
        "\n",
        "  print(\"Google Colab Logging set to INFO\")"
      ],
      "metadata": {
        "id": "p3zEqnxi9rk2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2 Learn about the datasets\n",
        "\n",
        "The images, annotations (tumor segmentation and seed point labels), and clinical data associated with this trial are described in detail at the following links.  These pages are publicly visible without logging in, and can be used to obtain an understanding of the dataset before going through the trouble of requesting access.  Instructions for obtaining access can be found on the **Collection Summary** page for each dataset.\n",
        "\n",
        "1. **Annotations for Chemotherapy and Radiation Therapy in Treating Young Patients With Newly Diagnosed, Previously Untreated, High-Risk Medulloblastoma/PNET (ACNS0332)**: [Image Collection Summary](https://doi.org/10.7937/TCIA.582B-XZ89), [Annotation Summary](https://doi.org/10.7937/D8A8-6252),   [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/838)\n",
        "2. **Combination Chemotherapy and Radiation Therapy in Treating Young Patients With Newly Diagnosed Hodgkin Lymphoma (AHOD0831)**: [Collection Summary](https://doi.org/10.7937/CV5M-1H59), [Annotation Summary](https://doi.org/10.7937/4QAD-4280), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/1137)\n",
        "3. **Vincristine, Dactinomycin, and Doxorubicin With or Without Radiation Therapy or Observation Only in Treating Younger Patients Who Are Undergoing Surgery for Newly Diagnosed Stage I, Stage II, or Stage III Wilms' Tumor (AREN0532)**: [Collection Summary](https://doi.org/10.7937/6PJ1-M859), [Annotation Summary](https://doi.org/10.7937/kja4-1z76), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/689)\n",
        "4. **Combination Chemotherapy With or Without Radiation Therapy in Treating Young Patients With Newly Diagnosed Stage III or Stage IV Wilms Tumor (AREN0533)**: [Collection Summary](https://doi.org/10.7937/SJEZ-CJ78), [Annotation Summary](https://doi.org/10.7937/WFCC-DA41), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/737)\n",
        "5. **Combination Chemotherapy and Surgery in Treating Young Patients With Wilms Tumor (AREN0534)**: [Collection Summary](https://doi.org/10.7937/TCIA.5M9S-6Y97), [Annotation Summary](https://doi.org/10.7937/N930-BM78), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/728)\n",
        "6. **Rituximab and Combination Chemotherapy in Treating Patients With Diffuse Large B-Cell Non-Hodgkin's Lymphoma (CALGB50303)**: [Collection Summary](https://doi.org/10.7937/CM65-A013), [Annotation Summary](https://doi.org/10.7937/9jer-g980), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/989)\n",
        "7. **Sorafenib Tosylate in Treating Patients With Desmoid Tumors or Aggressive Fibromatosis (A091105)**: [Collection Summary](https://doi.org/10.7937/0WF5-SJ50), [Annotation Summary](https://doi.org/10.7937/T8RN-J447), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/1266)\n",
        "8. **Radiation Therapy, Amifostine, and Chemotherapy in Treating Young Patients With Newly Diagnosed Nasopharyngeal Cancer (ARAR0331)**: [Collection Summary](https://doi.org/10.7937/WTEC-MN22), [Annotation Summary](https://doi.org/10.25737/H65S-8F58), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/1302)\n",
        "9. **ACRIN 6685 (ACRIN-HNSCC-FDG-PET-CT)**: [Collection Summary and clinical datasets](https://doi.org/10.7937/K9/TCIA.2016.JQEJZZNG), [Annotation Summary](https://doi.org/10.7937/JVGC-AQ36)\n",
        "10. **Risk-Based Therapy in Treating Younger Patients With Newly Diagnosed Liver Cancer (AHEP0731)**: [Collection Summary](https://doi.org/10.7937/F2DB-8826), [Annotation Summary](https://doi.org/10.7937/BDBN-NQ81), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/696)\n",
        "11. **A Randomized Phase III Study Comparing Carboplatin/Paclitaxel or Carboplatin/Paclitaxel/Bevacizumab With or Without Concurrent Cetuximab in Patients With Advanced Non-small Cell Lung Cancer (S0819)**: [Collection Summary](http://doi.org/10.7937/DT39-JS04), [Annotation Summary](https://doi.org/10.7937/R0R8-BN93), [Clinical datasets](https://nctn-data-archive.nci.nih.gov/node/850)\n",
        "\n",
        "**Note:** The **Clinical datasets** links above allow you to view data dictionaries outlining the specific clinical variables that were collected before you request access."
      ],
      "metadata": {
        "id": "AruUGe3lmjkh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "After obtaining access to the dataset(s) you're interested in, select the dataset you'd like to explore through the rest of this notebook by setting the collection variable below.  The variable should be set to the collection **short name** which is listed in parentheses at the end of each title above."
      ],
      "metadata": {
        "id": "BIAgDauev6Nw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "collection = \"S0819\""
      ],
      "metadata": {
        "id": "JHpYML61v-DT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3 Downloading images and annotations with the NBIA Data Retriever\n",
        "\n",
        "TCIA uses software called NBIA to manage its DICOM data.  One way to download TCIA data is to install the NBIA Data Retriever and use the predefined manifest files that are found on the summary pages mentioned in section 2.  \n",
        "\n",
        "This tool provides a number of useful features such as auto-retry if there are any problems, saving data in an organized hierarchy on your hard drive (Collection > Patient > Study > Series > Images), and providing a CSV file containing key DICOM metadata about the images you've downloaded.  \n",
        "\n",
        "There are versions available for Windows, Mac and Linux.  If you're working from a system with a GUI you can follow the [instructions](https://wiki.cancerimagingarchive.net/display/NBIA/Downloading+TCIA+Images) to install Data Retriever on your computer.  There is also a [Linux command-line version of the NBIA Data Retriever](https://wiki.cancerimagingarchive.net/x/2QKPBQ) which is demonstrated in [this notebook](https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Linux_Data_Retriever_App.ipynb)."
      ],
      "metadata": {
        "id": "eRvQa63-kJIY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVOOJWDckJIa"
      },
      "source": [
        "# 4 Accessing the REST APIs\n",
        "The [NBIA REST APIs](https://wiki.cancerimagingarchive.net/x/ZoATBg) are another useful way for TCIA users to query metadata and download image data, which will be the focus of the rest of this notebook.  We'll rely heavily on [tcia_utils](https://pypi.org/project/tcia-utils/) to simplify accessing them.  \n",
        "\n",
        "If you have questions that are not covered in this notebook you can find many additional examples in the other notebooks at https://github.com/kirbyju/TCIA_Notebooks."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create login token\n",
        "First, you must create a login token with your user name and password in order to access restricted data via the API.  "
      ],
      "metadata": {
        "id": "xVCXacXlH_7P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.getToken()"
      ],
      "metadata": {
        "id": "UTiHpYNTG9d_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.1 Explore the data with REST API queries\n",
        "\n",
        "Let's start by looking at what body parts and modalities are contained in the collection.  By default, most functions from **tcia_utils** return results in JSON.\n",
        "\n",
        "For these datasets, **RTSTRUCT** DICOM series were used to record the segmentations, seed points, and scans where no tumor was found. There is one exception to this, which is that DICOM **SEG** was the format used for segmentations in the **ACNS0332** dataset."
      ],
      "metadata": {
        "id": "g6D_3Dn7kJIa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# count patients for each modality\n",
        "data = nbia.getModalityCounts(collection)\n",
        "print(data)"
      ],
      "metadata": {
        "id": "oLDN4BAhbhvZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also use **format = \"df\"** to return the results as a dataframe.  Let's try that for viewing the body parts examined."
      ],
      "metadata": {
        "id": "VFrw8FAlkY8x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Count patients for each body part examined,\n",
        "# return results as dataframe\n",
        "df = nbia.getBodyPartCounts(collection, format = \"df\")\n",
        "\n",
        "# rename headers and sort by PatientCount\n",
        "df.rename(columns = {'criteria':'BodyPartExamined', 'count':'PatientCount'}, inplace = True)\n",
        "df.PatientCount = df.PatientCount.astype(int)\n",
        "display(df.sort_values(by='PatientCount', ascending=False, ignore_index = True))"
      ],
      "metadata": {
        "id": "_FZqff2fkJIa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's run **nbia.getPatient()** and **nbia.getStudy()** to see what we can learn about the patient cohort from the DICOM metadata.  The patient information can include things like age, gender, and ethnicity. The study information includes details recorded on the date the patient was scanned such as the patient's age at the time of imaging or how many days it has been since they were registered to the trial.  This information can augment the **Clinical data** available through the links at the top of the notebook."
      ],
      "metadata": {
        "id": "OJQQ4-_ZkJIa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = nbia.getPatient(collection, format = \"df\")\n",
        "\n",
        "display(df)"
      ],
      "metadata": {
        "id": "XUBlXp7P_JI8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's use **format = \"csv\"** this time to save a CSV file in addition to returning a dataframe.  Verify that it has been saved to your file system before proceeding."
      ],
      "metadata": {
        "id": "-IkG3w4Xb6W7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtain study/visit details (e.g. anonymized study date, age at the time of visit)\n",
        "df = nbia.getStudy(collection, format = \"csv\")\n",
        "display(df)"
      ],
      "metadata": {
        "id": "5Kn8nhMSB5u1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also create a report with **nbia.getSeries()** that gives useful metadata about each scan in the dataset (e.g. series description, modality, scanner manufacturer & software version, number of images)."
      ],
      "metadata": {
        "id": "e1s3VMGUUjSE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtain scan/series metadata and save to variable for use in next example\n",
        "df = nbia.getSeries(collection, format = \"df\")\n",
        "\n",
        "display(df)"
      ],
      "metadata": {
        "id": "dkK8TsGKUjSE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we can pass the results from the **getSeries()** query to **reportDoiSummary()** to generate some summary statistics about the data in the collection.  Note that there are separate rows summarizing the contents of the original collection and the contents of the annotation dataset."
      ],
      "metadata": {
        "id": "oM-Mmc6wUjSF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate summary statistics for a given collection\n",
        "nbia.reportDoiSummary(df, input_type = \"df\")"
      ],
      "metadata": {
        "id": "U3sCzz3KUjSF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.3 Downloading data with the REST API\n",
        "Next we'll demonstrate using the API to download data.  This can be useful if you'd like to download results from API queries rather than using an existing manifest file.  It's also useful if you can't install the NBIA Data Retriever or want to integrate TCIA downloads into other pipelines/tools.  \n",
        "\n",
        "Here we will focus on the following use cases:\n",
        "\n",
        "1. Download and visualize a sample case\n",
        "2. Download seed point labels\n",
        "2. Download 3d segmentation labels\n",
        "3. Download source images used to create seed points and segmentations\n",
        "4. Download source images with negative finding assessments\n",
        "\n",
        "To identify the subsets for our use cases, we'll leverage the **annotation metadata** spreadsheet the authors provided, which you can download manually from the **Annotation Summary** links in section 2 of the notebook or retrieve directly into a dataframe with the code below."
      ],
      "metadata": {
        "id": "aWUOGHyLkJIb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "metadata_urls = {\n",
        "    \"ACNS0332\": \"https://www.cancerimagingarchive.net/wp-content/uploads/ACNS0332_annotations_metadata-2023-08-03.csv\",\n",
        "    \"AHOD0831\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_AHOD0831_01222023.csv\",\n",
        "    \"AREN0532\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_AREN0532_01122023.csv\",\n",
        "    \"AREN0533\": \"https://www.cancerimagingarchive.net/wp-content/uploads/AREN0533_Annotations_Metadata__01-12-2023.csv\",\n",
        "    \"AREN0534\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_AREN0534_01122023.csv\",\n",
        "    \"CALGB50303\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_CALGB50303_02272023.csv\",\n",
        "    \"A091105\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_A091105_2023_11_06.csv\",\n",
        "    \"ARAR0331\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_ARAR0331_2023_11_13.csv\",\n",
        "    \"AHEP0731\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_AHEP0731_2024_1_3.csv\",\n",
        "    \"ACRIN-HNSCC-FDG-PET-CT\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_ACRIN-HNSCC_2023_11_07.csv\",\n",
        "    \"S0819\": \"https://www.cancerimagingarchive.net/wp-content/uploads/Metadata_Report_S0819_v3_2024_11_09-1.csv\"\n",
        "}\n",
        "\n",
        "if collection in metadata_urls:\n",
        "    spreadsheet_url = metadata_urls[collection]\n",
        "    annotation_Metadata = pd.read_csv(spreadsheet_url)\n",
        "    display(annotation_Metadata)\n",
        "else:\n",
        "    print(\"URL for collection not found.\")"
      ],
      "metadata": {
        "id": "Y2s8JKf74y-q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download and visualize a sample case\n",
        "Here we'll walk through some steps to identify an example segmentation file, find the corresponding reference series and visualize them together in the notebook.\n",
        "\n",
        "First, let's pull a series UID for a random segmentation from our annotation metadata."
      ],
      "metadata": {
        "id": "UmZ1gP5XsvD9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "random_row = annotation_Metadata.loc[annotation_Metadata['Annotation Type'] == 'Segmentation'].sample(n=1)\n",
        "\n",
        "segSeries = random_row['SeriesInstanceUID'].iloc[0]\n",
        "\n",
        "print(segSeries)"
      ],
      "metadata": {
        "id": "eBg-P-tvFXga"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this case we are fortunate to also have the corresponding reference Series UID in the spreadsheet so we can obtain that from our dataframe as well."
      ],
      "metadata": {
        "id": "fK3g_JmkuBi6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the row where 'SeriesInstanceUID' is equal to segSeries\n",
        "filtered_row = annotation_Metadata[annotation_Metadata['SeriesInstanceUID'] == segSeries]\n",
        "\n",
        "# Extract the value from 'ReferencedSeriesInstanceUID' column in the filtered row\n",
        "refSeries = filtered_row['ReferencedSeriesInstanceUID'].iloc[0]\n",
        "\n",
        "print(refSeries)"
      ],
      "metadata": {
        "id": "UKDaSEovuN9H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Alternatively, if you ever have a situation where you don't have a spreadsheet like this and want to determine the Reference Series UID you can use **getSegRefSeries()** to obtain it.  Note that it matches the UID from the spreadsheet in the previous step."
      ],
      "metadata": {
        "id": "8NVZAb3qu_X4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "refSeries = nbia.getSegRefSeries(segSeries)\n",
        "\n",
        "print(refSeries)"
      ],
      "metadata": {
        "id": "jogkmh9kvMoW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next let's download these two series.  Since we're working with Series UIDs saved as variables instead of JSON output from other API calls, we'll use the  **input_type = \"list\"** parameter in the remaining download steps.  "
      ],
      "metadata": {
        "id": "KkMkyHoYwC2F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.downloadSeries([refSeries, segSeries], input_type= \"list\")"
      ],
      "metadata": {
        "id": "1P-s8sZBwIIz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can look at the images and segmentation together with **viewSeriesAnnotation()** from [simpleDicomViewer](https://pypi.org/project/simpleDicomViewer/).  This function is only meant to be a  quick and dirty way to preview the data.  There are more comprehensive solutions such as [3D Slicer](https://slicer.org/) or [itkWidgets](https://github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_RTStruct_SEG_Visualization_with_itkWidgets.ipynb) if you want analyze the data.\n",
        "\n",
        "**Note:** Unfortunately this annotation visualization function doesn't support the type of DICOM SEG data that were generated for ACNS0332, but the images will still display.  These segmentations do work properly in 3D Slicer.  Make sure to install the [TCIA Browser extension](https://github.com/QIICR/TCIABrowser) first to easily access these data and the necessary Slicer depenencies to view RTSTRUCT/SEG data!"
      ],
      "metadata": {
        "id": "6HU0fGOwvxxO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Assuming you didn't change the default download options for downloadSeries\n",
        "imgPath = \"tciaDownload/\" + refSeries\n",
        "\n",
        "# The annotation path has to be a file name (not directory name).  Since there is generally\n",
        "# only one file in a segmentation series we can assume it will always be called 1-1.dcm\n",
        "segPath = \"tciaDownload/\" + segSeries + \"/1-1.dcm\"\n",
        "\n",
        "# Display the viewer\n",
        "dicomViewer.viewSeriesAnnotation(imgPath, segPath)"
      ],
      "metadata": {
        "id": "gZb33aCNv24K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download seed points\n",
        "These are setup to download a sample (2 scans).  Remove the **number** parameter to download the entire datasets.  We'll also specify a **csv_filename** to save the related metadata to a file."
      ],
      "metadata": {
        "id": "kgkWxt-QUM3i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include seed point rows\n",
        "seedPoints = annotation_Metadata[annotation_Metadata['Annotation Type'].str.contains('Seed point')]\n",
        "#display(seedPoints)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = seedPoints[\"SeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans\n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata\n",
        "nbia.downloadSeries(series_data, number = 2, input_type = \"list\", csv_filename = collection + \"-seedPoints\")"
      ],
      "metadata": {
        "id": "gIXTLO5dkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download 3D segmentations"
      ],
      "metadata": {
        "id": "1irWBSZvU05T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include segmentations\n",
        "segs = annotation_Metadata[annotation_Metadata['Annotation Type'].str.contains('Segmentation')]\n",
        "#display(segs)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = segs[\"SeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans\n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata\n",
        "nbia.downloadSeries(series_data, number = 2, input_type = \"list\", csv_filename = collection + \"-segs\")"
      ],
      "metadata": {
        "id": "_lG5TO7wkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Download source images for seed points and segmentations"
      ],
      "metadata": {
        "id": "9XsnZm2hkJIc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include seg and seed point rows (remove \"no findings\")\n",
        "ref_Series = annotation_Metadata[(annotation_Metadata['Annotation Type'] == 'Seed point') |\n",
        "                                 (annotation_Metadata['Annotation Type'] == 'Segmentation')]\n",
        "\n",
        "# remove duplicate ReferencedSeriesUIDs\n",
        "clean_refSeries = ref_Series.drop_duplicates(subset='ReferencedSeriesInstanceUID')\n",
        "#display(clean_refSeries)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = clean_refSeries[\"ReferencedSeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans\n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata\n",
        "nbia.downloadSeries(series_data, number = 2, input_type = \"list\", csv_filename = collection + \"-seg_seed_source_images\")"
      ],
      "metadata": {
        "id": "O4ivqjOHkJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following code will download the scans with negative finding assessments.  These are cases where the authors of the dataset did not find anything that could be annotated."
      ],
      "metadata": {
        "id": "f6exl7ORkJIc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# filter dataframe to only include scans with \"no findings\"\n",
        "ref_Series = annotation_Metadata[annotation_Metadata['Annotation Type'] == 'No findings']\n",
        "\n",
        "# remove duplicate ReferencedSeriesUIDs\n",
        "clean_refSeries = ref_Series.drop_duplicates(subset='ReferencedSeriesInstanceUID')\n",
        "#display(clean_refSeries)\n",
        "\n",
        "# extract series UID column to list for downloading\n",
        "series_data = clean_refSeries[\"ReferencedSeriesInstanceUID\"].tolist()\n",
        "\n",
        "# download a sample set of three scans\n",
        "# return metadata dataframe as dataframe\n",
        "# save a CSV of the metadata\n",
        "nbia.downloadSeries(series_data, number = 2, input_type = \"list\", csv_filename = collection + \"-noFinding_source_images\")"
      ],
      "metadata": {
        "id": "enSo8BLikJIc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Acknowledgements\n",
        "[The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) is a service which de-identifies and hosts a large publicly available archive of medical images of cancer.  TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/), and is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/).\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/).  If you leverage TCIA datasets in your work please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF) and include all relevant citations.\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7\n",
        "\n",
        "## Dataset Citations\n",
        "\n",
        "Instructions for citing the datasets can be found on their summary pages that are listed in section 1.  "
      ],
      "metadata": {
        "id": "DCOM4lO4_tET"
      }
    }
  ]
}