TCIA_Series_UID_Report.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_Series_UID_Report.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Series_UID_Report.ipynb)"
      ],
      "metadata": {
        "id": "OA_CwnKI1R4-"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KmXfYFZtja2F"
      },
      "source": [
        "# Summary\n",
        "\n",
        "This notebook can be used to generate reports to help understand the contents of TCIA datasets.  Currently we're focused on DICOM data hosted in NBIA but this may be expanded to additional data types later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqulqBEDMSS6"
      },
      "source": [
        "# 1 Setup\n",
        "\n",
        "Install the latest release of [**tcia_utils**](https://pypi.org/project/tcia-utils/) and import the modules we need."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YPrZ5pc-yYQl"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# install tcia utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [],
        "id": "nV7yJxgcKvab"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import pandas as pd\n",
        "import datetime\n",
        "from tcia_utils import nbia\n",
        "\n",
        "# set logging level to INFO in Google Colab (not necessary in Jupyter)\n",
        "if 'google.colab' in sys.modules:\n",
        "  import logging\n",
        "\n",
        "  for handler in logging.root.handlers[:]:\n",
        "      logging.root.removeHandler(handler)\n",
        "\n",
        "  # Set handler with level = info\n",
        "  logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s',\n",
        "                      level=logging.INFO)\n",
        "\n",
        "  print(\"Google Colab Logging set to INFO\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2uMbL1nwIJ5f"
      },
      "source": [
        "# 2 Create a Token (optional)\n",
        "If you're working with any restricted collections, you must enter your TCIA login/password to create a token.  If not, you can skip this step.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jgn4YvSa9GBZ"
      },
      "outputs": [],
      "source": [
        "nbia.getToken()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3 Create Collection or DOI-based summary reports\n",
        "\n",
        "This section will focus on analyzing a TCIA manifest or a spreadsheet/list of TCIA Series Instance UIDs.  \n",
        "\n",
        "[TCIA manifest files are used with the NBIA Data Retriever](https://wiki.cancerimagingarchive.net/x/egOnAg) to download DICOM data from TCIA.  Manifest files to download full collections can be found on their respective homepages.  Custom manifests can also be created via our search portal at https://nbia.cancerimagingarchive.net.\n",
        "\n",
        "It is also possible to use the Export Metadata function on the \"cart\" page of https://nbia.cancerimagingarchive.net or use the [REST API](https://wiki.cancerimagingarchive.net/x/NIIiAQ) to create spreadsheets or lists of Series Instance UIDs of interest.\n",
        "\n",
        "First we will create a series-level metadata report and then we can import that into the **reportCollectionSummary()** or **reportDoiSummary()** functions in **tcia_utils** to summarize the data including:\n",
        "\n",
        "    - Modalities: List of unique values by collection\n",
        "    - Licenses: List of unique values by collection\n",
        "    - Manufacturers: List of unique values in the collection\n",
        "    - Body Parts: List of unique values by collection\n",
        "    - Subjects: Number of subjects by collection\n",
        "    - Studies: Number of studies by collection\n",
        "    - Series: Number of series by collection\n",
        "    - Images: Number of images by collection\n",
        "    - Disk Space: Formatted as KB/MB/GB/TB/PB by collection\n",
        "\n",
        "Parameters:\n",
        "\n",
        "    series_data: The input data to be summarized (expects JSON by default).\n",
        "    input_type: Defaults to dataframe if not populated.  \n",
        "                Set to 'list' for python list, or 'manifest' for *.TCIA manifest file.\n",
        "                If manifest is used, series_data should be the path to the TCIA manifest file.\n",
        "    format (str): Output format (default is dataframe, 'csv' for CSV file, 'chart' for charts).\n",
        "    api_url: Only necessary if input_type = list or manifest.\n",
        "            Set to 'restricted' for limited-access collections or\n",
        "            'nlst' for National Lung Screening trial."
      ],
      "metadata": {
        "id": "pjEBz4IA_22C"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0EdHnIZTF3Xt"
      },
      "source": [
        "## 3.1 Prepare your Series UIDs\n",
        "\n",
        "To import a file to Colab from your hard drive, use the menu on the left sidebar to upload it and then proceed to the next section.\n",
        "\n",
        "To import a file from the web (e.g. TCIA), use the command in the next cell by updating it with the URL of the file you want to analyze.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jpbyz-Wi-MqK"
      },
      "outputs": [],
      "source": [
        "# OPTIONAL: only needed to directly import your UID file from the web\n",
        "url = \"https://cancerimagingarchive.net/wp-content/uploads/YourManifest.tcia\"\n",
        "downloaded_filename = \"YourManifest.tcia\"\n",
        "\n",
        "manifest = requests.get(url)\n",
        "with open(downloaded_filename, 'wb') as f:\n",
        "    f.write(manifest.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LHbh-qSHDhs3"
      },
      "source": [
        "Next we'll read in the UIDs from your file into a python list.  If you're using a manifest file, the code below will put the Series UIDs into a list while ignoring the parameter text.  \n",
        "\n",
        "If you're using a custom text/csv file of UIDs it will insert all rows into the list.  You must verify the file is formatted correctly **(one UID per row with no column header or commas)** or you may encounter errors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dyvgm8VRDzG-"
      },
      "outputs": [],
      "source": [
        "# enter manifest path/filename\n",
        "manifest = \"YourManifest.tcia\"\n",
        "\n",
        "# converts manifest to list of UIDs\n",
        "uids = nbia.manifestToList(manifest)\n",
        "\n",
        "print(\"Your data has been imported.\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHLc9W4Iei21"
      },
      "source": [
        "# 3.2 Download series metadata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pGDqJrkmNPJL"
      },
      "source": [
        "Using the next step you can create a dataframe and save **series_metadata.csv** containing the Collection Name, Subject ID, Study UID, Study Description, Study Date, Series UID, Series Description, Series Number, Number of Images, File Size (Bytes), Modality, Manufacturer, Data Description URI (DOI), 3rd party analysis status, License Name, and License URL for each series.\n",
        "\n",
        "**Note:** Due to its size (> 26,000 patients!) the [National Lung Screening Trial](https://doi.org/10.7937/TCIA.HMQ8-J677) resides on a separate server.  If you'd like to create a report about this collection use the 2nd option below."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# use empty api_url to default to main nbia server\n",
        "api_url = \"\"\n",
        "\n",
        "df = nbia.getSeriesList(uids)\n",
        "df = nbia.formatSeriesInput(df, input_type = \"df\", api_url = api_url)\n",
        "\n",
        "display(df)\n",
        "df.to_csv(\"series_metadata.csv\")"
      ],
      "metadata": {
        "id": "AEe04x3V02K6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# set api_url for NLST server\n",
        "api_url = \"nlst\"\n",
        "\n",
        "df = nbia.getSeriesList(uids, api_url = api_url)\n",
        "df = nbia.formatSeriesInput(df, input_type = \"df\", api_url = api_url)\n",
        "\n",
        "display(df)\n",
        "df.to_csv(\"series_metadata.csv\")"
      ],
      "metadata": {
        "id": "knY9A8MUZtRa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3.3 Create the summary report\n",
        "Now we can use the metadata we've downloaded to create the summary report.  Options are provided for creating a report to divide things up by collection or by DOI."
      ],
      "metadata": {
        "id": "ldXoxgIIaBSk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.reportCollectionSummary(df, format = \"chart\")"
      ],
      "metadata": {
        "id": "UPp0K5IpaSZl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "DOI-based reports are particularly useful when trying to understand manifests or series UID lists that contain [Analysis Result datasets](https://www.cancerimagingarchive.net/tcia-analysis-results/)."
      ],
      "metadata": {
        "id": "p289SdW6cOO9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nbia.reportDoiSummary(df, format = \"chart\")"
      ],
      "metadata": {
        "id": "Kp1iydx5cNEN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 4. Curator reports\n",
        "\n",
        "This section is geared at helping curators find issues with previously submitted data."
      ],
      "metadata": {
        "id": "15-gtm4gAXk3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4.1 Find series with missing DOIs, release dates or licensing information.\n",
        "\n",
        "This helps clean up data where these values may not have been properly stored during curation and publication.  First, log in with a token."
      ],
      "metadata": {
        "id": "nSL-fZXdAfiK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create a token to include restricted collections\n",
        "nbia.getToken()"
      ],
      "metadata": {
        "id": "n_tBNBoUA6Np"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Choose whether to assess all collections or specific collections.  \n",
        "\n",
        "**NOTE:** Run one of these cells, not both."
      ],
      "metadata": {
        "id": "huOcSy9CBr-5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# assess all collections\n",
        "collections = nbia.getCollections()\n",
        "\n",
        "# convert the Collection values in the json output to a list\n",
        "collections = [collection['Collection'] for collection in collections]\n",
        "\n",
        "print(len(collections), \"collections were selected.\")"
      ],
      "metadata": {
        "id": "PuS-2Rh3BVbO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# choose one or more specific collections\n",
        "collections = [\"LIDC-IDRI\", \"Breast-MRI-NACT-Pilot\"]"
      ],
      "metadata": {
        "id": "abReHDH0Bi8F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download the series metadata."
      ],
      "metadata": {
        "id": "ol0xrWBfCMwF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# compile series data for selected collections\n",
        "series = pd.DataFrame()\n",
        "\n",
        "for collection in collections:\n",
        "    seriesDescription = nbia.getSeries(collection)\n",
        "    series = pd.concat([series, pd.DataFrame(seriesDescription)], ignore_index=True)\n",
        "\n",
        "series.to_csv('series_metadata_{}.csv'.format(datetime.date.today()))"
      ],
      "metadata": {
        "id": "_bLLbTvMCA75"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create the reports about what is missing.  Summary info will be printed about any missing data and spreadsheets will be generated with details of the specific series affected."
      ],
      "metadata": {
        "id": "BSO4Kky0CPwO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter the dataframe to keep only the rows where CollectionURI is empty\n",
        "missing_CollectionURI = series[series['CollectionURI'].isnull() | (series['CollectionURI'] == '')]\n",
        "\n",
        "missing_CollectionURI.to_csv('missing_CollectionURI_series_metadata_{}.csv'.format(datetime.date.today()))\n",
        "\n",
        "missing_CollectionURI_collections = missing_CollectionURI['Collection'].unique().tolist()\n",
        "\n",
        "# print collections with missing data, if there are any\n",
        "if missing_CollectionURI_collections:\n",
        "  print(\"The following collections have missing DOIs:\", missing_CollectionURI_collections)\n",
        "else:\n",
        "  print(\"There are no collections with missing DOIs.\")\n"
      ],
      "metadata": {
        "id": "VFWUgf6CFKJR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter the dataframe to keep only the rows where DateReleased is empty\n",
        "missing_DateReleased = series[series['DateReleased'].isnull() | (series['DateReleased'] == '')]\n",
        "\n",
        "missing_DateReleased.to_csv('missing_DateReleased_series_metadata_{}.csv'.format(datetime.date.today()))\n",
        "\n",
        "missing_DateReleased_collections = missing_DateReleased['Collection'].unique().tolist()\n",
        "\n",
        "# print collections with missing data, if there are any\n",
        "if missing_DateReleased_collections:\n",
        "  print(\"The following collections have missing release date information:\", missing_DateReleased_collections)\n",
        "else:\n",
        "  print(\"There are no collections with missing release date information.\")\n"
      ],
      "metadata": {
        "id": "52bad1TYDkhq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter the dataframe to keep only the rows where LicenseName is empty\n",
        "missing_license = series[series['LicenseName'].isnull() | (series['LicenseName'] == '')]\n",
        "\n",
        "missing_license.to_csv('missing_license_series_metadata_{}.csv'.format(datetime.date.today()))\n",
        "\n",
        "missing_license_collections = missing_license['Collection'].unique().tolist()\n",
        "\n",
        "# print collections with missing data, if there are any\n",
        "if missing_license_collections:\n",
        "  print(\"The following collections have missing license information:\", missing_license_collections)\n",
        "else:\n",
        "  print(\"There are no collections with missing license information.\")"
      ],
      "metadata": {
        "id": "-72iM-yaCK4n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After applying fixes, you can update the specific collections you want to analyze in the first step and then re-run the remaining steps to be sure your changes were successful."
      ],
      "metadata": {
        "id": "Q8fr0TEnF-2h"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYXsNGcY93B8"
      },
      "source": [
        "# Acknowledgements\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/), and is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/).\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/).  If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to the following paper!\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}