notebooks/advanced_topics/clinical_data_intro.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/advanced_topics/clinical_data_intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_TcNSmCPUVC"
      },
      "source": [
        "# IDC clinical data exploration\n",
        "\n",
        "The goal of this notebook is to introduce the users of NCI Imaging Data Commons (IDC) to the organization of clinical data that accompany some of the IDC imaging data. For a quick visual summary of the clinical data available in IDC, please check out [this DataStudio dashboard](https://datastudio.google.com/u/0/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc).\n",
        "\n",
        "[NCI Imaging Data Commons (IDC)](https://imaging.datacommons.cancer.gov) is a cloud-based environment containing publicly available cancer imaging data co-located with analysis and exploration tools and resources. IDC is a node within the broader NCI Cancer Research Data Commons (CRDC) infrastructure that provides secure access to a large, comprehensive, and expanding collection of cancer research data.\n",
        "\n",
        "If you are not familiar with IDC, we recommend you first take a look at the [Getting started](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/) notebooks that are intended to serve as the introduction into working with IDC programmatically.\n",
        "\n",
        "If you have any questions about this tutorial, please post your questions on the [IDC user forum](https://discourse.canceridc.dev/) (preferred) or email IDC support at support@canceridc.dev!\n",
        "\n",
        "Authored by Andrey Fedorov and George White\n",
        "\n",
        "Prepared: July 2022\n",
        "\n",
        "Updated: Jan 2025"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "40SRMJpLRbx3"
      },
      "source": [
        "# Prerequisites\n",
        "\n",
        "The only prerequisite is [`idc-index`](https://github.com/ImagingDataCommons/idc-index) - python package that contains various utilities to simplify access to IDC data."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install --upgrade idc-index"
      ],
      "metadata": {
        "id": "QZMwa6Fg7QTm"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7BDaMjXykCZ"
      },
      "source": [
        "# Clinical data - background\n",
        "\n",
        "By clinical data we refer to the broad spectrum of image-related data that may accompany images. Such data may include demographics of the patients, observations related to their clinical history (therapies, diagnoses, findings), lab tests, surgeries.\n",
        "\n",
        "Clinical data is often critical in understanding imaging data, and is essential for the development and validation of imaging biomarkers. However, such data is most often stored in spreadsheets that follow conventions specific to the site that collected the data, may not be accompanied by the dictionary defining the terms used in describing clinical data, and is rarely harmonized. As an example, you can consider examing the clinical data that accompanies the [ACRIN 6698 collection](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=50135447) curated by The Cancer Imaging Archive (TCIA). File named \"Full Ancillary Patient Information file.xlsx\" linked from the collection page contains two sheets, one of which is the dictionary, and the othe one contains per-patient data with the columns defined by the dictionary.\n",
        "\n",
        "Not only the terms used in the clinical data accompanying individual collection are not harmonized, but the format of the spreadsheets is also collection-specific. In order to search and navigate clinical data, one has to parse those collection specific tables, and there is no interface to support searching across collections.\n",
        "\n",
        "With the release v11 of IDC, we make the attempt to lower the barriers for accessing clinical data accompanying IDC imaging collections. We parse collection-specific tables, and organize the underlying data into BigQuery tables that can be accessed using standard SQL queries. You can also see the summary of clinical data available for IDC collections in [this dashboard](https://datastudio.google.com/u/0/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc). Further, we make the content of the extracted clinical data available via the [`idc-index`](https://github.com/ImagingDataCommons/idc-index) python package.\n",
        "\n",
        "At the completion of this tutorial you will learn how IDC clinical data is organized, and how to write queries to interrogate this data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "29H1Kp5dTYBb"
      },
      "source": [
        "# Organization of clinical data in IDC\n",
        "\n",
        "`idc-index` packages _indices_ - tables containing key metadata describing data available in IDC. The main index that supports API calls related to download and search is installed by default. To support search of the clinical data accompanying IDC images you will need the `clinical_index` table, which helps navigating clinical data accompanying images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "TLhEHQkNWkac",
        "outputId": "344eca93-377c-48cf-e35f-2fd79dd7de4f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Columns avaialable in clinical_index:\n",
            "collection_id\n",
            "table_name\n",
            "short_table_name\n",
            "column\n",
            "column_label\n",
            "values\n"
          ]
        }
      ],
      "source": [
        "from idc_index import IDCClient\n",
        "\n",
        "c = IDCClient()\n",
        "\n",
        "c.fetch_index('clinical_index')\n",
        "\n",
        "print('Columns avaialable in clinical_index:\\n'+'\\n'.join(c.clinical_index.keys()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9Vgq-gtI9G4"
      },
      "source": [
        "This table is documented in https://idc-index.readthedocs.io/en/latest/column_descriptions.html#clinical-index.\n",
        "\n",
        "\n",
        "Accessing relevant clinical data for a given collection is a two-step process:\n",
        "\n",
        "* Step 1: use `clinical_index` table to identify relevant metadata attributes and the names of the tables where the corresponding metadata is located.\n",
        "* Step 2: load the specific clinical data table with the selected attribute referenced from `clinical_index` and access clinical metadata for the individual patients.\n",
        "\n",
        "As a reminder, all of the data - including clinical metadata - in IDC is anonymized!\n",
        "\n",
        "For the sake of this example, we will start with identifying clinical data attributes that accompany NLST collection.\n",
        "\n",
        "`collection_id` column of `clinical_index` can be used to associate clinical data attribute to the collection it accompanies."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Understanding individual clinical data attributes\n",
        "\n",
        "Here's an example of the columns that correspond to the clinical metadata in one of the ACRIN collections, where `column` is not particularly helpful, but `column_label` provides human readable information to allow interpretation of the column."
      ],
      "metadata": {
        "id": "CM-vI0FJClLy"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "nGsS6SvPdrWn",
        "outputId": "2f888f38-daff-410f-d150-96a7575b1265",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 653
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            collection_id                                   short_table_name  \\\n",
              "2359  acrin_nsclc_fdg_pet                             acrin_nsclc_fdg_pet_A0   \n",
              "2360  acrin_nsclc_fdg_pet                             acrin_nsclc_fdg_pet_A0   \n",
              "2361  acrin_nsclc_fdg_pet                             acrin_nsclc_fdg_pet_A0   \n",
              "2362  acrin_nsclc_fdg_pet                             acrin_nsclc_fdg_pet_A0   \n",
              "2363  acrin_nsclc_fdg_pet                             acrin_nsclc_fdg_pet_A0   \n",
              "...                   ...                                                ...   \n",
              "3643  acrin_nsclc_fdg_pet  acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation   \n",
              "3644  acrin_nsclc_fdg_pet  acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation   \n",
              "3645  acrin_nsclc_fdg_pet  acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation   \n",
              "3646  acrin_nsclc_fdg_pet  acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation   \n",
              "3647  acrin_nsclc_fdg_pet  acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation   \n",
              "\n",
              "                           column  \\\n",
              "2359                      inst_no   \n",
              "2360                     entryage   \n",
              "2361                        a0e26   \n",
              "2362                          rec   \n",
              "2363                        a0e4d   \n",
              "...                           ...   \n",
              "3643             studyinstanceuid   \n",
              "3644          ptseriesinstanceuid   \n",
              "3645          ctseriesinstanceuid   \n",
              "3646               aisegmentation   \n",
              "3647  commentsaboutaisegmentation   \n",
              "\n",
              "                                           column_label  \\\n",
              "2359                   De-identified Institution Number   \n",
              "2360                        Patient age at Registration   \n",
              "2361                            RTOG INSTITUTION NUMBER   \n",
              "2362                      Data receipt (from base date)   \n",
              "2363  \"Days from Base_dt to DATE THE  STUDY-SPECIFIC...   \n",
              "...                                                 ...   \n",
              "3643                                   StudyInstanceUID   \n",
              "3644                                PTSeriesInstanceUID   \n",
              "3645                                CTSeriesInstanceUID   \n",
              "3646                                     AISegmentation   \n",
              "3647                        CommentsAboutAISegmentation   \n",
              "\n",
              "                                                 values  \n",
              "2359  [{'option_code': '\"nan\"', 'option_description'...  \n",
              "2360  [{'option_code': '\"nan\"', 'option_description'...  \n",
              "2361  [{'option_code': '\"nan\"', 'option_description'...  \n",
              "2362  [{'option_code': '\"nan\"', 'option_description'...  \n",
              "2363  [{'option_code': '\"nan\"', 'option_description'...  \n",
              "...                                                 ...  \n",
              "3643                                                 []  \n",
              "3644                                                 []  \n",
              "3645                                                 []  \n",
              "3646                                                 []  \n",
              "3647                                                 []  \n",
              "\n",
              "[1289 rows x 5 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c19868ab-06c0-48fd-8619-1f65ede862ed\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>short_table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2359</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_A0</td>\n",
              "      <td>inst_no</td>\n",
              "      <td>De-identified Institution Number</td>\n",
              "      <td>[{'option_code': '\"nan\"', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2360</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_A0</td>\n",
              "      <td>entryage</td>\n",
              "      <td>Patient age at Registration</td>\n",
              "      <td>[{'option_code': '\"nan\"', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2361</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_A0</td>\n",
              "      <td>a0e26</td>\n",
              "      <td>RTOG INSTITUTION NUMBER</td>\n",
              "      <td>[{'option_code': '\"nan\"', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2362</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_A0</td>\n",
              "      <td>rec</td>\n",
              "      <td>Data receipt (from base date)</td>\n",
              "      <td>[{'option_code': '\"nan\"', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2363</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_A0</td>\n",
              "      <td>a0e4d</td>\n",
              "      <td>\"Days from Base_dt to DATE THE  STUDY-SPECIFIC...</td>\n",
              "      <td>[{'option_code': '\"nan\"', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3643</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation</td>\n",
              "      <td>studyinstanceuid</td>\n",
              "      <td>StudyInstanceUID</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3644</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation</td>\n",
              "      <td>ptseriesinstanceuid</td>\n",
              "      <td>PTSeriesInstanceUID</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3645</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation</td>\n",
              "      <td>ctseriesinstanceuid</td>\n",
              "      <td>CTSeriesInstanceUID</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3646</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation</td>\n",
              "      <td>aisegmentation</td>\n",
              "      <td>AISegmentation</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3647</th>\n",
              "      <td>acrin_nsclc_fdg_pet</td>\n",
              "      <td>acrin_nsclc_fdg_pet_bamf_lung_pet_ct_segmentation</td>\n",
              "      <td>commentsaboutaisegmentation</td>\n",
              "      <td>CommentsAboutAISegmentation</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1289 rows × 5 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c19868ab-06c0-48fd-8619-1f65ede862ed')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c19868ab-06c0-48fd-8619-1f65ede862ed button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c19868ab-06c0-48fd-8619-1f65ede862ed');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-eaa84ef7-34b8-443b-a32e-051b4e5b2093\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-eaa84ef7-34b8-443b-a32e-051b4e5b2093')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-eaa84ef7-34b8-443b-a32e-051b4e5b2093 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"acrin_nsclc_fdg_pet_clinical_columns[['collection_id','short_table_name','column','column_label','values']]\",\n  \"rows\": 1289,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"acrin_nsclc_fdg_pet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"short_table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 21,\n        \"samples\": [\n          \"acrin_nsclc_fdg_pet_A0\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1211,\n        \"samples\": [\n          \"aee34\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 789,\n        \"samples\": [\n          \"AREAS OF INTEREST WITHIN MRI SCAN - CHEST - IMAGING DATE\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "acrin_nsclc_fdg_pet_clinical_columns = c.clinical_index[c.clinical_index['collection_id']=='acrin_nsclc_fdg_pet']\n",
        "acrin_nsclc_fdg_pet_clinical_columns[['collection_id','short_table_name','column','column_label','values']]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0H78L-VdxHn"
      },
      "source": [
        "For some columns, the values come from a defined set. In the example above, we can, for example, examine the values assigned to encode patient race."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "9NCP15tRd2gU",
        "outputId": "b92fdc9a-7f19-406b-d885-0d27250154c5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([{'option_code': '1.0', 'option_description': 'American Indian or Alaska Native'},\n",
              "        {'option_code': '2.0', 'option_description': 'Asian'},\n",
              "        {'option_code': '3.0', 'option_description': 'Black or African American'},\n",
              "        {'option_code': '4.0', 'option_description': 'Native Hawaiian or other Pacific Islander'},\n",
              "        {'option_code': '5.0', 'option_description': 'White'},\n",
              "        {'option_code': '6.0', 'option_description': 'More than one race'},\n",
              "        {'option_code': '9.0', 'option_description': 'Unknown'}],\n",
              "       dtype=object)]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "race_values = acrin_nsclc_fdg_pet_clinical_columns[acrin_nsclc_fdg_pet_clinical_columns[\"column_label\"] == \"RACE\"][\"values\"]\n",
        "\n",
        "race_values.tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yaoJvf2nd6wL"
      },
      "source": [
        "On the other hand, if we look at the metadata available for the `c4kc_kits` collection, `column_label` and `column` are identical.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "AYMJ8kjTd9eb",
        "outputId": "1ce1396b-66f6-4b22-e372-c3848e5a52e6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 303
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     collection_id                                         table_name  \\\n",
              "3828     c4kc_kits  bigquery-public-data.idc_v20_clinical.c4kc_kit...   \n",
              "3829     c4kc_kits  bigquery-public-data.idc_v20_clinical.c4kc_kit...   \n",
              "3830     c4kc_kits  bigquery-public-data.idc_v20_clinical.c4kc_kit...   \n",
              "\n",
              "        short_table_name                                       column  \\\n",
              "3828  c4kc_kits_clinical                      comorbidities__dementia   \n",
              "3829  c4kc_kits_clinical                          comorbidities__aids   \n",
              "3830  c4kc_kits_clinical  intraoperative_complications__cardiac_event   \n",
              "\n",
              "                                     column_label  \\\n",
              "3828                      comorbidities__dementia   \n",
              "3829                          comorbidities__aids   \n",
              "3830  intraoperative_complications__cardiac_event   \n",
              "\n",
              "                                                 values  \n",
              "3828  [{'option_code': 'False', 'option_description'...  \n",
              "3829  [{'option_code': 'False', 'option_description'...  \n",
              "3830  [{'option_code': 'False', 'option_description'...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ad8384cf-190c-4879-842c-66a9ba0c6c4f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>table_name</th>\n",
              "      <th>short_table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3828</th>\n",
              "      <td>c4kc_kits</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.c4kc_kit...</td>\n",
              "      <td>c4kc_kits_clinical</td>\n",
              "      <td>comorbidities__dementia</td>\n",
              "      <td>comorbidities__dementia</td>\n",
              "      <td>[{'option_code': 'False', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3829</th>\n",
              "      <td>c4kc_kits</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.c4kc_kit...</td>\n",
              "      <td>c4kc_kits_clinical</td>\n",
              "      <td>comorbidities__aids</td>\n",
              "      <td>comorbidities__aids</td>\n",
              "      <td>[{'option_code': 'False', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3830</th>\n",
              "      <td>c4kc_kits</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.c4kc_kit...</td>\n",
              "      <td>c4kc_kits_clinical</td>\n",
              "      <td>intraoperative_complications__cardiac_event</td>\n",
              "      <td>intraoperative_complications__cardiac_event</td>\n",
              "      <td>[{'option_code': 'False', 'option_description'...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ad8384cf-190c-4879-842c-66a9ba0c6c4f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ad8384cf-190c-4879-842c-66a9ba0c6c4f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ad8384cf-190c-4879-842c-66a9ba0c6c4f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6915e002-eb06-45d4-8395-839d65258c39\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6915e002-eb06-45d4-8395-839d65258c39')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6915e002-eb06-45d4-8395-839d65258c39 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"c\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"c4kc_kits\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"bigquery-public-data.idc_v20_clinical.c4kc_kits_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"short_table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"c4kc_kits_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"comorbidities__dementia\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"comorbidities__dementia\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "c.clinical_index[c.clinical_index[\"collection_id\"] == \"c4kc_kits\"][:3]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voowbwXCeFBF"
      },
      "source": [
        "**As a general rule of thumb**:\n",
        "* when selecting specific columns from clinical tables, use `column` values\n",
        "* when searching for concepts of interest in `column_metadata`, use `column_label`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gWgLxoraN3mY"
      },
      "source": [
        "# Exploring IDC clinical data\n",
        "\n",
        "In the following sections of the notebook we go over some use cases to demonstrate various options for navigating IDC clinical data.\n",
        "\n",
        "As always, if you have a use case that is not addressed here, if you have suggestions or are confused - please start a discussion thread on the [IDC User forum](https://discourse.canceridc.dev/)!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "03rwe7OfvsnV"
      },
      "source": [
        "## Collection-focused exploration\n",
        "\n",
        "If you used clinical data stored in TCIA, perhaps you started with a specific collection of your interest, downloaded clinical data files for that collection, parsed them into your code. Let's go over those steps the IDC way!\n",
        "\n",
        "First, let's see which of the collections in IDC have clinical data in BigQuery. To do that we will reuse the pandas dataframe with the results of the query we executed earlier.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "TIZq1GbQoTIC",
        "outputId": "ef45376f-8c6e-4eed-910f-a76e0badae46",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['acrin_6698',\n",
              " 'acrin_contralateral_breast_mr',\n",
              " 'acrin_flt_breast',\n",
              " 'acrin_nsclc_fdg_pet',\n",
              " 'adrenal_acc_ki67_seg',\n",
              " 'advanced_mri_breast_lesions',\n",
              " 'anti_pd_1_lung',\n",
              " 'b_mode_and_ceus_liver',\n",
              " 'breast_diagnosis',\n",
              " 'breast_mri_nact_pilot',\n",
              " 'c4kc_kits',\n",
              " 'cc_tumor_heterogeneity',\n",
              " 'cmmd',\n",
              " 'colorectal_liver_metastases',\n",
              " 'covid_19_ar',\n",
              " 'covid_19_ny_sbu',\n",
              " 'cptac_brca',\n",
              " 'cptac_ccrcc',\n",
              " 'cptac_coad',\n",
              " 'cptac_gbm',\n",
              " 'cptac_hnscc',\n",
              " 'cptac_lscc',\n",
              " 'cptac_luad',\n",
              " 'cptac_ov',\n",
              " 'cptac_pda',\n",
              " 'cptac_ucec',\n",
              " 'ctpred_sunitinib_pannet',\n",
              " 'duke_breast_cancer_mri',\n",
              " 'ea1141',\n",
              " 'hcc_tace_seg',\n",
              " 'htan_hms',\n",
              " 'htan_ohsu',\n",
              " 'htan_vanderbilt',\n",
              " 'htan_wustl',\n",
              " 'ispy1',\n",
              " 'ispy2',\n",
              " 'lidc_idri',\n",
              " 'lung_fused_ct_pathology',\n",
              " 'lung_pet_ct_dx',\n",
              " 'mediastinal_lymph_node_seg',\n",
              " 'midrc_ricord_1a',\n",
              " 'midrc_ricord_1b',\n",
              " 'midrc_ricord_1c',\n",
              " 'nlst',\n",
              " 'nsclc_radiogenomics',\n",
              " 'nsclc_radiomics',\n",
              " 'nsclc_radiomics_genomics',\n",
              " 'nsclc_radiomics_interobserver1',\n",
              " 'prostate_diagnosis',\n",
              " 'prostatex',\n",
              " 'qin_breast',\n",
              " 'remind',\n",
              " 'rider_lung_pet_ct',\n",
              " 'rms_mutation_prediction',\n",
              " 'soft_tissue_sarcoma',\n",
              " 'spie_aapm_lung_ct_challenge',\n",
              " 'spine_mets_ct_seg',\n",
              " 'tcga_acc',\n",
              " 'tcga_blca',\n",
              " 'tcga_brca',\n",
              " 'tcga_cesc',\n",
              " 'tcga_chol',\n",
              " 'tcga_coad',\n",
              " 'tcga_dlbc',\n",
              " 'tcga_esca',\n",
              " 'tcga_gbm',\n",
              " 'tcga_hnsc',\n",
              " 'tcga_kich',\n",
              " 'tcga_kirc',\n",
              " 'tcga_kirp',\n",
              " 'tcga_lgg',\n",
              " 'tcga_lihc',\n",
              " 'tcga_luad',\n",
              " 'tcga_lusc',\n",
              " 'tcga_meso',\n",
              " 'tcga_ov',\n",
              " 'tcga_paad',\n",
              " 'tcga_pcpg',\n",
              " 'tcga_prad',\n",
              " 'tcga_read',\n",
              " 'tcga_sarc',\n",
              " 'tcga_skcm',\n",
              " 'tcga_stad',\n",
              " 'tcga_tgct',\n",
              " 'tcga_thca',\n",
              " 'tcga_thym',\n",
              " 'tcga_ucec',\n",
              " 'tcga_ucs',\n",
              " 'tcga_uvm',\n",
              " 'upenn_gbm']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "c.clinical_index[\"collection_id\"].unique().tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_KoEJ_Wvwa79"
      },
      "source": [
        "If you are interested in what clinical data is available for the specific collection, you can select only the rows corresponding to that collection in the `column_metadata` table. Here we select a subset of columns to improve readability of the dataframe.\n",
        "\n",
        "Note that for some collections, clinical data sheets are accompanied by dictionaries, which formalize the values encountered. Examples of such collections are [ISPY1](https://wiki.cancerimagingarchive.net/display/Public/ISPY1) or ACRIN trials.\n",
        "\n",
        "For many other collections there are no such dictionaries available. In those situations, the values you will see in the `values` columns have been derived by examining the distinct values encountered in the clinical data sheets.\n",
        "\n",
        "In the following we look at the clinical data columns (\"dictionary terms\") for the [ACRIN 6698 collection](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=50135447) mentioned earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "zrtRWr_SwmIJ",
        "outputId": "b16a2902-be23-4b1d-bb96-2151271eaa4d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   collection_id                                         table_name  \\\n",
              "0     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "1     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "2     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "3     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "4     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "5     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "6     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "7     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "8     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "9     acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "10    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "11    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "12    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "13    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "14    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "15    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "16    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "17    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "18    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "19    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "20    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "21    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "22    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "23    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "24    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "25    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "26    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "27    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "28    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "29    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "30    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "31    acrin_6698  bigquery-public-data.idc_v20_clinical.acrin_66...   \n",
              "\n",
              "       short_table_name                column  \\\n",
              "0   acrin_6698_clinical          source_batch   \n",
              "1   acrin_6698_clinical                    t0   \n",
              "2   acrin_6698_clinical                    t1   \n",
              "3   acrin_6698_clinical                    t2   \n",
              "4   acrin_6698_clinical                    t3   \n",
              "5   acrin_6698_clinical           bmmr2_train   \n",
              "6   acrin_6698_clinical            bmmr2_test   \n",
              "7   acrin_6698_clinical        primary_aim_t0   \n",
              "8   acrin_6698_clinical        primary_aim_t1   \n",
              "9   acrin_6698_clinical        primary_aim_t2   \n",
              "10  acrin_6698_clinical        primary_aim_t3   \n",
              "11  acrin_6698_clinical       analysis_cohort   \n",
              "12  acrin_6698_clinical            eligigible   \n",
              "13  acrin_6698_clinical           test_retest   \n",
              "14  acrin_6698_clinical       qc_pass_fail_t0   \n",
              "15  acrin_6698_clinical       qc_pass_fail_t1   \n",
              "16  acrin_6698_clinical       qc_pass_fail_t2   \n",
              "17  acrin_6698_clinical       qc_pass_fail_t3   \n",
              "18  acrin_6698_clinical  qc_roi_confidence_t0   \n",
              "19  acrin_6698_clinical  qc_roi_confidence_t1   \n",
              "20  acrin_6698_clinical  qc_roi_confidence_t2   \n",
              "21  acrin_6698_clinical  qc_roi_confidence_t3   \n",
              "22  acrin_6698_clinical                  race   \n",
              "23  acrin_6698_clinical                 ltype   \n",
              "24  acrin_6698_clinical               hrher4g   \n",
              "25  acrin_6698_clinical              sbrgrade   \n",
              "26  acrin_6698_clinical                   pcr   \n",
              "27  acrin_6698_clinical      dicom_patient_id   \n",
              "28  acrin_6698_clinical   i_spy_2_research_id   \n",
              "29  acrin_6698_clinical       tcia_patient_id   \n",
              "30  acrin_6698_clinical                   age   \n",
              "31  acrin_6698_clinical                  mrld   \n",
              "\n",
              "                                         column_label  \\\n",
              "0                         idc_provenance_source_batch   \n",
              "1      T0 (baseline) MRI study included in collection   \n",
              "2      T1 (early-Tx) MRI study included in collection   \n",
              "3   T2 (inter-regimen) MRI study included in colle...   \n",
              "4   T3 (pre-surgery) MRI study included in collection   \n",
              "5   Patient included in the BMMR2 challenge traini...   \n",
              "6   Patient included in the BMMR2 challenge test c...   \n",
              "7   T0 MRI study included in the ACRIN-6698 primar...   \n",
              "8   T1 MRI study included in the ACRIN-6698 primar...   \n",
              "9   T2 MRI study included in the ACRIN-6698 primar...   \n",
              "10  T3 MRI study included in the ACRIN-6698 primar...   \n",
              "11         Patient in primary analysis cohort (n=242)   \n",
              "12                                         eligigible   \n",
              "13  Study time point that includes test/retest DWI...   \n",
              "14             T0 MRI DWI Quality control assessment:   \n",
              "15             T0 MRI DWI Quality control assessment:   \n",
              "16             T0 MRI DWI Quality control assessment:   \n",
              "17             T0 MRI DWI Quality control assessment:   \n",
              "18  T0 quality control confidence level for whole-...   \n",
              "19  T1 quality control confidence level for whole-...   \n",
              "20  T2 quality control confidence level for whole-...   \n",
              "21  T3 quality control confidence level for whole-...   \n",
              "22                                      Patient race:   \n",
              "23                                 Index lesion type:   \n",
              "24             Hormone receptor (HR) and HER2 status:   \n",
              "25                                       Tumor grade:   \n",
              "26      Pathologic complete response (pCR) at surgery   \n",
              "27                    idc_provenance_dicom_patient_id   \n",
              "28      Double-blinded patient ID number from I-SPY 2   \n",
              "29  Patient identification string in TCIA collecti...   \n",
              "30                         Age in years at enrollment   \n",
              "31  MRI measured longest diameter (cm) at baseline...   \n",
              "\n",
              "                                               values  \n",
              "0   [{'option_code': '0', 'option_description': No...  \n",
              "1   [{'option_code': '0', 'option_description': ' ...  \n",
              "2   [{'option_code': '0', 'option_description': ' ...  \n",
              "3   [{'option_code': '0', 'option_description': ' ...  \n",
              "4   [{'option_code': '0', 'option_description': ' ...  \n",
              "5   [{'option_code': '0', 'option_description': ' ...  \n",
              "6   [{'option_code': '0', 'option_description': ' ...  \n",
              "7   [{'option_code': '0', 'option_description': ' ...  \n",
              "8   [{'option_code': '0', 'option_description': ' ...  \n",
              "9   [{'option_code': '0', 'option_description': ' ...  \n",
              "10  [{'option_code': '0', 'option_description': ' ...  \n",
              "11  [{'option_code': '0', 'option_description': ' ...  \n",
              "12  [{'option_code': '0', 'option_description': No...  \n",
              "13  [{'option_code': 'T0', 'option_description': N...  \n",
              "14  [{'option_code': 'PASS', 'option_description':...  \n",
              "15  [{'option_code': 'PASS', 'option_description':...  \n",
              "16  [{'option_code': 'PASS', 'option_description':...  \n",
              "17  [{'option_code': 'PASS', 'option_description':...  \n",
              "18  [{'option_code': 'HIGH', 'option_description':...  \n",
              "19  [{'option_code': 'HIGH', 'option_description':...  \n",
              "20  [{'option_code': 'HIGH', 'option_description':...  \n",
              "21  [{'option_code': 'HIGH', 'option_description':...  \n",
              "22  [{'option_code': 'White', 'option_description'...  \n",
              "23  [{'option_code': 'Single mass', 'option_descri...  \n",
              "24  [{'option_code': 'HR + / HER2 +', 'option_desc...  \n",
              "25  [{'option_code': 'I (Low)', 'option_descriptio...  \n",
              "26  [{'option_code': 'non-pCR', 'option_descriptio...  \n",
              "27                                                 []  \n",
              "28                                                 []  \n",
              "29                                                 []  \n",
              "30                                                 []  \n",
              "31                                                 []  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f21ecf89-f728-408b-a5d4-82b2b9721c87\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>table_name</th>\n",
              "      <th>short_table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>source_batch</td>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "      <td>[{'option_code': '0', 'option_description': No...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t0</td>\n",
              "      <td>T0 (baseline) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t1</td>\n",
              "      <td>T1 (early-Tx) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t2</td>\n",
              "      <td>T2 (inter-regimen) MRI study included in colle...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t3</td>\n",
              "      <td>T3 (pre-surgery) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>bmmr2_train</td>\n",
              "      <td>Patient included in the BMMR2 challenge traini...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>bmmr2_test</td>\n",
              "      <td>Patient included in the BMMR2 challenge test c...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t0</td>\n",
              "      <td>T0 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t1</td>\n",
              "      <td>T1 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t2</td>\n",
              "      <td>T2 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t3</td>\n",
              "      <td>T3 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>analysis_cohort</td>\n",
              "      <td>Patient in primary analysis cohort (n=242)</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>eligigible</td>\n",
              "      <td>eligigible</td>\n",
              "      <td>[{'option_code': '0', 'option_description': No...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>test_retest</td>\n",
              "      <td>Study time point that includes test/retest DWI...</td>\n",
              "      <td>[{'option_code': 'T0', 'option_description': N...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t0</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t1</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t2</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t3</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t0</td>\n",
              "      <td>T0 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t1</td>\n",
              "      <td>T1 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t2</td>\n",
              "      <td>T2 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t3</td>\n",
              "      <td>T3 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>race</td>\n",
              "      <td>Patient race:</td>\n",
              "      <td>[{'option_code': 'White', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>ltype</td>\n",
              "      <td>Index lesion type:</td>\n",
              "      <td>[{'option_code': 'Single mass', 'option_descri...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>hrher4g</td>\n",
              "      <td>Hormone receptor (HR) and HER2 status:</td>\n",
              "      <td>[{'option_code': 'HR + / HER2 +', 'option_desc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>sbrgrade</td>\n",
              "      <td>Tumor grade:</td>\n",
              "      <td>[{'option_code': 'I (Low)', 'option_descriptio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>pcr</td>\n",
              "      <td>Pathologic complete response (pCR) at surgery</td>\n",
              "      <td>[{'option_code': 'non-pCR', 'option_descriptio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>dicom_patient_id</td>\n",
              "      <td>idc_provenance_dicom_patient_id</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>i_spy_2_research_id</td>\n",
              "      <td>Double-blinded patient ID number from I-SPY 2</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>tcia_patient_id</td>\n",
              "      <td>Patient identification string in TCIA collecti...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>age</td>\n",
              "      <td>Age in years at enrollment</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.acrin_66...</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>mrld</td>\n",
              "      <td>MRI measured longest diameter (cm) at baseline...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f21ecf89-f728-408b-a5d4-82b2b9721c87')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f21ecf89-f728-408b-a5d4-82b2b9721c87 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f21ecf89-f728-408b-a5d4-82b2b9721c87');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-74b7689b-126c-49f8-8297-8f4e4d6f02c4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-74b7689b-126c-49f8-8297-8f4e4d6f02c4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-74b7689b-126c-49f8-8297-8f4e4d6f02c4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_6e2ef9d9-d7bb-46eb-ba22-f750713f2f77\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('acrin6698_clinical_columns')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_6e2ef9d9-d7bb-46eb-ba22-f750713f2f77 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('acrin6698_clinical_columns');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "acrin6698_clinical_columns",
              "summary": "{\n  \"name\": \"acrin6698_clinical_columns\",\n  \"rows\": 32,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"acrin_6698\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"bigquery-public-data.idc_v20_clinical.acrin_6698_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"short_table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"acrin_6698_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 32,\n        \"samples\": [\n          \"tcia_patient_id\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 29,\n        \"samples\": [\n          \"Age in years at enrollment\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "acrin6698_clinical_columns = c.clinical_index[c.clinical_index[\"collection_id\"] == \"acrin_6698\"]\n",
        "\n",
        "acrin6698_clinical_columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dm5cAn73SQJW"
      },
      "source": [
        "Here's how you can select just the specific columns in the table - this way it is easier to examine the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "4mY1gbR7ysxx",
        "outputId": "62e65454-2faa-43f1-9155-c6779c94cec9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   collection_id     short_table_name                column  \\\n",
              "0     acrin_6698  acrin_6698_clinical          source_batch   \n",
              "1     acrin_6698  acrin_6698_clinical                    t0   \n",
              "2     acrin_6698  acrin_6698_clinical                    t1   \n",
              "3     acrin_6698  acrin_6698_clinical                    t2   \n",
              "4     acrin_6698  acrin_6698_clinical                    t3   \n",
              "5     acrin_6698  acrin_6698_clinical           bmmr2_train   \n",
              "6     acrin_6698  acrin_6698_clinical            bmmr2_test   \n",
              "7     acrin_6698  acrin_6698_clinical        primary_aim_t0   \n",
              "8     acrin_6698  acrin_6698_clinical        primary_aim_t1   \n",
              "9     acrin_6698  acrin_6698_clinical        primary_aim_t2   \n",
              "10    acrin_6698  acrin_6698_clinical        primary_aim_t3   \n",
              "11    acrin_6698  acrin_6698_clinical       analysis_cohort   \n",
              "12    acrin_6698  acrin_6698_clinical            eligigible   \n",
              "13    acrin_6698  acrin_6698_clinical           test_retest   \n",
              "14    acrin_6698  acrin_6698_clinical       qc_pass_fail_t0   \n",
              "15    acrin_6698  acrin_6698_clinical       qc_pass_fail_t1   \n",
              "16    acrin_6698  acrin_6698_clinical       qc_pass_fail_t2   \n",
              "17    acrin_6698  acrin_6698_clinical       qc_pass_fail_t3   \n",
              "18    acrin_6698  acrin_6698_clinical  qc_roi_confidence_t0   \n",
              "19    acrin_6698  acrin_6698_clinical  qc_roi_confidence_t1   \n",
              "20    acrin_6698  acrin_6698_clinical  qc_roi_confidence_t2   \n",
              "21    acrin_6698  acrin_6698_clinical  qc_roi_confidence_t3   \n",
              "22    acrin_6698  acrin_6698_clinical                  race   \n",
              "23    acrin_6698  acrin_6698_clinical                 ltype   \n",
              "24    acrin_6698  acrin_6698_clinical               hrher4g   \n",
              "25    acrin_6698  acrin_6698_clinical              sbrgrade   \n",
              "26    acrin_6698  acrin_6698_clinical                   pcr   \n",
              "27    acrin_6698  acrin_6698_clinical      dicom_patient_id   \n",
              "28    acrin_6698  acrin_6698_clinical   i_spy_2_research_id   \n",
              "29    acrin_6698  acrin_6698_clinical       tcia_patient_id   \n",
              "30    acrin_6698  acrin_6698_clinical                   age   \n",
              "31    acrin_6698  acrin_6698_clinical                  mrld   \n",
              "\n",
              "                                         column_label  \\\n",
              "0                         idc_provenance_source_batch   \n",
              "1      T0 (baseline) MRI study included in collection   \n",
              "2      T1 (early-Tx) MRI study included in collection   \n",
              "3   T2 (inter-regimen) MRI study included in colle...   \n",
              "4   T3 (pre-surgery) MRI study included in collection   \n",
              "5   Patient included in the BMMR2 challenge traini...   \n",
              "6   Patient included in the BMMR2 challenge test c...   \n",
              "7   T0 MRI study included in the ACRIN-6698 primar...   \n",
              "8   T1 MRI study included in the ACRIN-6698 primar...   \n",
              "9   T2 MRI study included in the ACRIN-6698 primar...   \n",
              "10  T3 MRI study included in the ACRIN-6698 primar...   \n",
              "11         Patient in primary analysis cohort (n=242)   \n",
              "12                                         eligigible   \n",
              "13  Study time point that includes test/retest DWI...   \n",
              "14             T0 MRI DWI Quality control assessment:   \n",
              "15             T0 MRI DWI Quality control assessment:   \n",
              "16             T0 MRI DWI Quality control assessment:   \n",
              "17             T0 MRI DWI Quality control assessment:   \n",
              "18  T0 quality control confidence level for whole-...   \n",
              "19  T1 quality control confidence level for whole-...   \n",
              "20  T2 quality control confidence level for whole-...   \n",
              "21  T3 quality control confidence level for whole-...   \n",
              "22                                      Patient race:   \n",
              "23                                 Index lesion type:   \n",
              "24             Hormone receptor (HR) and HER2 status:   \n",
              "25                                       Tumor grade:   \n",
              "26      Pathologic complete response (pCR) at surgery   \n",
              "27                    idc_provenance_dicom_patient_id   \n",
              "28      Double-blinded patient ID number from I-SPY 2   \n",
              "29  Patient identification string in TCIA collecti...   \n",
              "30                         Age in years at enrollment   \n",
              "31  MRI measured longest diameter (cm) at baseline...   \n",
              "\n",
              "                                               values  \n",
              "0   [{'option_code': '0', 'option_description': No...  \n",
              "1   [{'option_code': '0', 'option_description': ' ...  \n",
              "2   [{'option_code': '0', 'option_description': ' ...  \n",
              "3   [{'option_code': '0', 'option_description': ' ...  \n",
              "4   [{'option_code': '0', 'option_description': ' ...  \n",
              "5   [{'option_code': '0', 'option_description': ' ...  \n",
              "6   [{'option_code': '0', 'option_description': ' ...  \n",
              "7   [{'option_code': '0', 'option_description': ' ...  \n",
              "8   [{'option_code': '0', 'option_description': ' ...  \n",
              "9   [{'option_code': '0', 'option_description': ' ...  \n",
              "10  [{'option_code': '0', 'option_description': ' ...  \n",
              "11  [{'option_code': '0', 'option_description': ' ...  \n",
              "12  [{'option_code': '0', 'option_description': No...  \n",
              "13  [{'option_code': 'T0', 'option_description': N...  \n",
              "14  [{'option_code': 'PASS', 'option_description':...  \n",
              "15  [{'option_code': 'PASS', 'option_description':...  \n",
              "16  [{'option_code': 'PASS', 'option_description':...  \n",
              "17  [{'option_code': 'PASS', 'option_description':...  \n",
              "18  [{'option_code': 'HIGH', 'option_description':...  \n",
              "19  [{'option_code': 'HIGH', 'option_description':...  \n",
              "20  [{'option_code': 'HIGH', 'option_description':...  \n",
              "21  [{'option_code': 'HIGH', 'option_description':...  \n",
              "22  [{'option_code': 'White', 'option_description'...  \n",
              "23  [{'option_code': 'Single mass', 'option_descri...  \n",
              "24  [{'option_code': 'HR + / HER2 +', 'option_desc...  \n",
              "25  [{'option_code': 'I (Low)', 'option_descriptio...  \n",
              "26  [{'option_code': 'non-pCR', 'option_descriptio...  \n",
              "27                                                 []  \n",
              "28                                                 []  \n",
              "29                                                 []  \n",
              "30                                                 []  \n",
              "31                                                 []  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a4ddfc88-bd0d-4633-9a1c-a17d90797845\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>short_table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>source_batch</td>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "      <td>[{'option_code': '0', 'option_description': No...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t0</td>\n",
              "      <td>T0 (baseline) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t1</td>\n",
              "      <td>T1 (early-Tx) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t2</td>\n",
              "      <td>T2 (inter-regimen) MRI study included in colle...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>t3</td>\n",
              "      <td>T3 (pre-surgery) MRI study included in collection</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>bmmr2_train</td>\n",
              "      <td>Patient included in the BMMR2 challenge traini...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>bmmr2_test</td>\n",
              "      <td>Patient included in the BMMR2 challenge test c...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t0</td>\n",
              "      <td>T0 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t1</td>\n",
              "      <td>T1 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t2</td>\n",
              "      <td>T2 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>primary_aim_t3</td>\n",
              "      <td>T3 MRI study included in the ACRIN-6698 primar...</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>analysis_cohort</td>\n",
              "      <td>Patient in primary analysis cohort (n=242)</td>\n",
              "      <td>[{'option_code': '0', 'option_description': ' ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>eligigible</td>\n",
              "      <td>eligigible</td>\n",
              "      <td>[{'option_code': '0', 'option_description': No...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>test_retest</td>\n",
              "      <td>Study time point that includes test/retest DWI...</td>\n",
              "      <td>[{'option_code': 'T0', 'option_description': N...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t0</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t1</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t2</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_pass_fail_t3</td>\n",
              "      <td>T0 MRI DWI Quality control assessment:</td>\n",
              "      <td>[{'option_code': 'PASS', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t0</td>\n",
              "      <td>T0 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t1</td>\n",
              "      <td>T1 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t2</td>\n",
              "      <td>T2 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>qc_roi_confidence_t3</td>\n",
              "      <td>T3 quality control confidence level for whole-...</td>\n",
              "      <td>[{'option_code': 'HIGH', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>race</td>\n",
              "      <td>Patient race:</td>\n",
              "      <td>[{'option_code': 'White', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>ltype</td>\n",
              "      <td>Index lesion type:</td>\n",
              "      <td>[{'option_code': 'Single mass', 'option_descri...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>hrher4g</td>\n",
              "      <td>Hormone receptor (HR) and HER2 status:</td>\n",
              "      <td>[{'option_code': 'HR + / HER2 +', 'option_desc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>sbrgrade</td>\n",
              "      <td>Tumor grade:</td>\n",
              "      <td>[{'option_code': 'I (Low)', 'option_descriptio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>pcr</td>\n",
              "      <td>Pathologic complete response (pCR) at surgery</td>\n",
              "      <td>[{'option_code': 'non-pCR', 'option_descriptio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>dicom_patient_id</td>\n",
              "      <td>idc_provenance_dicom_patient_id</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>i_spy_2_research_id</td>\n",
              "      <td>Double-blinded patient ID number from I-SPY 2</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>tcia_patient_id</td>\n",
              "      <td>Patient identification string in TCIA collecti...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>age</td>\n",
              "      <td>Age in years at enrollment</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>acrin_6698</td>\n",
              "      <td>acrin_6698_clinical</td>\n",
              "      <td>mrld</td>\n",
              "      <td>MRI measured longest diameter (cm) at baseline...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a4ddfc88-bd0d-4633-9a1c-a17d90797845')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a4ddfc88-bd0d-4633-9a1c-a17d90797845 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a4ddfc88-bd0d-4633-9a1c-a17d90797845');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0b18f794-9ae2-443c-8af2-991aba775f79\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0b18f794-9ae2-443c-8af2-991aba775f79')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0b18f794-9ae2-443c-8af2-991aba775f79 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"acrin6698_clinical_columns[[\\\"collection_id\\\", \\\"short_table_name\\\", \\\"column\\\", \\\"column_label\\\", \\\"values\\\"]]\",\n  \"rows\": 32,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"acrin_6698\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"short_table_name\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"acrin_6698_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 32,\n        \"samples\": [\n          \"tcia_patient_id\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 29,\n        \"samples\": [\n          \"Age in years at enrollment\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "acrin6698_clinical_columns[[\"collection_id\", \"short_table_name\", \"column\", \"column_label\", \"values\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CXy4mOdj428I"
      },
      "source": [
        "## From attribute names to data: accessing clinical data at patient level\n",
        "\n",
        "`short_table_name` gives us the table stored locally that contains the values for the column described in the `column_metadata` row. All of the clinical data tables were downloaded as part of `idc-index` installation.\n",
        "\n",
        "Here is how we can load the clinical data from the `acrin_6698_clinical` table into a pandas dataframe."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "acrin_6698_clinical_df = c.get_clinical_table(\"acrin_6698_clinical\")"
      ],
      "metadata": {
        "id": "USHHDVjLDRN-"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Given the information available in the per-collection clinical data, we can proceed with selecting a subset of patients that meet the criteria of your interest. As an example, the following query will select all of the distinct combination of patient ID and tumor grade, as defined by the `sbrgrade` column contents.\n"
      ],
      "metadata": {
        "id": "KQvhHGuTP3wK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: select distinct combinations of values in dicom_patient_id and sbrgrade columns from the acrin_6698_clinical_df pandas dataframe\n",
        "acrin_6698_tumors = acrin_6698_clinical_df[['dicom_patient_id', 'sbrgrade']].drop_duplicates()\n",
        "acrin_6698_tumors"
      ],
      "metadata": {
        "id": "CKYHeu1pQK8V",
        "outputId": "4ceb6c77-d9fe-4a3a-8647-146fdd21cd1d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      dicom_patient_id           sbrgrade\n",
              "0    ACRIN-6698-453236  II (Intermediate)\n",
              "1    ACRIN-6698-229047                 NA\n",
              "2    ACRIN-6698-384705  II (Intermediate)\n",
              "3    ACRIN-6698-415631         III (High)\n",
              "4    ACRIN-6698-793283                 NA\n",
              "..                 ...                ...\n",
              "380  ACRIN-6698-765671  II (Intermediate)\n",
              "381  ACRIN-6698-962153         III (High)\n",
              "382  ACRIN-6698-711476  II (Intermediate)\n",
              "383  ACRIN-6698-361701  II (Intermediate)\n",
              "384  ACRIN-6698-241998            I (Low)\n",
              "\n",
              "[385 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c8d4f416-aa84-4c49-a260-f295bebf58a2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>dicom_patient_id</th>\n",
              "      <th>sbrgrade</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ACRIN-6698-453236</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ACRIN-6698-229047</td>\n",
              "      <td>NA</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ACRIN-6698-384705</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ACRIN-6698-415631</td>\n",
              "      <td>III (High)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ACRIN-6698-793283</td>\n",
              "      <td>NA</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>380</th>\n",
              "      <td>ACRIN-6698-765671</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>381</th>\n",
              "      <td>ACRIN-6698-962153</td>\n",
              "      <td>III (High)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>382</th>\n",
              "      <td>ACRIN-6698-711476</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>383</th>\n",
              "      <td>ACRIN-6698-361701</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>384</th>\n",
              "      <td>ACRIN-6698-241998</td>\n",
              "      <td>I (Low)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>385 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c8d4f416-aa84-4c49-a260-f295bebf58a2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c8d4f416-aa84-4c49-a260-f295bebf58a2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c8d4f416-aa84-4c49-a260-f295bebf58a2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7e3c6a5b-97f9-4779-83ab-cc4658119bb7\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7e3c6a5b-97f9-4779-83ab-cc4658119bb7')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7e3c6a5b-97f9-4779-83ab-cc4658119bb7 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_1cb558c1-92b9-4f6e-908f-b9e6b7cfa343\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('distinct_combinations')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_1cb558c1-92b9-4f6e-908f-b9e6b7cfa343 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('distinct_combinations');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "distinct_combinations",
              "summary": "{\n  \"name\": \"distinct_combinations\",\n  \"rows\": 385,\n  \"fields\": [\n    {\n      \"column\": \"dicom_patient_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 385,\n        \"samples\": [\n          \"ACRIN-6698-227055\",\n          \"ACRIN-6698-522140\",\n          \"ACRIN-6698-712785\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sbrgrade\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"NA\",\n          \"I (Low)\",\n          \"II (Intermediate)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EE5N8y7RUw6a"
      },
      "source": [
        "We can next use `dicom_patient_id` to link clinical data with the imaging studies available for the given patient (which is available in the `index` table included in `idc-index`). The query becomes a bit more complex, since we need to join data across two tables."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"\"\"\n",
        "SELECT\n",
        "  ANY_VALUE(PatientID) AS PatientID,\n",
        "  STRING_AGG(DISTINCT(acrin_6698_clinical_df.sbrgrade)) as tumor_grade,\n",
        "  STRING_AGG(DISTINCT(Modality)) AS modalities,\n",
        "  COUNT(DISTINCT(SeriesInstanceUID)) AS num_series,\n",
        "  ANY_VALUE(CONCAT('https://viewer.imaging.datacommons.cancer.gov/viewer/', StudyInstanceUID)) AS viewer_url\n",
        "FROM\n",
        "  index\n",
        "JOIN\n",
        "  acrin_6698_clinical_df\n",
        "ON\n",
        "  index.PatientID = acrin_6698_clinical_df.dicom_patient_id\n",
        "GROUP BY\n",
        "  StudyInstanceUID\n",
        "ORDER BY\n",
        "  PatientID\n",
        "\"\"\"\n",
        "\n",
        "acrin_6698_viewable = c.sql_query(query)"
      ],
      "metadata": {
        "id": "lITbSQ2jQ17K"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "tqb7ONv5a_6_",
        "outputId": "0505b6d3-38fd-4cb4-f1f9-03880b88c682",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1620
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              PatientID        tumor_grade modalities  num_series  \\\n",
              "0     ACRIN-6698-102212         III (High)     MR,SEG          19   \n",
              "1     ACRIN-6698-102212         III (High)     SEG,MR          19   \n",
              "2     ACRIN-6698-102212         III (High)     SEG,MR          19   \n",
              "3     ACRIN-6698-102212         III (High)     MR,SEG          15   \n",
              "4     ACRIN-6698-103939         III (High)     SEG,MR          18   \n",
              "...                 ...                ...        ...         ...   \n",
              "1118  ACRIN-6698-995480         III (High)     SEG,MR          18   \n",
              "1119  ACRIN-6698-995480         III (High)     SEG,MR          18   \n",
              "1120  ACRIN-6698-995480         III (High)     SEG,MR          18   \n",
              "1121  ACRIN-6698-995480         III (High)     MR,SEG          14   \n",
              "1122  ACRIN-6698-999368  II (Intermediate)     SEG,MR          18   \n",
              "\n",
              "                                             viewer_url  \n",
              "0     https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1     https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "2     https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "3     https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "4     https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "...                                                 ...  \n",
              "1118  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1119  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1120  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1121  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1122  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "\n",
              "[1123 rows x 5 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3f21ee63-893c-45c4-a051-cb87b998defd\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PatientID</th>\n",
              "      <th>tumor_grade</th>\n",
              "      <th>modalities</th>\n",
              "      <th>num_series</th>\n",
              "      <th>viewer_url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ACRIN-6698-102212</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>MR,SEG</td>\n",
              "      <td>19</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ACRIN-6698-102212</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>19</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ACRIN-6698-102212</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>19</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ACRIN-6698-102212</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>MR,SEG</td>\n",
              "      <td>15</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ACRIN-6698-103939</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>18</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1118</th>\n",
              "      <td>ACRIN-6698-995480</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>18</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1119</th>\n",
              "      <td>ACRIN-6698-995480</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>18</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1120</th>\n",
              "      <td>ACRIN-6698-995480</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>18</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1121</th>\n",
              "      <td>ACRIN-6698-995480</td>\n",
              "      <td>III (High)</td>\n",
              "      <td>MR,SEG</td>\n",
              "      <td>14</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1122</th>\n",
              "      <td>ACRIN-6698-999368</td>\n",
              "      <td>II (Intermediate)</td>\n",
              "      <td>SEG,MR</td>\n",
              "      <td>18</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1123 rows × 5 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f21ee63-893c-45c4-a051-cb87b998defd')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3f21ee63-893c-45c4-a051-cb87b998defd button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3f21ee63-893c-45c4-a051-cb87b998defd');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3c186072-c51f-4a36-b05f-693af61a6a7b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3c186072-c51f-4a36-b05f-693af61a6a7b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3c186072-c51f-4a36-b05f-693af61a6a7b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_7853b0f0-1a49-4a3c-9305-bd445b45cc8a\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('acrin_6698_viewable')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_7853b0f0-1a49-4a3c-9305-bd445b45cc8a button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('acrin_6698_viewable');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "acrin_6698_viewable",
              "summary": "{\n  \"name\": \"acrin_6698_viewable\",\n  \"rows\": 1123,\n  \"fields\": [\n    {\n      \"column\": \"PatientID\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 385,\n        \"samples\": [\n          \"ACRIN-6698-721099\",\n          \"ACRIN-6698-687913\",\n          \"ACRIN-6698-945948\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tumor_grade\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"NA\",\n          \"I (Low)\",\n          \"III (High)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"modalities\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"MR,SEG\",\n          \"SEG,MR\",\n          \"MR\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_series\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3,\n        \"min\": 2,\n        \"max\": 32,\n        \"num_unique_values\": 28,\n        \"samples\": [\n          2,\n          27,\n          14\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"viewer_url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1123,\n        \"samples\": [\n          \"https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.7695.4164.102691710349159895369529927942\",\n          \"https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.7695.4164.314494137701007936430926991782\",\n          \"https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.7695.4164.179120149527078018237837173947\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 26
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/data_table.py:200: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  return self._dataframe._repr_html_()  # pylint: disable=protected-access\n"
          ]
        }
      ],
      "source": [
        "acrin_6698_viewable"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-38R6Z_3veHG"
      },
      "source": [
        "## Discovery mode\n",
        "\n",
        "Sometime you may want to find whether specific clinical attribute is available for the imaging data you can find in IDC.\n",
        "\n",
        "We can start by looking at the distinct values of `column_label` (which in the general case will be either more descriptive, or identical to `column`).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "tt3htrbxr6KC",
        "outputId": "bbceac91-6c1d-4479-e7f7-c9ed930e9a5e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           column_label\n",
              "0                           idc_provenance_source_batch\n",
              "1        T0 (baseline) MRI study included in collection\n",
              "2        T1 (early-Tx) MRI study included in collection\n",
              "3     T2 (inter-regimen) MRI study included in colle...\n",
              "4     T3 (pre-surgery) MRI study included in collection\n",
              "...                                                 ...\n",
              "3707                                                 ID\n",
              "3708                 Survival_from_surgery_days_UPDATED\n",
              "3709                                    Survival_Censor\n",
              "3710                          Time_since_baseline_preop\n",
              "3711                                  Age_at_scan_years\n",
              "\n",
              "[3712 rows x 1 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2020b699-f599-4c35-956e-bf344abc3aa3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>column_label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>idc_provenance_source_batch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>T0 (baseline) MRI study included in collection</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>T1 (early-Tx) MRI study included in collection</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>T2 (inter-regimen) MRI study included in colle...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>T3 (pre-surgery) MRI study included in collection</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3707</th>\n",
              "      <td>ID</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3708</th>\n",
              "      <td>Survival_from_surgery_days_UPDATED</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3709</th>\n",
              "      <td>Survival_Censor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3710</th>\n",
              "      <td>Time_since_baseline_preop</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3711</th>\n",
              "      <td>Age_at_scan_years</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3712 rows × 1 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2020b699-f599-4c35-956e-bf344abc3aa3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2020b699-f599-4c35-956e-bf344abc3aa3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2020b699-f599-4c35-956e-bf344abc3aa3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8ff08438-cc9c-4303-a321-1f2a80b9269c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8ff08438-cc9c-4303-a321-1f2a80b9269c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8ff08438-cc9c-4303-a321-1f2a80b9269c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 3712,\n  \"fields\": [\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3712,\n        \"samples\": [\n          \"TNM staging \",\n          \"BASELINE SYMPTOMS - SYMPTOM GRADE 19\",\n          \"DATE FORM COMPLETED (MM/DD/YYYY)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "pd.DataFrame({\"column_label\":c.clinical_index[\"column_label\"].unique()})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7iZpB1Foi1hd"
      },
      "source": [
        "Let's say we want to know which of the cases have information related to therapy. We can search column metadata for the presence of word \"therapy\" (since the terms in clinical data are not harmonized, we need to account for the variability in capitalization)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "qCTFR-QpjJSw",
        "outputId": "9a3e6dfb-497c-4e4d-d12e-f0dbe6490208",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           column_label  \\\n",
              "59    current or recent hx (6 months prior to MRI) c...   \n",
              "201         current use of estrogen replacement therapy   \n",
              "202               current use of tamoxifen/serm therapy   \n",
              "203             current use aromatase inhibitor therapy   \n",
              "205            past use of estrogen replacement therapy   \n",
              "206                  past use of tamoxifen/serm therapy   \n",
              "207               prior use aromatase inhibitor therapy   \n",
              "3891           Early response (1-2 months post-therapy)   \n",
              "4000  Did the patient have documented renal replacem...   \n",
              "4004  Patient had other anticoagulation therapy as l...   \n",
              "5570              chemotherapy used for TACE procedure    \n",
              "5888                         Regimen_or_Line_of_Therapy   \n",
              "5892                               Treatment_or_Therapy   \n",
              "6194                         Regimen_or_Line_of_Therapy   \n",
              "6209                               Treatment_or_Therapy   \n",
              "6515                         Regimen_or_Line_of_Therapy   \n",
              "6522                               Treatment_or_Therapy   \n",
              "6766                               Treatment_or_Therapy   \n",
              "6768                         Regimen_or_Line_of_Therapy   \n",
              "7141                                       Chemotherapy   \n",
              "7457                    primary_therapy_outcome_success   \n",
              "7534                    primary_therapy_outcome_success   \n",
              "7635                    primary_therapy_outcome_success   \n",
              "7684                    primary_therapy_outcome_success   \n",
              "7784                    primary_therapy_outcome_success   \n",
              "7830                    primary_therapy_outcome_success   \n",
              "7894                    primary_therapy_outcome_success   \n",
              "7984                    primary_therapy_outcome_success   \n",
              "8045                    primary_therapy_outcome_success   \n",
              "8129                    primary_therapy_outcome_success   \n",
              "8198                    primary_therapy_outcome_success   \n",
              "8290                    primary_therapy_outcome_success   \n",
              "8367                    primary_therapy_outcome_success   \n",
              "8435                    primary_therapy_outcome_success   \n",
              "8573                    primary_therapy_outcome_success   \n",
              "8647                    primary_therapy_outcome_success   \n",
              "8721                    primary_therapy_outcome_success   \n",
              "8857                    primary_therapy_outcome_success   \n",
              "8900                    primary_therapy_outcome_success   \n",
              "8978                    primary_therapy_outcome_success   \n",
              "9043                    primary_therapy_outcome_success   \n",
              "9125                    primary_therapy_outcome_success   \n",
              "9201                    primary_therapy_outcome_success   \n",
              "9301                    primary_therapy_outcome_success   \n",
              "9374                    primary_therapy_outcome_success   \n",
              "9423                    primary_therapy_outcome_success   \n",
              "9497                    primary_therapy_outcome_success   \n",
              "9597                    primary_therapy_outcome_success   \n",
              "9670                    primary_therapy_outcome_success   \n",
              "9715                    primary_therapy_outcome_success   \n",
              "9790                    primary_therapy_outcome_success   \n",
              "9892                    primary_therapy_outcome_success   \n",
              "\n",
              "                      collection_id  \\\n",
              "59    acrin_contralateral_breast_mr   \n",
              "201   acrin_contralateral_breast_mr   \n",
              "202   acrin_contralateral_breast_mr   \n",
              "203   acrin_contralateral_breast_mr   \n",
              "205   acrin_contralateral_breast_mr   \n",
              "206   acrin_contralateral_breast_mr   \n",
              "207   acrin_contralateral_breast_mr   \n",
              "3891         cc_tumor_heterogeneity   \n",
              "4000                covid_19_ny_sbu   \n",
              "4004                covid_19_ny_sbu   \n",
              "5570                   hcc_tace_seg   \n",
              "5888                       htan_hms   \n",
              "5892                       htan_hms   \n",
              "6194                      htan_ohsu   \n",
              "6209                      htan_ohsu   \n",
              "6515                htan_vanderbilt   \n",
              "6522                htan_vanderbilt   \n",
              "6766                     htan_wustl   \n",
              "6768                     htan_wustl   \n",
              "7141            nsclc_radiogenomics   \n",
              "7457                       tcga_acc   \n",
              "7534                      tcga_blca   \n",
              "7635                      tcga_brca   \n",
              "7684                      tcga_cesc   \n",
              "7784                      tcga_chol   \n",
              "7830                      tcga_coad   \n",
              "7894                      tcga_dlbc   \n",
              "7984                      tcga_esca   \n",
              "8045                       tcga_gbm   \n",
              "8129                      tcga_hnsc   \n",
              "8198                      tcga_kich   \n",
              "8290                      tcga_kirc   \n",
              "8367                      tcga_kirp   \n",
              "8435                       tcga_lgg   \n",
              "8573                      tcga_lihc   \n",
              "8647                      tcga_luad   \n",
              "8721                      tcga_lusc   \n",
              "8857                      tcga_meso   \n",
              "8900                        tcga_ov   \n",
              "8978                      tcga_paad   \n",
              "9043                      tcga_pcpg   \n",
              "9125                      tcga_prad   \n",
              "9201                      tcga_read   \n",
              "9301                      tcga_sarc   \n",
              "9374                      tcga_skcm   \n",
              "9423                      tcga_stad   \n",
              "9497                      tcga_tgct   \n",
              "9597                      tcga_thca   \n",
              "9670                      tcga_thym   \n",
              "9715                      tcga_ucec   \n",
              "9790                       tcga_ucs   \n",
              "9892                       tcga_uvm   \n",
              "\n",
              "                                                 values  \n",
              "59    [{'option_code': '1.0', 'option_description': ...  \n",
              "201   [{'option_code': '1.0', 'option_description': ...  \n",
              "202   [{'option_code': '1.0', 'option_description': ...  \n",
              "203   [{'option_code': '1.0', 'option_description': ...  \n",
              "205   [{'option_code': '1.0', 'option_description': ...  \n",
              "206   [{'option_code': '1.0', 'option_description': ...  \n",
              "207   [{'option_code': '1.0', 'option_description': ...  \n",
              "3891  [{'option_code': 'Non-responder', 'option_desc...  \n",
              "4000  [{'option_code': 'Yes', 'option_description': ...  \n",
              "4004  [{'option_code': 'apixaban', 'option_descripti...  \n",
              "5570  [{'option_code': 'Cisplastin', 'option_descrip...  \n",
              "5888  [{'option_code': 'FOLFOX', 'option_description...  \n",
              "5892  [{'option_code': 'No', 'option_description': N...  \n",
              "6194  [{'option_code': 'Abemaciclib', 'option_descri...  \n",
              "6209  [{'option_code': 'None', 'option_description':...  \n",
              "6515  [{'option_code': 'None', 'option_description':...  \n",
              "6522  [{'option_code': 'Not Reported', 'option_descr...  \n",
              "6766  [{'option_code': 'Yes', 'option_description': ...  \n",
              "6768  [{'option_code': 'Fifth', 'option_description'...  \n",
              "7141  [{'option_code': 'No', 'option_description': N...  \n",
              "7457  [{'option_code': 'Complete Remission/Response'...  \n",
              "7534  [{'option_code': 'Complete Remission/Response'...  \n",
              "7635  [{'option_code': 'None', 'option_description':...  \n",
              "7684  [{'option_code': 'Complete Remission/Response'...  \n",
              "7784  [{'option_code': 'None', 'option_description':...  \n",
              "7830  [{'option_code': 'Complete Remission/Response'...  \n",
              "7894  [{'option_code': 'Complete Remission/Response'...  \n",
              "7984  [{'option_code': 'Complete Remission/Response'...  \n",
              "8045  [{'option_code': 'Complete Remission/Response'...  \n",
              "8129  [{'option_code': 'Complete Remission/Response'...  \n",
              "8198  [{'option_code': 'Complete Remission/Response'...  \n",
              "8290  [{'option_code': 'Complete Remission/Response'...  \n",
              "8367  [{'option_code': 'Complete Remission/Response'...  \n",
              "8435  [{'option_code': 'Complete Remission/Response'...  \n",
              "8573  [{'option_code': 'None', 'option_description':...  \n",
              "8647  [{'option_code': 'Complete Remission/Response'...  \n",
              "8721  [{'option_code': 'Complete Remission/Response'...  \n",
              "8857  [{'option_code': 'None', 'option_description':...  \n",
              "8900  [{'option_code': 'Complete Remission/Response'...  \n",
              "8978  [{'option_code': 'Complete Remission/Response'...  \n",
              "9043  [{'option_code': 'Complete Remission/Response'...  \n",
              "9125  [{'option_code': 'Complete Remission/Response'...  \n",
              "9201  [{'option_code': 'Complete Remission/Response'...  \n",
              "9301  [{'option_code': 'None', 'option_description':...  \n",
              "9374  [{'option_code': 'None', 'option_description':...  \n",
              "9423  [{'option_code': 'Complete Remission/Response'...  \n",
              "9497  [{'option_code': 'Complete Remission/Response'...  \n",
              "9597  [{'option_code': 'None', 'option_description':...  \n",
              "9670  [{'option_code': 'None', 'option_description':...  \n",
              "9715  [{'option_code': 'Complete Remission/Response'...  \n",
              "9790  [{'option_code': 'Complete Remission/Response'...  \n",
              "9892  [{'option_code': 'None', 'option_description':...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b9608398-c631-4d08-abcc-199ec352f902\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>column_label</th>\n",
              "      <th>collection_id</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>current or recent hx (6 months prior to MRI) c...</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>201</th>\n",
              "      <td>current use of estrogen replacement therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>202</th>\n",
              "      <td>current use of tamoxifen/serm therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>203</th>\n",
              "      <td>current use aromatase inhibitor therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>205</th>\n",
              "      <td>past use of estrogen replacement therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>206</th>\n",
              "      <td>past use of tamoxifen/serm therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>207</th>\n",
              "      <td>prior use aromatase inhibitor therapy</td>\n",
              "      <td>acrin_contralateral_breast_mr</td>\n",
              "      <td>[{'option_code': '1.0', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3891</th>\n",
              "      <td>Early response (1-2 months post-therapy)</td>\n",
              "      <td>cc_tumor_heterogeneity</td>\n",
              "      <td>[{'option_code': 'Non-responder', 'option_desc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4000</th>\n",
              "      <td>Did the patient have documented renal replacem...</td>\n",
              "      <td>covid_19_ny_sbu</td>\n",
              "      <td>[{'option_code': 'Yes', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4004</th>\n",
              "      <td>Patient had other anticoagulation therapy as l...</td>\n",
              "      <td>covid_19_ny_sbu</td>\n",
              "      <td>[{'option_code': 'apixaban', 'option_descripti...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5570</th>\n",
              "      <td>chemotherapy used for TACE procedure</td>\n",
              "      <td>hcc_tace_seg</td>\n",
              "      <td>[{'option_code': 'Cisplastin', 'option_descrip...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5888</th>\n",
              "      <td>Regimen_or_Line_of_Therapy</td>\n",
              "      <td>htan_hms</td>\n",
              "      <td>[{'option_code': 'FOLFOX', 'option_description...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5892</th>\n",
              "      <td>Treatment_or_Therapy</td>\n",
              "      <td>htan_hms</td>\n",
              "      <td>[{'option_code': 'No', 'option_description': N...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6194</th>\n",
              "      <td>Regimen_or_Line_of_Therapy</td>\n",
              "      <td>htan_ohsu</td>\n",
              "      <td>[{'option_code': 'Abemaciclib', 'option_descri...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6209</th>\n",
              "      <td>Treatment_or_Therapy</td>\n",
              "      <td>htan_ohsu</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6515</th>\n",
              "      <td>Regimen_or_Line_of_Therapy</td>\n",
              "      <td>htan_vanderbilt</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6522</th>\n",
              "      <td>Treatment_or_Therapy</td>\n",
              "      <td>htan_vanderbilt</td>\n",
              "      <td>[{'option_code': 'Not Reported', 'option_descr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6766</th>\n",
              "      <td>Treatment_or_Therapy</td>\n",
              "      <td>htan_wustl</td>\n",
              "      <td>[{'option_code': 'Yes', 'option_description': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6768</th>\n",
              "      <td>Regimen_or_Line_of_Therapy</td>\n",
              "      <td>htan_wustl</td>\n",
              "      <td>[{'option_code': 'Fifth', 'option_description'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7141</th>\n",
              "      <td>Chemotherapy</td>\n",
              "      <td>nsclc_radiogenomics</td>\n",
              "      <td>[{'option_code': 'No', 'option_description': N...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7457</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_acc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7534</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_blca</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7635</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_brca</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7684</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_cesc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7784</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_chol</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7830</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_coad</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7894</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_dlbc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7984</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_esca</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8045</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_gbm</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8129</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_hnsc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8198</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_kich</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8290</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_kirc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8367</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_kirp</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8435</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_lgg</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8573</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_lihc</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8647</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_luad</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8721</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_lusc</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8857</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_meso</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8900</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_ov</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8978</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_paad</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9043</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_pcpg</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9125</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_prad</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9201</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_read</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9301</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_sarc</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9374</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_skcm</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9423</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_stad</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9497</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_tgct</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9597</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_thca</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9670</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_thym</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9715</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_ucec</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9790</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_ucs</td>\n",
              "      <td>[{'option_code': 'Complete Remission/Response'...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9892</th>\n",
              "      <td>primary_therapy_outcome_success</td>\n",
              "      <td>tcga_uvm</td>\n",
              "      <td>[{'option_code': 'None', 'option_description':...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b9608398-c631-4d08-abcc-199ec352f902')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b9608398-c631-4d08-abcc-199ec352f902 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b9608398-c631-4d08-abcc-199ec352f902');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-dd67b513-98bf-4711-a26d-2d2e7acdc226\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dd67b513-98bf-4711-a26d-2d2e7acdc226')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-dd67b513-98bf-4711-a26d-2d2e7acdc226 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"c\",\n  \"rows\": 52,\n  \"fields\": [\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 15,\n        \"samples\": [\n          \"Patient had other anticoagulation therapy as listed\",\n          \"Regimen_or_Line_of_Therapy\",\n          \"current or recent hx (6 months prior to MRI) chemo therapy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 41,\n        \"samples\": [\n          \"tcga_luad\",\n          \"tcga_chol\",\n          \"nsclc_radiogenomics\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "c.clinical_index[c.clinical_index[\"column_label\"].str.contains(\"[tT]herapy\", na=False)][[\"column_label\", \"collection_id\", \"values\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V23hIu1jo-We"
      },
      "source": [
        "We observe that there are several collections that contain column named \"Chemotherapy\". Let's filter these values further, in order to identify subjects that underwent chemotherapy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "uACJ3N_GFHFG",
        "outputId": "54733fc0-defa-4374-e88d-d24909034628",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            collection_id                                         table_name  \\\n",
              "5570         hcc_tace_seg  bigquery-public-data.idc_v20_clinical.hcc_tace...   \n",
              "7141  nsclc_radiogenomics  bigquery-public-data.idc_v20_clinical.nsclc_ra...   \n",
              "\n",
              "            column                           column_label  \\\n",
              "5570  chemotherapy  chemotherapy used for TACE procedure    \n",
              "7141  chemotherapy                           Chemotherapy   \n",
              "\n",
              "                                                 values  \n",
              "5570  [{'option_code': 'Cisplastin', 'option_descrip...  \n",
              "7141  [{'option_code': 'No', 'option_description': N...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8e270772-8c27-4911-8d96-27fdfaf1e4a1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>5570</th>\n",
              "      <td>hcc_tace_seg</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.hcc_tace...</td>\n",
              "      <td>chemotherapy</td>\n",
              "      <td>chemotherapy used for TACE procedure</td>\n",
              "      <td>[{'option_code': 'Cisplastin', 'option_descrip...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7141</th>\n",
              "      <td>nsclc_radiogenomics</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.nsclc_ra...</td>\n",
              "      <td>chemotherapy</td>\n",
              "      <td>Chemotherapy</td>\n",
              "      <td>[{'option_code': 'No', 'option_description': N...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8e270772-8c27-4911-8d96-27fdfaf1e4a1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8e270772-8c27-4911-8d96-27fdfaf1e4a1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8e270772-8c27-4911-8d96-27fdfaf1e4a1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b3c61081-60b8-4f72-bace-11c58c51131c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b3c61081-60b8-4f72-bace-11c58c51131c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b3c61081-60b8-4f72-bace-11c58c51131c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"c\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"nsclc_radiogenomics\",\n          \"hcc_tace_seg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"table_name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"bigquery-public-data.idc_v20_clinical.nsclc_radiogenomics_clinical\",\n          \"bigquery-public-data.idc_v20_clinical.hcc_tace_seg_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"chemotherapy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"Chemotherapy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 32
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/data_table.py:200: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  return self._dataframe._repr_html_()  # pylint: disable=protected-access\n"
          ]
        }
      ],
      "source": [
        "c.clinical_index[c.clinical_index[\"column_label\"].str.contains(\"[Cc]hemotherapy\", na=False)][[ \"collection_id\", \"table_name\", \"column\", \"column_label\",\"values\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kke1DZgwGHuA"
      },
      "source": [
        "From the table above we can observe that there are several collections that have clinical metadata related to chemotherapy regimen of the subject.\n",
        "\n",
        "Looking at the value sets for the collections/columns that have those, we can observe that subjects that had any chemotherapy could be selected as follows for the respective collections (non-exhaustive list):\n",
        "* `nsclc_radiogenomics`: subjects that have value `Yes` in table `nsclc_radiogenomics_clinical` column `chemotherapy`\n",
        "* `hcc_tace_seg` column `chemotherapy` in the `hcc_tace_seg_clinical` table\n",
        "\n",
        "Let's focus on the clinical data related to chemotherapy for the collection `hcc_tace_seg`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "EhrEq84tezyL",
        "outputId": "b1982490-a95f-458b-a8a1-a47aec802ed4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     collection_id                                         table_name  \\\n",
              "5570  hcc_tace_seg  bigquery-public-data.idc_v20_clinical.hcc_tace...   \n",
              "\n",
              "            column                           column_label  \\\n",
              "5570  chemotherapy  chemotherapy used for TACE procedure    \n",
              "\n",
              "                                                 values  \n",
              "5570  [{'option_code': 'Cisplastin', 'option_descrip...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ee45441e-e416-4bdb-bd04-0dcd62935a39\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>collection_id</th>\n",
              "      <th>table_name</th>\n",
              "      <th>column</th>\n",
              "      <th>column_label</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>5570</th>\n",
              "      <td>hcc_tace_seg</td>\n",
              "      <td>bigquery-public-data.idc_v20_clinical.hcc_tace...</td>\n",
              "      <td>chemotherapy</td>\n",
              "      <td>chemotherapy used for TACE procedure</td>\n",
              "      <td>[{'option_code': 'Cisplastin', 'option_descrip...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ee45441e-e416-4bdb-bd04-0dcd62935a39')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ee45441e-e416-4bdb-bd04-0dcd62935a39 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ee45441e-e416-4bdb-bd04-0dcd62935a39');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_2844e6e8-4263-47eb-951f-4393f532e115\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('chemotherapy_subset')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_2844e6e8-4263-47eb-951f-4393f532e115 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('chemotherapy_subset');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "chemotherapy_subset",
              "summary": "{\n  \"name\": \"chemotherapy_subset\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"hcc_tace_seg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"table_name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"bigquery-public-data.idc_v20_clinical.hcc_tace_seg_clinical\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"chemotherapy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"column_label\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"chemotherapy used for TACE procedure \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 33
        }
      ],
      "source": [
        "chemotherapy_subset = c.clinical_index[c.clinical_index[\"column_label\"].str.contains(\"[Cc]hemotherapy\", na=False)][[ \"collection_id\", \"table_name\", \"column\", \"column_label\",\"values\"]]\n",
        "\n",
        "chemotherapy_subset = chemotherapy_subset[chemotherapy_subset[\"collection_id\"] == \"hcc_tace_seg\"]\n",
        "\n",
        "chemotherapy_subset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGu6dVggelsQ"
      },
      "source": [
        "Before we select subjects that meet the criteria defined above, let's confirm  the values encountered in table `hcc_tace_seg_clinical` column `chemotherapy` match the value set in the `column_metadata` table. Here are the values of the dictionary we observe from the `column_metadata` inventory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "pcJeHJrifjIG",
        "outputId": "e4916bb9-80f1-4d56-9bc2-c273e393752a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([{'option_code': 'Cisplastin', 'option_description': None},\n",
              "        {'option_code': 'Cisplatin, Mitomycin-C', 'option_description': None},\n",
              "        {'option_code': 'Cisplatin, doxorubicin, Mitomycin-C', 'option_description': None},\n",
              "        {'option_code': 'NA', 'option_description': None},\n",
              "        {'option_code': 'doxorubicin LC beads', 'option_description': None}],\n",
              "       dtype=object)]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ],
      "source": [
        "chemotherapy_subset[\"values\"].tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GEKd7J4Dfa46"
      },
      "source": [
        "The query below will select the distinct values encountered in the `chemotherapy` column of the `hcc_tace_seg_clinical` table, which we confirm matches those in the dictionary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "sUdb9uOoJg7p",
        "outputId": "b6e54a83-9803-40b0-922b-24b79e0247d4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['Cisplatin, doxorubicin, Mitomycin-C', 'doxorubicin LC beads',\n",
              "       'NA', 'Cisplastin', 'Cisplatin, Mitomycin-C'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "hcc_tace_seg_clinical_df = c.get_clinical_table(\"hcc_tace_seg_clinical\")\n",
        "\n",
        "hcc_tace_seg_clinical_df[\"chemotherapy\"].unique()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQRPfWs7GsEg"
      },
      "source": [
        "Next we can examine the data to see what therapy individual patients had."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: select distinct combinations of values in dicom_patient_id and chemotherapy columns from the hcc_tace_seg_clinical_df pandas dataframe where chemotherapy column is defined\n",
        "\n",
        "distinct_combinations = hcc_tace_seg_clinical_df[hcc_tace_seg_clinical_df['chemotherapy'].notna()][['dicom_patient_id', 'chemotherapy']].drop_duplicates()\n",
        "distinct_combinations"
      ],
      "metadata": {
        "id": "mx7mxP5xUEYX",
        "outputId": "9f166baf-ea82-42c0-86dc-5c81b0a6c285",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    dicom_patient_id                         chemotherapy\n",
              "0            HCC_024  Cisplatin, doxorubicin, Mitomycin-C\n",
              "1            HCC_045  Cisplatin, doxorubicin, Mitomycin-C\n",
              "2            HCC_050  Cisplatin, doxorubicin, Mitomycin-C\n",
              "3            HCC_065  Cisplatin, doxorubicin, Mitomycin-C\n",
              "4            HCC_073                 doxorubicin LC beads\n",
              "..               ...                                  ...\n",
              "100          HCC_056  Cisplatin, doxorubicin, Mitomycin-C\n",
              "101          HCC_070                 doxorubicin LC beads\n",
              "102          HCC_061                                   NA\n",
              "103          HCC_075                 doxorubicin LC beads\n",
              "104          HCC_099                 doxorubicin LC beads\n",
              "\n",
              "[105 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-69c6b2b0-8564-4cb0-8eb8-ab5fc43cee9f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>dicom_patient_id</th>\n",
              "      <th>chemotherapy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>HCC_024</td>\n",
              "      <td>Cisplatin, doxorubicin, Mitomycin-C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>HCC_045</td>\n",
              "      <td>Cisplatin, doxorubicin, Mitomycin-C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>HCC_050</td>\n",
              "      <td>Cisplatin, doxorubicin, Mitomycin-C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>HCC_065</td>\n",
              "      <td>Cisplatin, doxorubicin, Mitomycin-C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>HCC_073</td>\n",
              "      <td>doxorubicin LC beads</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>100</th>\n",
              "      <td>HCC_056</td>\n",
              "      <td>Cisplatin, doxorubicin, Mitomycin-C</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>101</th>\n",
              "      <td>HCC_070</td>\n",
              "      <td>doxorubicin LC beads</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>102</th>\n",
              "      <td>HCC_061</td>\n",
              "      <td>NA</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>HCC_075</td>\n",
              "      <td>doxorubicin LC beads</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>HCC_099</td>\n",
              "      <td>doxorubicin LC beads</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>105 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-69c6b2b0-8564-4cb0-8eb8-ab5fc43cee9f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-69c6b2b0-8564-4cb0-8eb8-ab5fc43cee9f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-69c6b2b0-8564-4cb0-8eb8-ab5fc43cee9f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-38204da8-62fc-448b-b524-e96a614da87b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-38204da8-62fc-448b-b524-e96a614da87b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-38204da8-62fc-448b-b524-e96a614da87b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_3cbb1c7c-cdaa-4912-82d9-e4277172522a\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('distinct_combinations')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_3cbb1c7c-cdaa-4912-82d9-e4277172522a button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('distinct_combinations');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "distinct_combinations",
              "summary": "{\n  \"name\": \"distinct_combinations\",\n  \"rows\": 105,\n  \"fields\": [\n    {\n      \"column\": \"dicom_patient_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 105,\n        \"samples\": [\n          \"HCC_008\",\n          \"HCC_059\",\n          \"HCC_058\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chemotherapy\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"doxorubicin LC beads\",\n          \"Cisplatin, Mitomycin-C\",\n          \"NA\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjdEJe1rHvQC"
      },
      "source": [
        "`dicom_patient_id` is the key to connect clinical data and imaging data. Let's see what imaging studies we have for patient `HCC_101` that was treated with \tdoxorubicin LC beads. Along the way we can also generate viewer URLs to conveniently examine the images."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"\"\"\n",
        "SELECT\n",
        "  StudyInstanceUID,\n",
        "  STRING_AGG(DISTINCT(Modality)) AS modalities,\n",
        "  STRING_AGG(DISTINCT(collection_id)) AS collection_id,\n",
        "  COUNT(DISTINCT(SeriesInstanceUID)) AS num_series,\n",
        "  ANY_VALUE(CONCAT('https://viewer.imaging.datacommons.cancer.gov/viewer/', StudyInstanceUID)) as viewer_url\n",
        "FROM index\n",
        "WHERE PatientID = 'HCC_103'\n",
        "GROUP BY StudyInstanceUID\n",
        "\"\"\"\n",
        "\n",
        "c.sql_query(query)"
      ],
      "metadata": {
        "id": "RqjVoU_7VP65",
        "outputId": "d0bf6313-26fc-4db4-ed1d-39cbaf6962e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: Index.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n",
            "/usr/local/lib/python3.10/dist-packages/google/colab/_interactive_table_hint_button.py:178: FutureWarning: RangeIndex.format is deprecated and will be removed in a future version. Convert using index.astype(str) or index.map(formatter) instead.\n",
            "  df_html=dataframe._repr_html_(),  # pylint: disable=protected-access\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                    StudyInstanceUID modalities collection_id  \\\n",
              "0  1.3.6.1.4.1.14519.5.2.1.1706.8374.304819071818...     CT,SEG  hcc_tace_seg   \n",
              "1  1.3.6.1.4.1.14519.5.2.1.1706.8374.121752675166...         CT  hcc_tace_seg   \n",
              "\n",
              "   num_series                                         viewer_url  \n",
              "0           3  https://viewer.imaging.datacommons.cancer.gov/...  \n",
              "1           2  https://viewer.imaging.datacommons.cancer.gov/...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9b24b369-80cd-483d-adba-89e4a3320bd3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>StudyInstanceUID</th>\n",
              "      <th>modalities</th>\n",
              "      <th>collection_id</th>\n",
              "      <th>num_series</th>\n",
              "      <th>viewer_url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.3.6.1.4.1.14519.5.2.1.1706.8374.304819071818...</td>\n",
              "      <td>CT,SEG</td>\n",
              "      <td>hcc_tace_seg</td>\n",
              "      <td>3</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.3.6.1.4.1.14519.5.2.1.1706.8374.121752675166...</td>\n",
              "      <td>CT</td>\n",
              "      <td>hcc_tace_seg</td>\n",
              "      <td>2</td>\n",
              "      <td>https://viewer.imaging.datacommons.cancer.gov/...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9b24b369-80cd-483d-adba-89e4a3320bd3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9b24b369-80cd-483d-adba-89e4a3320bd3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9b24b369-80cd-483d-adba-89e4a3320bd3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-4a98ca71-ae88-4fd9-afff-7c64e9de394b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4a98ca71-ae88-4fd9-afff-7c64e9de394b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-4a98ca71-ae88-4fd9-afff-7c64e9de394b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"c\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"StudyInstanceUID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"1.3.6.1.4.1.14519.5.2.1.1706.8374.121752675166759110829287848146\",\n          \"1.3.6.1.4.1.14519.5.2.1.1706.8374.304819071818674684593765963305\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"modalities\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"CT\",\n          \"CT,SEG\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"collection_id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"hcc_tace_seg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_series\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 2,\n        \"max\": 3,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          2\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"viewer_url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.1706.8374.121752675166759110829287848146\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rixPlivvj9de"
      },
      "source": [
        "# Want to learn more?\n",
        "\n",
        "* check out other notebooks: https://github.com/ImagingDataCommons/IDC-Tutorials\n",
        "* join our community forum to ask any questions about IDC: https://discourse.canceridc.dev/\n",
        "* ask your questions during live discussions with IDC developers at the IDC weekly office hours - join us on Google Meet at https://meet.google.com/xyt-vody-tvb every Tuesday 16:30 – 17:30 (New York) and Wednesday 10:30-11:30 (New York)\n",
        "* browse IDC portal: https://imaging.datacommons.cancer.gov/explore/\n",
        "* read IDC paper: https://doi.org/10.1148/rg.230180\n",
        "* watch a recent presentation about IDC: https://youtu.be/P9ateg9ZUEs"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}