pyserini_covid19_paragraph.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "pyserini_covid19_paragraph.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks/blob/master/pyserini_covid19_paragraph.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUpmSFflwzR1",
        "colab_type": "text"
      },
      "source": [
        "# Pyserini Demo on COVID-19 Dataset (Paragraph Index)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0RtnmBN6A-Oq",
        "colab_type": "text"
      },
      "source": [
        "This notebook provides a demo on how to get started in searching the [COVID-19 Open Research Dataset](https://pages.semanticscholar.org/coronavirus-research) (release of 2020/04/03) from AI2.\n",
        "Here, we'll be working with the paragraph index.\n",
        "We have [another notebook](https://github.com/castorini/anserini-notebooks/blob/master/pyserini_covid19_default.ipynb) for working with the simpler title + abstract index."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2vNd7bpJlDZ",
        "colab_type": "text"
      },
      "source": [
        "First, install Python dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_lt0-pXJia0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!pip install pyserini==0.9.0.0\n",
        "\n",
        "import json\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-11-openjdk-amd64\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbXmstbGypef",
        "colab_type": "text"
      },
      "source": [
        "Let's grab the pre-built index:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JT_OJKftdqGP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!wget https://www.dropbox.com/s/ivk87journyajw3/lucene-index-covid-paragraph-2020-04-10.tar.gz\n",
        "!tar xvfz lucene-index-covid-paragraph-2020-04-10.tar.gz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwVSvHBz9RMM",
        "colab_type": "text"
      },
      "source": [
        "Sanity check of index size (should be 5.3G):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVXWA6WS0aqJ",
        "colab_type": "code",
        "outputId": "cae8faef-ab1f-45c0-8449-7f0e8bada083",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!du -h lucene-index-covid-paragraph-2020-04-10"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5.8G\tlucene-index-covid-paragraph-2020-04-10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WW-vWrP41fAz",
        "colab_type": "text"
      },
      "source": [
        "Now, a bit of explanation of how the index is organized.\n",
        "For each source article, we create a paragraph-level index as follows, for a hypothetical article with id `docid`, in the index there'll be:\n",
        "\n",
        "+ `docid`: title + abstract\n",
        "+ `docid.00001`: title + abstract + 1st paragraph\n",
        "+ `docid.00002`: title + abstract + 2nd paragraph\n",
        "+ `docid.00003`: title + abstract + 3rd paragraph\n",
        "+ ...\n",
        "\n",
        "That is, each article is chopped up into individual paragraphs.\n",
        "Each paragraph is indexed as a \"document\" (with the title and abstract). \n",
        "The suffix of the `docid`, `.XXXXX` identifies which paragraph is being indexed (numbered sequentially)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NkD0kKxW9mHP",
        "colab_type": "text"
      },
      "source": [
        "You can use `pysearch` to search over an index. Here's the basic usage:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFZlcqEX0t1f",
        "colab_type": "code",
        "outputId": "51ea2792-5f22-472b-e713-23b4460e2a9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        }
      },
      "source": [
        "from pyserini.search import pysearch\n",
        "\n",
        "searcher = pysearch.SimpleSearcher('lucene-index-covid-paragraph-2020-04-10/')\n",
        "hits = searcher.search('nsp1 synthesis degradation', 10)\n",
        "\n",
        "# Prints the first 10 hits\n",
        "for i in range(0, 10):\n",
        "    print(f'{i+1:2} {hits[i].docid:14} {hits[i].score:.5f} {hits[i].lucene_document.get(\"title\")} {hits[i].lucene_document.get(\"doi\")}')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 1 o328y8ax.00004 10.87830 Modulation of type I interferon induction by porcine reproductive and respiratory syndrome virus and degradation of CREB-binding protein by non-structural protein 1 in MARC-145 and HeLa cells 10.1016/j.virol.2010.03.039\n",
            " 2 06z7p7rc       10.79910 Severe Acute Respiratory Syndrome Coronavirus nsp1 Suppresses Host Gene Expression, Including That of Type I Interferon, in Infected Cells 10.1128/jvi.02472-07\n",
            " 3 ncufofro.00026 10.77400 Chapter Five Viral and Cellular mRNA Translation in Coronavirus-Infected Cells 10.1016/bs.aivir.2016.08.001\n",
            " 4 mtj46j82.00029 10.62580 MERS coronavirus nsp1 participates in an efficient propagation through a specific interaction with viral RNA 10.1016/j.virol.2017.08.026\n",
            " 5 42saxb98.00002 10.61430 A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein 10.1038/nsmb.1680\n",
            " 6 42saxb98.00001 10.60960 A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein 10.1038/nsmb.1680\n",
            " 7 vj3wk150.00035 10.60060 Regulation of Stress Responses and Translational Control by Coronavirus 10.3390/v8070184\n",
            " 8 sq9hh50d.00005 10.54510 Unique SARS-CoV protein nsp1: bioinformatics, biochemistry and potential effects on virulence 10.1016/j.tim.2006.12.005\n",
            " 9 42saxb98.00020 10.54150 A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein 10.1038/nsmb.1680\n",
            "10 pdfs6ojs.00027 10.52290 Coronavirus nonstructural protein 1: Common and distinct functions in the regulation of host and viral gene expression 10.1016/j.virusres.2014.11.019\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6fk6kKQ2t-s",
        "colab_type": "text"
      },
      "source": [
        "From the hits array, use `.lucene_document` to access the underlying indexed Lucene `Document`, and from there, call `.get(field)` to fetch specific fields, like \"title\", \"doc\", etc.\n",
        "The complete list of available fields is [here](https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/generator/CovidGenerator.java#L46).\n",
        "\n",
        "Note that we retrieve multiple paragraphs from the same article \"A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein\" (hits #5 and #6). Note that this actually a good thing, because a downstream module can do evidence integration.\n",
        "\n",
        "Considering hit #5 (`42saxb98.00002`) and hit #6 (`42saxb98.00001`), use `.contents` of the hit to see exactly what was indexed.\n",
        "\n",
        "For hit #5:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-EqwvCMU2-Ft",
        "colab_type": "code",
        "outputId": "0ac90491-5613-4a14-ef25-3f38573e73aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "hits[4].contents.split('\\n')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein',\n",
              " \"Severe acute respiratory syndrome coronavirus nsp1 protein suppresses host gene expression, including type I interferon production, by promoting host mRNA degradation and inhibiting host translation, in infected cells. We present evidence that nsp1 uses a novel, two-pronged strategy to inhibit host translation/gene expression. Nsp1 bound to the 40S ribosomal subunit and inactivated the translational activity of the 40S subunits. Furthermore, the nsp1-40S ribosome complex induced the modification of the 5'-region of capped mRNA template and rendered the template RNA translationally incompetent. Nsp1 also induced RNA cleavage in templates carrying the internal ribosome entry site (IRES) from encephalomyocarditis virus, but not in those carrying IRESs from hepatitis C and cricket paralysis viruses, demonstrating that the nsp1-induced RNA modification was template-dependent. We speculate that the mRNAs that underwent the nsp1-mediated modification are marked for rapid turnover by the host RNA degradation machinery.\",\n",
              " 'We tested whether nsp1 protein could suppress translation in rabbit reticulocyte lysate (RRL). We expressed the full-length, wild-type nsp1 protein and its mutant form (nsp1-mt) carrying K164A and H165A mutations as glutathione S-transferase (GST)-tagged fusion proteins in E.coli followed by the removal of the GST tag to generate recombinant nsp1 and nsp1-mt proteins, respectively; nsp1-mt neither suppresses host translation nor promotes host mRNA degradation in expressing cells and infected cells10. After incubation of the nsp1 protein with RRL at 4°C for 30 min, we added different concentrations of capped and polyadenylated Renilla luciferase mRNA transcripts (rLuc RNA) to the mixture, and incubated the samples in the presence of [ 35 S]methionine for 30 min. In control samples in which rLuc RNA was incubated with GST and nsp1-mt proteins, rLuc activity and labeled rLuc protein levels increased with rising mRNA concentrations. In contrast, the rLuc activity and labeled rLuc protein levels in the nsp1-containing sample was substantially lower than in the control samples; the rLuc activity and the radiolabeled rLuc protein levels in the presence of nsp1 were about 6-8% of the levels observed with GST or nsp1-mt ( Fig.  1a and b) , which clearly demonstrated that nsp1 efficiently inhibited the rLuc protein synthesis from capped rLuc RNA in RRL.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OfL1RYGE31fH",
        "colab_type": "text"
      },
      "source": [
        "For hit #6:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HcvgFupQ3xPJ",
        "colab_type": "code",
        "outputId": "438a09ea-4054-44c6-8f4c-1ed7d9617c5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "hits[5].contents.split('\\n')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein',\n",
              " \"Severe acute respiratory syndrome coronavirus nsp1 protein suppresses host gene expression, including type I interferon production, by promoting host mRNA degradation and inhibiting host translation, in infected cells. We present evidence that nsp1 uses a novel, two-pronged strategy to inhibit host translation/gene expression. Nsp1 bound to the 40S ribosomal subunit and inactivated the translational activity of the 40S subunits. Furthermore, the nsp1-40S ribosome complex induced the modification of the 5'-region of capped mRNA template and rendered the template RNA translationally incompetent. Nsp1 also induced RNA cleavage in templates carrying the internal ribosome entry site (IRES) from encephalomyocarditis virus, but not in those carrying IRESs from hepatitis C and cricket paralysis viruses, demonstrating that the nsp1-induced RNA modification was template-dependent. We speculate that the mRNAs that underwent the nsp1-mediated modification are marked for rapid turnover by the host RNA degradation machinery.\",\n",
              " \"Severe acute respiratory syndrome (SARS) coronavirus (SCoV), the causative agent of a newly emerged disease, SARS, is an enveloped virus that contains a single-stranded, positive-sense RNA genome of about 29.7 kb.1 Upon infection, the genome expression begins with the translation of gene 1, which constitutes the 5'-end two-thirds of the viral genome, to produce two large precursor polyproteins2,3 that are proteolytically processed by 2 virally encoded proteinases to generate 16 mature proteins, namely nsp1 to nsp162. While Users may view, print, copy, download and text and data-mine the content in such documents, for the purposes of academic research, subject always to the full Conditions of use: http://www.nature.com/authors/editorial_policies/license.html#terms most of these gene 1 proteins are essential for viral RNA synthesis, some of them appear to have other biological functions. [4] [5] [6] [7] The most N-terminal gene 1 protein, nsp1, has unique biological functions; expressed SCoV nsp1 induces host mRNA degradation and suppresses host translation8. The expressed nsp1 suppresses the host antiviral signaling pathways as well9. Furthermore, nsp1 suppresses host gene expression, including type I interferon (IFN) production, by promoting host mRNA degradation and host translation suppression in infected cells10. The nsp1 of a closely related mouse hepatitis virus also suppresses host gene expression and is a viral virulence factor11. These data suggest that the SCoV nsp1-mediated suppression of host genes plays an important role in the pathogenesis of SARS. Accordingly, a delineation of the mechanisms of the nsp1-induced suppression of host gene expression is considerably important for gaining insight into SCoV pathogenesis at the molecular level. We designed the present study to uncover the mechanism of nsp1-induced suppression of host gene expression primarily by using an in vitro system. Our data revealed that nsp1 utilizes a novel, two-pronged strategy to inhibit host translation/gene expression.\"]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qrwpdqXL3_5j",
        "colab_type": "text"
      },
      "source": [
        "The first two lines contain the title and abstract, respectively, and they are exactly the same for both, since they're from the same article."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iiXaE-TDH7uc",
        "colab_type": "text"
      },
      "source": [
        "To access the full text, we need to fetch the \"base\" document, which is `42saxb98` (without the `.XXXXX` suffix).\n",
        "This is to avoid wasting space by repeatedly storing the full text.\n",
        "\n",
        "We can use the `searcher` to fetch the document, and then fetch the underlying raw article JSON, as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WAYzpshzL7St",
        "colab_type": "code",
        "outputId": "53d56c18-5e98-4fdc-a6cc-fc9def5cd4a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "article = json.loads(searcher.doc('42saxb98').raw())\n",
        "\n",
        "# Uncomment to print the entire article... warning, it's long! :)\n",
        "#print(json.dumps(article, indent=4))\n",
        "\n",
        "article['metadata']['title']"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'A novel two-pronged strategy to suppress host protein synthesis by SARS coronavirus Nsp1 protein'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NerHPfUHynKc",
        "colab_type": "text"
      },
      "source": [
        "Finally, if you want to create a DataFrame comprising all the results, here's a snippet of code to do so:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fJggiIeTyipT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "outputId": "ce65b532-97c9-4c5a-87c6-6070b34254b0"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "ranks = list(range(1, len(hits)+1))\n",
        "docids = [ hit.docid for hit in hits]\n",
        "scores = [ hit.score for hit in hits]\n",
        "titles = [ hit.lucene_document.get('title') for hit in hits]\n",
        "dois = [ hit.lucene_document.get('doi') for hit in hits]\n",
        "data = {'rank': ranks, 'docid': docids, 'score': scores, 'title': titles, 'doi': dois} \n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rank</th>\n",
              "      <th>docid</th>\n",
              "      <th>score</th>\n",
              "      <th>title</th>\n",
              "      <th>doi</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>o328y8ax.00004</td>\n",
              "      <td>10.8783</td>\n",
              "      <td>Modulation of type I interferon induction by p...</td>\n",
              "      <td>10.1016/j.virol.2010.03.039</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>06z7p7rc</td>\n",
              "      <td>10.7991</td>\n",
              "      <td>Severe Acute Respiratory Syndrome Coronavirus ...</td>\n",
              "      <td>10.1128/jvi.02472-07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>ncufofro.00026</td>\n",
              "      <td>10.7740</td>\n",
              "      <td>Chapter Five Viral and Cellular mRNA Translati...</td>\n",
              "      <td>10.1016/bs.aivir.2016.08.001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>mtj46j82.00029</td>\n",
              "      <td>10.6258</td>\n",
              "      <td>MERS coronavirus nsp1 participates in an effic...</td>\n",
              "      <td>10.1016/j.virol.2017.08.026</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>42saxb98.00002</td>\n",
              "      <td>10.6143</td>\n",
              "      <td>A novel two-pronged strategy to suppress host ...</td>\n",
              "      <td>10.1038/nsmb.1680</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>42saxb98.00001</td>\n",
              "      <td>10.6096</td>\n",
              "      <td>A novel two-pronged strategy to suppress host ...</td>\n",
              "      <td>10.1038/nsmb.1680</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>vj3wk150.00035</td>\n",
              "      <td>10.6006</td>\n",
              "      <td>Regulation of Stress Responses and Translation...</td>\n",
              "      <td>10.3390/v8070184</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>sq9hh50d.00005</td>\n",
              "      <td>10.5451</td>\n",
              "      <td>Unique SARS-CoV protein nsp1: bioinformatics, ...</td>\n",
              "      <td>10.1016/j.tim.2006.12.005</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>42saxb98.00020</td>\n",
              "      <td>10.5415</td>\n",
              "      <td>A novel two-pronged strategy to suppress host ...</td>\n",
              "      <td>10.1038/nsmb.1680</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>pdfs6ojs.00027</td>\n",
              "      <td>10.5229</td>\n",
              "      <td>Coronavirus nonstructural protein 1: Common an...</td>\n",
              "      <td>10.1016/j.virusres.2014.11.019</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   rank  ...                             doi\n",
              "0     1  ...     10.1016/j.virol.2010.03.039\n",
              "1     2  ...            10.1128/jvi.02472-07\n",
              "2     3  ...    10.1016/bs.aivir.2016.08.001\n",
              "3     4  ...     10.1016/j.virol.2017.08.026\n",
              "4     5  ...               10.1038/nsmb.1680\n",
              "5     6  ...               10.1038/nsmb.1680\n",
              "6     7  ...                10.3390/v8070184\n",
              "7     8  ...       10.1016/j.tim.2006.12.005\n",
              "8     9  ...               10.1038/nsmb.1680\n",
              "9    10  ...  10.1016/j.virusres.2014.11.019\n",
              "\n",
              "[10 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    }
  ]
}