huggingface_pytorch_pretrained_bert_bert.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "huggingface_pytorch-pretrained-bert_bert.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4VivGJAWcRNn",
        "colab_type": "text"
      },
      "source": [
        "### This notebook is optionally accelerated with a GPU runtime.\n",
        "### If you would like to use this acceleration, please select the menu option \"Runtime\" -> \"Change runtime type\", select \"Hardware Accelerator\" -> \"GPU\" and click \"SAVE\"\n",
        "\n",
        "----------------------------------------------------------------------\n",
        "\n",
        "# BERT\n",
        "\n",
        "*Author: HuggingFace Team*\n",
        "\n",
        "**Bidirectional Encoder Representations from Transformers.**\n",
        "\n",
        "_ | _\n",
        "- | -\n",
        "![alt](https://pytorch.org/assets/images/bert1.png) | ![alt](https://pytorch.org/assets/images/bert2.png)\n",
        "\n",
        "\n",
        "### Model Description\n",
        "\n",
        "BERT was released together with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin et al. The model is based on the Transformer architecture introduced in [Attention Is All You Need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani et al and has led to significant improvements on a wide range of downstream tasks.\n",
        "\n",
        "Here are 8 models based on BERT with [Google's pre-trained models](https://github.com/google-research/bert) along with the associated Tokenizer.\n",
        "It includes:\n",
        "- `bertTokenizer`: perform end-to-end tokenization, i.e. basic tokenization followed by WordPiece tokenization\n",
        "- `bertModel`: raw BERT Transformer model (fully pre-trained)\n",
        "- `bertForMaskedLM`: BERT Transformer with the pre-trained masked language modeling head on top (fully pre-trained)\n",
        "- `bertForNextSentencePrediction`: BERT Transformer with the pre-trained next sentence prediction classifier on top (fully pre-trained)\n",
        "- `bertForPreTraining`: BERT Transformer with masked language modeling head and next sentence prediction classifier on top (fully pre-trained)\n",
        "- `bertForSequenceClassification`: BERT Transformer with a sequence classification head on top (BERT Transformer is pre-trained, the sequence classification head is only initialized and has to be trained)\n",
        "- `bertForMultipleChoice`: BERT Transformer with a multiple choice head on top (used for task like Swag) (BERT Transformer is pre-trained, the multiple choice classification head is only initialized and has to be trained)\n",
        "- `bertForTokenClassification`: BERT Transformer with a token classification head on top (BERT Transformer is pre-trained, the token classification head is only initialized and has to be trained)\n",
        "- `bertForQuestionAnswering`: BERT Transformer with a token classification head on top (BERT Transformer is pre-trained, the token classification head is only initialized and has to be trained)\n",
        "\n",
        "### Requirements\n",
        "\n",
        "Unlike most other PyTorch Hub models, BERT requires a few additional Python packages to be installed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mY19uZF8cRNw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "a2fc46a7-bdf9-46fd-ec9c-465d5c81009c"
      },
      "source": [
        "%%bash\n",
        "\n",
        "pip install tqdm boto3 requests regex"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (4.28.1)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (1.9.189)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (2.21.0)\n",
            "Collecting regex\n",
            "  Downloading https://files.pythonhosted.org/packages/6f/4e/1b178c38c9a1a184288f72065a65ca01f3154df43c6ad898624149b8b4e0/regex-2019.06.08.tar.gz (651kB)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3) (0.9.4)\n",
            "Requirement already satisfied: botocore<1.13.0,>=1.12.189 in /usr/local/lib/python3.6/dist-packages (from boto3) (1.12.189)\n",
            "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3) (0.2.1)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests) (2019.6.16)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests) (2.8)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests) (1.24.3)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.13.0,>=1.12.189->boto3) (2.5.3)\n",
            "Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.13.0,>=1.12.189->boto3) (0.14)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\"->botocore<1.13.0,>=1.12.189->boto3) (1.12.0)\n",
            "Building wheels for collected packages: regex\n",
            "  Building wheel for regex (setup.py): started\n",
            "  Building wheel for regex (setup.py): finished with status 'done'\n",
            "  Stored in directory: /root/.cache/pip/wheels/35/e4/80/abf3b33ba89cf65cd262af8a22a5a999cc28fbfabea6b38473\n",
            "Successfully built regex\n",
            "Installing collected packages: regex\n",
            "Successfully installed regex-2019.6.8\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HdLb-2MffrOG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from datetime import datetime\n",
        "from tensorflow import keras\n",
        "import os\n",
        "import re\n",
        "\n",
        "# Load all files from a directory in a DataFrame.\n",
        "def load_directory_data(directory):\n",
        "  data = {}\n",
        "  data[\"sentence\"] = []\n",
        "  data[\"sentiment\"] = []\n",
        "  for file_path in os.listdir(directory):\n",
        "    with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n",
        "      data[\"sentence\"].append(f.read())\n",
        "      data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n",
        "  return pd.DataFrame.from_dict(data)\n",
        "\n",
        "# Merge positive and negative examples, add a polarity column and shuffle.\n",
        "def load_dataset(directory):\n",
        "  pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n",
        "  neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n",
        "  pos_df[\"polarity\"] = 1\n",
        "  neg_df[\"polarity\"] = 0\n",
        "  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n",
        "\n",
        "# Download and process the dataset files.\n",
        "def download_and_load_datasets(force_download=False):\n",
        "  dataset = tf.keras.utils.get_file(\n",
        "      fname=\"aclImdb.tar.gz\", \n",
        "      origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
        "      extract=True)\n",
        "  \n",
        "  train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                       \"aclImdb\", \"train\"))\n",
        "  test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                      \"aclImdb\", \"test\"))\n",
        "  \n",
        "  return train_df, test_df\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OtCFIhO6ftDq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "f15c1747-9734-4301-af86-7c72a59d5c57"
      },
      "source": [
        "train, test = download_and_load_datasets()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
            "84131840/84125825 [==============================] - 3s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_v0a9QtZgELi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "68dbb643-6819-48e2-877b-60615957161b"
      },
      "source": [
        "train.head()\n"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>polarity</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I've been watching this movie by hoping to fin...</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Carlo Verdone once managed to combine superb c...</td>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>One of the more satisfying Western all'italian...</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>I love this movie. My only disappointment was ...</td>\n",
              "      <td>10</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The acting was horrendous as well as the scree...</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            sentence sentiment  polarity\n",
              "0  I've been watching this movie by hoping to fin...         3         0\n",
              "1  Carlo Verdone once managed to combine superb c...         4         0\n",
              "2  One of the more satisfying Western all'italian...         7         1\n",
              "3  I love this movie. My only disappointment was ...        10         1\n",
              "4  The acting was horrendous as well as the scree...         3         0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ogGE5S4gx0aK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WPMcBSbG2pZ9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 207
        },
        "outputId": "44f90409-d901-4cfd-aedd-0ad925cdba83"
      },
      "source": [
        "[i for i in train['sentence'][0:10]]"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I\\'ve been watching this movie by hoping to find a pretty and interesting story yet the story line wasn\\'t good at all. The play of the actors weren\\'t any better.<br /><br />Of course Shahrukh Khan was there yet he wasn\\'t enough to make this movie \"credible\" and interesting.<br /><br />I\\'ve read that this movie was based on the novel of Flaubert \"Madame Bovary\" yet for me I didn\\'t see it matching with the Indian mentality.<br /><br />In general we buy movie to dream and have a good time, not to waste our time and change our mood into worse. I just can\\'t understand how it could get such a \"high\" vote with an average of 6.8/10.<br /><br />So it\\'s the kind of movie you should run away & ignore because there is nothing to appreciate in it! You will just waste your time unless if you like \"dark movie\" with \"strange and non sense story\".',\n",
              " 'Carlo Verdone once managed to combine superb comedy with smart and subtle social analysis and criticism.<br /><br />Then something happened, and he turned into just another dull \"holier-than-thou\" director.<br /><br />Il Mio Miglior Nemico can more or less be summarized in one line \"working class = kind and warm, while upper-class = snob and devious. But love wins in the end\".<br /><br />Such a trite clichè for such a smart director.<br /><br />There isn\\'t really too much to talk about in the movie. Every character is a walking stereotype: the self-made-man who forgets his roots but who\\'ll become \"good\" again, the scorned wife, the rebellious rich girl who falls for the honest-but-poor guy... Acting is barely average.<br /><br />Severely disappointing under every aspect.',\n",
              " 'One of the more satisfying Western all\\'italiana, Johnny Yuma has the freshness of many WAI made during the heyday of the genre and is highly recommended for fans of the genre or offbeat, intelligent cinema.<br /><br />Johnny Yuma is, in most respects, not terribly original, but this actually does not count against it. The success of a genre film depends on how well it meets the audience\\'s expectations as well as provides surprising variations on these expected elements. Earlier, pleasing experiences are recreated but with subtle (or major) twist that provide continuing interest. The quality of the execution is also, obviously, important. A tired retread will be less successful than a sincere attempt to entertain or move the audience.<br /><br />Given these criteria, Johnny Yuma succeeds. There are numerous reprises of elements from earlier films. The setting is the brutal, twisted semi-feudal twilight world of shared by many of the best \"Gothic family\" westerns made 1964-1968 such as Tempo di massacre (1966). The plot is a combination of the basic Fistful of Dollars (1964) plot and the Ringo films, a fact not surprising as screenwriter Fendiando di Leo was involved in both. Di Leo was one of the best screenwriters in the popular cinema coming out of Cinecitta in the 1960s-70s and his work helped provide much of the thematic continuities and coherency to the genre (Along with a couple of other personalities in a few distinct circles of actors, directors, and screenwriters). In the FOD plot, the protagonist arrives in town, stirs up a tense situation, then undergoes a near-death followed by a resurrection (in some films, like Quella sporca storia nel west (1968) it is quite literally a crucifixion). The Catholic undertone to the narrative and the symbolism is intriguing, especially given the implicit populist/explicit socialist leanings of the filmmakers and their films. The Ringo plot, developed more fully by screenwriter Ernesto Gastaldi in a series of films starring Guliano Gemma, a egoistic protagonist chooses the interest of a community over his own through the medium of a relationship with a member of that community (with a healthy dash ironic uncertainty).<br /><br />The relationship between Carradine and Johnny is clearly based on that of Manco/Mortimer from a Fistful of Dollar (1965). The two scene of the exchange of the gun belts provides a clever dialog and understanding between the two. Numerous films, including Da uomo a uomo (1968) or even El Chuncho, quién sabe? (1967), use this relationship between an older and younger man (father/son, older/younger brother, Anglo adviser/adversary and peasant revolutionary) as a central dynamic to the plot.<br /><br />Additionally, there is the focus on deception and misdirection, mazes and mirrors, that recur throughout the best early WAI. The canons and pueblos of Almeria become literal mazes through which protagonist and antagonist play shifting games of cat and mouse.<br /><br />What distinguishes Johnny Yuma from other WAI is the quality of director Romolo Guerriri\\'s use of visual/psychological space together arrangement with the script\\'s intelligent mechanisms to forward the plot. Dialogue was never very important to the WAI and often absurdly unintelligible (thought there are exceptions, such as the cynical commentaries in Django (1966) or Faccia a faccia (1967).<br /><br />Psychological depth of character is created almost entirely through iconic imagery, it\\'s juxtapositions, and it\\'s description of the overall narrative situation. See how the presence of the deadly Samantha is felt during the beating scene \\x96 watching from the roof or from the background of the action. Or how Johnny strips Samantha and Pedro of their security and confidence in their power through his stealthy invasions of their ranch, hotel, even bedroom (this, again, is a theme from FOD). Finally, note how there is a focus on the search for information. Like many elements, this is borrowed from FOD which was ultimately based on the hard-boiled mystery novel Red Harvest. It is through incidental contacts, wanted posters, overheard conversations, glances out of windows, watches left in the dust, or mistaken identities and movements through the ripples created by the actions of Pedro and Samantha within this surreal and absurd reality that the narrative tacks forward to it\\'s conclusion.<br /><br />The movie was notable in it\\'s time for what were perceived of as excesses in violence. Of course, these films were hardly more violent than many American westerns. What was different was the psychological intensity of the violence and the causes to which it was attributed, which is to say that it was not the violence but it\\'s meaning that had changed. Johnny Yuma is distinct and interesting in it\\'s use and portrayal of violence and this is another interesting aspect of the film.<br /><br />What I personally find most interesting about most of this genre is the link it provides to the anonymous, nameless audiences in Italy and Spain to whom these recurrent narratives held some significance and interest. The artifact may have no intrinsic worth in and of itself \\x96 some flint debitage from a prehistoric site, a shard of cruse pottery, or a moldering piece of leather and rusted metal \\x96 but it is reference to some nameless presence, lives, that were significant simply because they existed. While Johnny Yuma has intrinsic worth, much of it\\'s interest for me derives from this connection and mystery.<br /><br />Top spaghetti western list http://imdb.com/mymovies/list?l=21849907<br /><br />Average SWs http://imdb.com/mymovies/list?l=21849889<br /><br />For fanatics only (bottom of the barrel) http://imdb.com/mymovies/list?l=21849890',\n",
              " 'I love this movie. My only disappointment was that some of the original songs were changed.<br /><br />It\\'s true that Frank Sinatra does not get a chance to sing as much in this movie but it\\'s also nice that it\\'s not just another Frank Sinatra movie where it\\'s mostly him doing the singing.<br /><br />I actually thought it was better to use Marlon Brando\\'s own voice as he has the voice that fits and I could not see someone with this great voice pulling off the gangster feel of his voice.<br /><br />Stubby Kaye\\'s \"Sit Down, You\\'re Rockin\\' the Boat\" is a foot-tappin\\', sing-a-long that I just love. He is a hard act to follow with his version and I still like his the best.<br /><br />Vivian Blaine is just excellent in this part and \"Adelaide\\'s Lament\" is my favorite of her songs.<br /><br />I really thought Jean Simmons was perfect for this part. Maybe I would not have first considered her but after seeing her in the part, it made sense.<br /><br />Michael Kidd\\'s choreography is timeless. If it were being re staged in the year 2008, I would not change a thing.<br /><br />I find that many times something is lost from the stage version to the movie version but this kept the feel of the stage, even though it was on film.<br /><br />I thought the movie was well cast. I performed in regional versions of this and it\\'s one of my favorites of that period.',\n",
              " 'The acting was horrendous as well as the screenplay. It was poorly put together and made you almost want to laugh at the several terribly acted out murder scenes. The ending was even worse. Everyone kept dying, but somehow the ending made it look like everything was perfectly OK! They did not give enough history about the obsession the teacher had, etc. The movie needed more time to perhaps develop a better storyline. The only reason I give this 3/10 is that I kind of feel bad for the young actors. They needed better coaching. They could have really made this an OK film, but the screenplay and acting failed miserably.',\n",
              " 'I can\\'t for the life of me remember why--I must have had a free ticket or something--but I saw this movie in the theater when it was released. I don\\'t remember who I went with, which theater I was in, or even which city. All I remember was how offended I was at this travesty someone dared to call a film, and how half the people in the theater walked out before the movie was over. Unfortunately I stuck it out to end, which I still consider to be one of the worst mistakes of my life thus far. My offense became pure horror when just before the closing credits the smarmy demon child sticks his head out from behind a sign and says \"Look for Problem Child 2, coming soon!\" That was hands-down THE most terrifying moment ever recorded on film.<br /><br />The plot, if I recall correctly, involved John Ritter and perhaps his wife (Lord, how I\\'ve tried without success to block this film out of my mind) adopting a \"problem child.\" Maybe they think they can reform him, or something. I really don\\'t know. If that was their intent, they fail miserably because from first frame to last this child remains the brattiest, rudest, most horrid demon-spawn ever to hit the big screen. Forget Damian, forget Rosemary\\'s Baby. This kid takes the cake. The only difference is, we are supposed to feel sorry for him because he\\'s a \"problem child.\" However, this is impossible since this child is quite likely the most unsympathetic character ever portrayed. You want to kill him through the entire film, and when (SPOILER, like anyone cares) John Ritter decides to keep the vile hell-child you will be yelling \"Send him back!\" in shocked disgust (like several of the people at the theater where I saw it did).<br /><br />This is only the second movie I have given a \"1\" to on the IMDb. The other was Superman IV, and by God I couldn\\'t tell you which was worse. John Ritter had a quote in TV Guide about the time that Problem Child 3, which he was not in, came out. He said something like \"The only way I would do another [Problem Child] sequel is if they dragged my dead body back to perform.\" Amen to that!<br /><br />I would rather watch a 24-hour marathon of Police Academy sequels than see even twenty minutes of Problem Child again. 1/10, only because I can\\'t give it a negative score, which is what it really deserves. Someone burn the original negatives of this film, please!',\n",
              " 'This is a terrible production of Bartleby, though not, as the other reviewer put it because it is \"unfilmable,\" but rather because this version does not maintain the spirit of the book. It tells the story, almost painfully so. Watching it, I could turn the pages in my book and follow along, which is not as much fun when dealing with an adaptation. Rather, see the 2001 version of Bartleby featuring Crispin Glover. That version, while humorous, brings new details to the film while maintaining the spirit of the novel. What\\'s important is the spirit, not the minutiae of things like setting, character names, and costumes. The difference between these film versions is like night and day, tedious and hilarious. This version is a lesson as to what can go wrong if an adaptation is handled poorly, painful, mind-numbing schlock.',\n",
              " 'Tobe Hooper has made great movies so I was certain this couldn\\'t be BAD. I didn\\'t read any reviews and tried to watch this unintentionally humorous film. At times this made me laugh, sometimes I almost fell asleep, sometimes made me almost CRY for Hooper.<br /><br />I rated this 3/10 because its 1990 \"horror\"-movie and many interesting or funny things happened there. Throughout the movie I was thinking something like \"they simply CAN\\'T add more things in this movie...\" .. but they did.<br /><br />Some tell this is some sort of Firestarter clone but truly isn\\'t. It\\'s based on that idea but thats all. This is combination of horror, comedy, weird religion/god things, funny gore, simple effects, drama, horrible acting, unbelievable script..and more.<br /><br />*spoilers* Story is: Government tries to create ultimate weapon using nuclear power or something and fails, during process child is born for 2 test persons. When mom sings to her child after the birth, both husband and wife burns and it is SPONTANEOUS COMBUSTION. Government buries whole thing and leaves this child live amongst other people and ... then after x years this kid is grown up and realizes he has been born for a reason and whoa he can burn things with his brains. Then everything goes unbelievable messy nothing really explains anything and .. Well when The Government realizes \"okay now he can set this fire thing to work\" they take him to normal hospital where is some nuclear toxic what they are going to use on this man BECAUSE they could kill him, no they can\\'t shoot him no! .. and argh, I guess thats enough to tell, I promise there is 100 more weird things in this movie.<br /><br />Well if you want good laughs watch this one. Gosh.',\n",
              " \"When John Singleton is on, he's *on*!! And this is one of his better films. Not quite as tight as Boyz-n-the-Hood, but close to it (and with much of the same stellar cast). This film was very well written, very well put together, and very well shot. There's very little to criticize, and most of my complaints are superficial (eg: where did Fudge get the money for 6 years of college and a lot of expensive stuff? No mention of a rich background... And why doesn't Professor Phibbs have an office? A professor of his stature *should* have one... And while we're at it, for an engineering student, hick or not, Remy's a pretty dumb character - I'd think that he'd have a bit more in the way of basic intelligence - he talks and acts like a total buffoon).<br /><br />But that aside, the film was very sharp. A good array of characters and points of view; and Singleton doesn't take sides in the story - many of the characters are unsympathetic, and he does a good job of interspersing the Panthers and Supremacist scenes together to show the folly on both sides.<br /><br />Much of the cinematography was excellent; I especially loved the scene where Kirsty Swanson gets intimate with Taryn and Wayne each scene spliced together really well. Also the Malik/Deja scenes were really well shot as well.<br /><br />The dialogue was a bit much at times; this film had a tendency to get *really* preachy at times, and it also tends to hammer the points it was making over your head when the points would be just as clear with out the bluntness (we really didn't need the US flag with 'UNLEARN' typed onto it, give some credit, we're not morons...). And to top it off, although *most* of the time Singleton uses melodrama quite well, sometimes it gets *way* too cheezy (like Deja's death, which is fine until she screams out 'WHY!!!' which simply ruined the entire effect and scene).<br /><br />But the acting, in general, was top of the line. Fabulous performances by Omar Epps (perhaps the best I've ever seen), Kirsty Swanson (who knew Buffy could act??), Michael Rapaport (surprised the hell out of me...after True Romance and Beautiful Girls I though he was a one-role actor), and of course Ice Cube and Laurence Fishburne are *always* outstanding.<br /><br />Downside? Jennifer Connelly was flat; though it's not completely her fault: her role was stereotypical and one-dimensional. Generic to the highest degree. And Tyra Banks, who had the role, was nothing short of horrid. She whined and whined and whined. Yet another in the long line of models-turned-actresses who failed miserably (though there are a few who prove the exception to this rule).<br /><br />Finally, the soundtrack! Wow! An amazing soundtrack (which is definitely worth buying!) which fits the film like a glove. Each scene has a twin song (although the Tori Amos songs started to *really* annoy me by the end...not her best work). Liz Phair, Rage Against the Machine, Ice Cube...how can one go wrong??<br /><br />All in all: a really good watch, a really strong cast, great script, great film. 8/10.\",\n",
              " \"1st watched 12/26/2008 -(Dir-Eugene Levy): Corny comedy murder mystery with very few laughs. The movie appears to be based on an earlier Italian movie according to the credits but was re-written by two fairly popular American romantic comedy writers. But this one by Charles Shyer & Nancy Meyers does not cut it compared to their other efforts. The story is about a couple of down-and-out traveling Americans, played by Richard Lewis and Sean Young, who stumble upon a lost dog and hope to make a fortune in reward money after seeing an ad in the paper for the dachsund's return. Upon trying to return it, they see a hand sticking out of a garage door at the lady's residence that they believe is attached to the rest of the dead body of the woman who is supposed to give them the money. They freak out and instead of contacting the police and telling them the truth they make out like runaways from the scene expecting to be framed for the murder. The other characters in the film are met on a train prior to this and hang around a Monte Carlo gambling resort doing various things to be pulled into the story. The other cast members include character actors John Candy, James Belushi, Cybill Shepherd, George Hamilton and others. After the police find out about the death, they start questioning the main characters and, of course, they have to work thru their goofy lies to figure out what really happened. None of the character actors mentioned earlier can bring this movie out of it's mediocre state despite some funny moments mostly provided by the Belushi/Shepherd couple. This isn't a horrible movie, it just isn't that good. There are plenty of average movies out there and this is just another one for the pile. Try it, maybe you'll like it, probably you won't.\"]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_ZdxmqoxFkC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "trainloader = torch.utils.data.DataLoader(train, batch_size=4,\n",
        "                                          shuffle=True, num_workers=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GCVJ_Ntmxo5z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 163
        },
        "outputId": "abbefa56-c4a0-4493-c3fe-47582590de8f"
      },
      "source": [
        "bert.run_classifer.InputExample"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-54-d8fd60752d3c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbert\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_classifer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mInputExample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'bert' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6TomBlhNcRN6",
        "colab_type": "text"
      },
      "source": [
        "### Example\n",
        "\n",
        "Here is an example on how to tokenize the input text with `bertTokenizer`, and then get the hidden states computed by `bertModel` or predict masked tokens using `bertForMaskedLM`. The example also includes snippets showcasing how to use `bertForNextSentencePrediction`, `bertForQuestionAnswering`, `bertForSequenceClassification`, `bertForMultipleChoice`, `bertForTokenClassification`, and `bertForPreTraining`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uBHCeul6cRN8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "3ccb8274-8714-45c0-fb17-e0a83f81975b"
      },
      "source": [
        "### First, tokenize the input\n",
        "import torch\n",
        "tokenizer = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'bertTokenizer', 'bert-base-cased', do_basic_tokenize=False)\n",
        "\n",
        "# Tokenized input\n",
        "text = \"[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]\"\n",
        "tokenized_text = tokenizer.tokenize(text)\n",
        "indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading: \"https://github.com/huggingface/pytorch-pretrained-BERT/archive/master.zip\" to /root/.cache/torch/hub/master.zip\n",
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0723 16:04:03.172120 140287905322880 tokenization_bert.py:190] The pre-trained model you are loading is a cased model but you have not set `do_lower_case` to False. We are setting `do_lower_case=False` for you but you may want to check this behavior.\n",
            "100%|██████████| 213450/213450 [00:00<00:00, 2378126.32B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oNMdHnSKglRK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "b78c9958-22a3-4472-9123-8fec6382fdcb"
      },
      "source": [
        "print(text)\n",
        "print(tokenized_text)\n",
        "print(indexed_tokens)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]\n",
            "['[CLS]', 'Who', 'was', 'Jim', 'He', '##nson', '?', '[SEP]', 'Jim', 'He', '##nson', 'was', 'a', 'puppet', '##eer', '[SEP]']\n",
            "[101, 2627, 1108, 3104, 1124, 15703, 136, 102, 3104, 1124, 15703, 1108, 170, 16797, 8284, 102]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u3IRRm8tcROD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### Get the hidden states computed by `bertModel`\n",
        "# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)\n",
        "segments_ids = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]\n",
        "\n",
        "# Convert inputs to PyTorch tensors\n",
        "segments_tensors = torch.tensor([segments_ids])\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gcgpfbsHs3OY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "a75fd9d2-a13f-40fd-eddb-b6519b773187"
      },
      "source": [
        "print(tokenizer.encode(\"Hello, my dog is cute\"))\n",
        "print(tokenizer.convert_tokens_to_ids(tokenizer.tokenize(\"Hello, my dog is cute\")))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[8667, 28136, 1139, 3676, 1110, 10509]\n",
            "[8667, 28136, 1139, 3676, 1110, 10509]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OV0H7IN7tQ9g",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "981fc613-fc3f-4715-e814-54e0233b5d1c"
      },
      "source": [
        "model = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'bertForSequenceClassification', 'bert-base-cased', num_labels=2)\n",
        "\n",
        "input_id = torch.tensor(tokenizer.encode(\"Hello, my dog is cute\")).unsqueeze(0) #batch_size 1\n",
        "label = torch.tensor(1).unsqueeze(0) # batch_size 1\n",
        "  \n",
        "outputs = model(input_id, labels = label)\n",
        "\n",
        "loss, logit = outputs"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using cache found in /root/.cache/torch/hub/huggingface_pytorch-pretrained-BERT_master\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6pq3QM7zvRhg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ee91a02b-4b25-46c7-d90a-453b59fe73c1"
      },
      "source": [
        "torch.nn.functional.softmax(logit, dim=1)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.3678, 0.6322]], grad_fn=<SoftmaxBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3GmdZblfv0cJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 1e-3\n",
        "num_total_steps = 1000\n",
        "num_warmup_steps = 100\n",
        "warmup_proportion = float(num_warmup_steps) / float(num_total_steps)  # 0.1\n",
        "\n",
        "### Previously BertAdam optimizer was instantiated like this:\n",
        "optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_total_steps)\n",
        "### and used like this:\n",
        "\n",
        "for batch in train_data:\n",
        "    loss = model(batch)\n",
        "    loss.backward()\n",
        "    optimizer.step()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H_eQyZpLcROl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "115f47f1-3e94-491a-94a3-f0238582e968"
      },
      "source": [
        "\n",
        "model = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'bertForSequenceClassification', 'bert-base-cased', num_labels=2)\n",
        "model.eval()\n",
        "\n",
        "# Predict the sequence classification logits\n",
        "with torch.no_grad():\n",
        "    seq_classif_logits = model(tokens_tensor, segments_tensors)\n",
        "\n",
        "# Or get the sequence classification loss (set model to train mode before if used for training)\n",
        "labels = torch.tensor([1])\n",
        "seq_classif_loss = model(tokens_tensor, segments_tensors, labels=labels)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using cache found in /root/.cache/torch/hub/huggingface_pytorch-pretrained-BERT_master\n"
          ],
          "name": "stderr"
        }
      ]
    }
  ]
}