Mackerel 16S Data Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mackerel Data Analysis\n",
    "Roughly following the structure of the QIIME 2 \"moving pictures\" tutorial -- this focuses on just getting the data ready for analysis in Songbird and Qurro.\n",
    "\n",
    "The data is from [study 11721 on Qiita](https://qiita.ucsd.edu/study/description/11721), and is associated with a manuscript currently in preparation (Minich et al. 2019, preprint available on bioRxiv [here](https://www.biorxiv.org/content/10.1101/721555v1)).\n",
    "\n",
    "The input data in this notebook is 150nt sOTU data, corresponding to artifact ID `56427` on Qiita. These data were processed on Qiita using QIIME 1.9.1 (`Split libraries FASTQ` and `Trimming`), then denoised using Deblur 1.1.0. The `reference-hit` BIOM table and FASTA file (both located in the `input/` directory) are used as the starting point for this analysis.\n",
    "\n",
    "**If you run into any trouble trying to replicate this analysis, feel free to raise an issue [on this analysis' GitHub repository](https://github.com/knightlab-analyses/qurro-mackerel-analysis) and I'll do my best to help troubleshoot the problem.**\n",
    "\n",
    "(Or, alternately, if you don't like how I did something here then also feel free to raise an issue ;)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Setting up\n",
    "### 0.1. Declare the locations of the input files and output directory\n",
    "Before rerunning this notebook on your system, there are two (hopefully painless) things you'll need to do:\n",
    "\n",
    "1. Download the SILVA 132 database ([link here](https://www.arb-silva.de/download/archive/qiime)), and\n",
    "2. Update the absolute filepaths in the cell below according to where on your system this notebook and/or the SILVA 132 database files are located.\n",
    "\n",
    "For reference, the taxonomic classification steps in this notebook use the SILVA 132 99% reference 16S sequences and majority 7-level taxonomy (downloaded from [here](https://www.arb-silva.de/download/archive/qiime) on November 10, 2019)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: INPUT_BIOM_TABLE_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.biom\n",
      "env: INPUT_REP_SEQS_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.seqs.fa\n",
      "env: INPUT_SAMPLE_METADATA_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/11721_prep_4638_qiime_20190722-104633.txt\n",
      "env: INPUT_REFERENCE_SEQS_PATH=/home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/rep_set/rep_set_16S_only/99/silva_132_99_16S.fna\n",
      "env: INPUT_REFERENCE_TAXONOMY_PATH=/home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/taxonomy/16S_only/99/majority_taxonomy_7_levels.txt\n",
      "env: OUTPUT_DIRECTORY=/home/mfedarko/analyses/qurro-mackerel-analysis/output\n"
     ]
    }
   ],
   "source": [
    "# Input Data Locations (trimmed-to-150-nt and deblurred BIOM table and sequences,\n",
    "# as well as sample metadata)\n",
    "%env INPUT_BIOM_TABLE_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.biom\n",
    "%env INPUT_REP_SEQS_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.seqs.fa\n",
    "%env INPUT_SAMPLE_METADATA_PATH=/home/mfedarko/analyses/qurro-mackerel-analysis/input/11721_prep_4638_qiime_20190722-104633.txt\n",
    "%env INPUT_REFERENCE_SEQS_PATH=/home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/rep_set/rep_set_16S_only/99/silva_132_99_16S.fna\n",
    "%env INPUT_REFERENCE_TAXONOMY_PATH=/home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/taxonomy/16S_only/99/majority_taxonomy_7_levels.txt\n",
    "\n",
    "# Output directory (will contain all .qza and .qzv files generated by this analysis)\n",
    "%env OUTPUT_DIRECTORY=/home/mfedarko/analyses/qurro-mackerel-analysis/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.2. Move into the output directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Moved into output directory: /home/mfedarko/analyses/qurro-mackerel-analysis/output\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "odir = os.environ[\"OUTPUT_DIRECTORY\"]\n",
    "os.chdir(odir)\n",
    "print(\"Moved into output directory: {}\".format(odir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.3. Get information about the current QIIME 2 environment\n",
    "(For future reference.)\n",
    "\n",
    "We assume that a QIIME 2 2019.7 conda environment is already activated when you start up this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSystem versions\u001b[0m\n",
      "Python version: 3.6.7\n",
      "QIIME 2 release: 2019.7\n",
      "QIIME 2 version: 2019.7.0\n",
      "q2cli version: 2019.7.0\n",
      "\u001b[32m\n",
      "Installed plugins\u001b[0m\n",
      "alignment: 2019.7.0\n",
      "composition: 2019.7.0\n",
      "cutadapt: 2019.7.0\n",
      "dada2: 2019.7.0\n",
      "deblur: 2019.7.0\n",
      "deicode: 0.2.3\n",
      "demux: 2019.7.0\n",
      "diversity: 2019.7.0\n",
      "emperor: 2019.7.0\n",
      "feature-classifier: 2019.7.0\n",
      "feature-table: 2019.7.0\n",
      "fragment-insertion: 2019.7.0\n",
      "gneiss: 2019.7.0\n",
      "longitudinal: 2019.7.0\n",
      "metadata: 2019.7.0\n",
      "phylogeny: 2019.7.0\n",
      "quality-control: 2019.7.0\n",
      "quality-filter: 2019.7.0\n",
      "qurro: 0.4.0\n",
      "sample-classifier: 2019.7.0\n",
      "songbird: 1.0.1\n",
      "taxa: 2019.7.0\n",
      "types: 2019.7.0\n",
      "vsearch: 2019.7.0\n",
      "\u001b[32m\n",
      "Application config directory\u001b[0m\n",
      "/home/mfedarko/.config/q2cli\u001b[0m\n",
      "\u001b[32m\n",
      "Getting help\u001b[0m\n",
      "To get help with QIIME 2, visit https://qiime2.org\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import data into QIIME 2 artifacts\n",
    "See [the QIIME 2 documentation on importing data](https://docs.qiime2.org/2019.4/tutorials/importing/) for context on why this is necessary and useful.\n",
    "\n",
    "### 1.1. Import the study's data\n",
    "Note that this dataset doesn't just contain data about the microbiota of pacific chub mackerel: it also contains samples taken from other species of fish, as well as well as environmental samples. We'll filter some of these samples out of the dataset soon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported /home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.biom as BIOMV210DirFmt to table-unfiltered.qza\u001b[0m\n",
      "\u001b[32mImported /home/mfedarko/analyses/qurro-mackerel-analysis/input/reference-hit.seqs.fa as DNASequencesDirectoryFormat to rep-seqs-unfiltered.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime tools import \\\n",
    "    --type \"FeatureTable[Frequency]\" \\\n",
    "    --input-path $INPUT_BIOM_TABLE_PATH \\\n",
    "    --output-path table-unfiltered.qza\n",
    "!qiime tools import \\\n",
    "    --type \"FeatureData[Sequence]\" \\\n",
    "    --input-path $INPUT_REP_SEQS_PATH \\\n",
    "    --output-path rep-seqs-unfiltered.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Summarize the imported study table and representative sequence data\n",
    "This gives us information about the number of samples and sequences present in these files. It's useful for sanity-checking the filtering that will be done in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: table-unfiltered-summary.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: rep-seqs-unfiltered-summary.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "    --i-table table-unfiltered.qza \\\n",
    "    --o-visualization table-unfiltered-summary.qzv \\\n",
    "    --m-sample-metadata-file $INPUT_SAMPLE_METADATA_PATH\n",
    "\n",
    "!qiime feature-table tabulate-seqs \\\n",
    "    --i-data rep-seqs-unfiltered.qza \\\n",
    "    --o-visualization rep-seqs-unfiltered-summary.qzv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Filter the study's data\n",
    "In particular, we'll filter the feature table (and representative sequences) to only contain samples that satisfy both of the following two conditions:\n",
    "\n",
    "1. have a `host_common_name` of `pacific chub mackerel` OR have a `sample_type_body_site` of `sea water`\n",
    "2. contain at least 1,370 sequences\n",
    "\n",
    "### 2.1. Why do we filter to just pacific chub mackerel and sea water samples?\n",
    "If you examine `table-unfiltered-summary.qzv` (in particular the \"Interactive Sample Detail\" tab), you should see that only 1,167 / 1,522 samples have a `host_common_name` of `pacific chub mackerel`. We're going to look at how samples taken from various body sites of these mackerel differ from environmental samples (in particular, just samples taken from sea water).\n",
    "\n",
    "In order to perform this analysis, we filter the table to just samples where `host_common_name` is `pacific chub mackerel` *or* samples where `sample_type_body_site` is `sea water`.\n",
    "\n",
    "### 2.2. Why do we filter out samples with less than 1,370 sequences?\n",
    "This isn't rarefaction—the remaining samples have all their sequences preserved, at least until Songbird imposes its default cutoffs—but samples with less than this number of sequences are removed from the analysis from here on down.\n",
    "\n",
    "This number was obtained using the KatharoSeq protocol, which involves looking at the positive controls processed alongside these samples: see [Minich et al. 2018](https://msystems.asm.org/content/3/3/e00218-17.abstract) for a description of how KatharoSeq works. (The actual sample exclusion threshold number for this dataset, on the `reference-hit` table processed through Deblur 1.1.0, was 1,369.5, but we round up here since \"half a count\" doesn't really make sense.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: table.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureData[Sequence] to: rep-seqs.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table filter-samples \\\n",
    "    --i-table table-unfiltered.qza \\\n",
    "    --m-metadata-file $INPUT_SAMPLE_METADATA_PATH \\\n",
    "    --p-where \"host_common_name='pacific chub mackerel' OR sample_type_body_site='sea water'\" \\\n",
    "    --p-min-frequency 1370 \\\n",
    "    --o-filtered-table table.qza\n",
    "\n",
    "# Filter rep-seqs-unfiltered.qza to only include sequences present in the partially-filtered table\n",
    "!qiime feature-table filter-seqs \\\n",
    "    --i-table table.qza \\\n",
    "    --i-data rep-seqs-unfiltered.qza \\\n",
    "    --o-filtered-data rep-seqs.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Summarize the filtered data\n",
    "This will let us double-check that the filtering steps above were done properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: table-summary.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: rep-seqs-summary.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-table summarize \\\n",
    "    --i-table table.qza \\\n",
    "    --m-sample-metadata-file $INPUT_SAMPLE_METADATA_PATH \\\n",
    "    --o-visualization table-summary.qzv\n",
    "\n",
    "!qiime feature-table tabulate-seqs \\\n",
    "    --i-data rep-seqs.qza \\\n",
    "    --o-visualization rep-seqs-summary.qzv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Taxonomic classification\n",
    "You don't *need* taxonomy information (also known as \"feature metadata\", in the sense that taxonomy is information about your features [provided that, uh, your features are sequences]) to run Qurro (or Songbird or DEICODE, for that matter). However, having this information available is extremely useful in interpreting a Qurro visualization -- this is why we'll perform taxonomic classification on our dataset's features.\n",
    "\n",
    "We're going to do this taxonomic classification using q2-feature-classifier's `classify-sklearn` Naive Bayes classifier (see [Bokulich et al. 2018](https://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-018-0470-z)) and based on the SILVA 132 database (see [Quast et al. 2012](https://academic.oup.com/nar/article/41/D1/D590/1069277)). Rather than use [the pre-trained QIIME 2 SILVA classifier that targets the V4 region](https://docs.qiime2.org/2019.7/data-resources/), we train our own because our samples' primers are very slightly different:\n",
    "\n",
    "Forward primers:\n",
    "```\n",
    " This study's samples: GTGYCAGCMGCCGCGGTAA\n",
    "Pretrained Classifier: GTGCCAGCMGCCGCGGTAA\n",
    "```\n",
    "\n",
    "Reverse primers:\n",
    "```\n",
    " This study's samples: GGACTACNVGGGTWTCTAAT\n",
    "Pretrained Classifier: GGACTACHVGGGTWTCTAAT\n",
    "```\n",
    "\n",
    "So we can see that there are two differences in the primers (`Y` vs. `C` in the forward, and `N` vs. `H` in the reverse). As is explained on the [EMP page describing 16S protocols](http://press.igsb.anl.gov/earthmicrobiome/protocols-and-standards/16s/), these differences seem to be due to the EMP-protocol primers used to target the V4 region being updated: this study uses the \"updated\" primers, while the pre-trained QIIME 2 SILVA classifier uses the \"original\" primers (at least, as of QIIME 2 2019.7).\n",
    "\n",
    "This discrepancy [really shouldn't make a difference](https://forum.qiime2.org/t/primer-sequences-used-for-trained-silva119-515f-806r/2888/6), but I figure we might as well account for this anyway just so we can be super extra confident about our results.\n",
    "\n",
    "Note that, during the process of importing SILVA files / training the classifier, most of the steps / parameters taken were based on replicating the provenance of the SILVA 515F/806R pre-trained QIIME 2 Naive Bayes classifier available at https://docs.qiime2.org/2019.7/data-resources/ (as well as the [tutorial on training feature classifiers](https://docs.qiime2.org/2019.10/tutorials/feature-classifier/), of course).\n",
    "\n",
    "### 4.1. Import SILVA database files\n",
    "\n",
    "#### 4.1.1. Why do we use the `rep_set` sequence data, rather than the `rep_set_aligned` sequence data?\n",
    "I wasn't sure about this, so I did some digging (see [this post](https://forum.qiime2.org/t/classifier-training-questions/1162/3)) and found out that `rep_set` is what should be used with QIIME 2's classifiers. (That particular post is describing Greengenes, not SILVA, but the SILVA 132 documentation makes it clear that these folders are mostly analogous to those in Greengenes.)\n",
    "\n",
    "#### 4.1.2. Why do we use the majority 7-level taxonomy?\n",
    "The 7-level thing is because we're working with 16S data (i.e. our sequences should all be derived from bacterial or archaeal genomes), and as I understand it using the 7-level taxonomy is needed to get many classifiers working properly.\n",
    "\n",
    "The use of the \"majority\" taxonomy (instead of the raw or consensus taxonomy) is a bit more subjective; since I am by no means a phylogenetics expert, I went with the majority taxonomy because it seems to be a reasonable thing to do, based on comments from both [QIIME 2](https://forum.qiime2.org/t/silva-taxonomy-classifier-clarification/5087/3) and [SILVA](https://forum.qiime2.org/t/silva-132-classifiers/3698/11) folks.\n",
    "\n",
    "#### 4.1.3. Why did you bother explaining that? That was all obvious to me!\n",
    "\n",
    "Well, a lot of it wasn't obvious to me! ;) I'm trying to be this explicit for my own sanity, so I can come back to this notebook in the future and understand why I did the things I did. Furthermore, a lot of this is information that isn't laid out in a single clear place (at least that I could find), which is why I had to go searching for it... maybe seeing a description like this will help someone with their analysis.\n",
    "\n",
    "That all being said, I'm sure you could rerun taxonomic classification (or really, any other part of this notebook) with slightly different files/parameters and maybe get slightly different results, but I'll be honest -- at this point I've reran this notebook upwards of 5 times (I think, I haven't been keeping track) because I keep finding things to improve. So, I've reran it with different classifiers, different reference databases, ..., and things look mostly the same regardless."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mImported /home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/rep_set/rep_set_16S_only/99/silva_132_99_16S.fna as DNASequencesDirectoryFormat to silva-132-99-16S-seqs.qza\u001b[0m\n",
      "\u001b[32mImported /home/mfedarko/analyses/silva-132/SILVA_132_QIIME_release/taxonomy/16S_only/99/majority_taxonomy_7_levels.txt as HeaderlessTSVTaxonomyFormat to silva-132-99-16S-majority-7-level-taxonomy.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime tools import \\\n",
    "    --type \"FeatureData[Sequence]\" \\\n",
    "    --input-path $INPUT_REFERENCE_SEQS_PATH \\\n",
    "    --output-path silva-132-99-16S-seqs.qza\n",
    "\n",
    "!qiime tools import \\\n",
    "    --type \"FeatureData[Taxonomy]\" \\\n",
    "    --input-format HeaderlessTSVTaxonomyFormat \\\n",
    "    --input-path $INPUT_REFERENCE_TAXONOMY_PATH \\\n",
    "    --output-path silva-132-99-16S-majority-7-level-taxonomy.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Extract reads based on primers\n",
    "We use the exact same primers used in this study (being able to use our exact primers is why we're going through the trouble of training a classifier in the first place...), and otherwise use default parameters. (The use of these default parameters seems to match up with the parameters used when extracting reads for the aforemented pre-trained SILVA 515F/806R QIIME 2 classifier, based on that classifier's provenance.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureData[Sequence] to: silva-132-99-16S-extracted-seqs.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-classifier extract-reads \\\n",
    "    --i-sequences silva-132-99-16S-seqs.qza \\\n",
    "    --p-f-primer GTGYCAGCMGCCGCGGTAA \\\n",
    "    --p-r-primer GGACTACNVGGGTWTCTAAT \\\n",
    "    --verbose \\\n",
    "    --o-reads silva-132-99-16S-extracted-seqs.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3. Train the classifier\n",
    "I haven't split this across multiple jobs because I couldn't find an option to do that for this command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_feature_classifier/classifier.py:102: UserWarning: The TaxonomicClassifier artifact that results from this method was trained using scikit-learn version 0.21.2. It cannot be used with other versions of scikit-learn. (While the classifier may complete successfully, the results will be unreliable.)\n",
      "  warnings.warn(warning, UserWarning)\n",
      "[Pipeline] .......... (step 1 of 2) Processing feat_ext, total=  42.0s\n",
      "[Pipeline] .......... (step 2 of 2) Processing classify, total=99.5min\n",
      "\u001b[32mSaved TaxonomicClassifier to: silva-132-99-16S-v4-classifier.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-classifier fit-classifier-naive-bayes \\\n",
    "    --i-reference-reads silva-132-99-16S-extracted-seqs.qza \\\n",
    "    --i-reference-taxonomy silva-132-99-16S-majority-7-level-taxonomy.qza \\\n",
    "    --p-verbose \\\n",
    "    --verbose \\\n",
    "    --o-classifier silva-132-99-16S-v4-classifier.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4. Actually use the classifier to perform taxonomic classification!\n",
    "Phew."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureData[Taxonomy] to: taxonomy.qza\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime feature-classifier classify-sklearn \\\n",
    "    --i-reads rep-seqs.qza \\\n",
    "    --i-classifier silva-132-99-16S-v4-classifier.qza \\\n",
    "    --p-n-jobs 4 \\\n",
    "    --verbose \\\n",
    "    --o-classification taxonomy.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Run Songbird\n",
    "This will generate feature differentials, which we'll visualize in Qurro.\n",
    "For details on what Songbird does and how it works, please see [Songbird's GitHub page](https://github.com/biocore/songbird/), as well as [Morton and Marotz et al. 2019](https://www.nature.com/articles/s41467-019-10656-5)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1. Why do we manually set certain Songbird parameters?\n",
    "These parameters were chosen based on consulting with the `songbird-regression-summary.qzv` generated below to ensure that they resulted in a reasonable model fit. See [this section of Songbird's README](https://github.com/biocore/songbird#interpreting-model-fitting) for details on model fitting.\n",
    "\n",
    "#### `--p-formula`\n",
    "This parameter is used by Songbird to determine what sample metadata fields should be used as covariates when generating differentials. Here, we generate differentials relative to the `sample_type_body_site` field (using the `sea water` values of this field as a reference), but there are of course plenty of other options for fields that could be used here if we'd like to ask different questions about this data.\n",
    "\n",
    "#### `--p-epochs`, `--p-learning-rate`\n",
    "These parameters, in particular, are closely related to Songbird's runtime:\n",
    "\n",
    "- We've increased `--p-epochs` from the default of 1,000 to 10,000 to make Songbird run longer (we're working with a fairly large dataset).\n",
    "- We've decreased `--p-learning-rate` from the default of 0.001 to 0.0001 to similarly increase Songbird's run time.\n",
    "\n",
    "#### `--p-batch-size`\n",
    "We've increased `--p-batch-size` from the default of 5 to 40 to make Songbird process a larger amount of samples at once in each iteration. Since we have a lot of samples in this dataset, and since our samples fall into six \"categories\" (the five mackerel body sites, plus sea water samples), using a larger batch size (that stands a better chance of reflecting this diversity) makes sense.\n",
    "\n",
    "#### `--p-num-random-test-examples`\n",
    "Quoting Songbird's documentation again, this is \"\\[the number\\] of random samples to hold out for cross-validation if `training-column` is not specified.\" The default for this is 5 (i.e. use just 5 samples for cross-validation); since we have the luxury of having a lot of samples in this dataset, we can afford to hold out more samples. This is why we've increased this to 40 samples, analogously to how and why we increased `--p-batch-size`.\n",
    "\n",
    "#### `--p-summary-interval`\n",
    "This is the frequency (in seconds) of how often Songbird saves model fitting statistics in the `--o-regression-stats` output artifact. More frequent measurements will help us more accurately diagnose if Songbird's model is fitting reasonably to the dataset.\n",
    "\n",
    "### 5.2. A note about Songbird's default `--p-min-sample-count` and `--p-min-feature-count`\n",
    "Songbird imposes these cutoffs by default (the default values are 1,000 and 10 respectively), and we don't change them. This means that:\n",
    "\n",
    "- Samples with less than 1,000 counts total will not be used in Songbird's analysis\n",
    "  - This shouldn't impact this analysis, since we already filtered out samples with less than 1,370 counts due to using KatharoSeq in section 2.\n",
    "  - Also, the sample filtering is done before the feature filtering in Songbird's code, at least [as of writing](https://github.com/biocore/songbird/blob/beceeec550f775e64f371ede96af33218fff8b35/songbird/util.py#L154-L165). This means that we don't have to worry about silly situations where all samples initially pass the cutoff, but then we filter out some features, and then suddenly those samples don't pass the cutoff any more when the sample filtering is done...\n",
    "  \n",
    "  \n",
    "- Features present in less than 10 samples will not be used in Songbird's analysis\n",
    "  - This results in a decent amount of features being excluded from the differentials produced here.\n",
    "  - Qurro recognizes this sort of situation by default, and will explicitly warn you about the dropped features -- see the text printed out when we run Qurro at the bottom of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W1115 21:22:45.085559 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/q2/_method.py:53: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "2019-11-15 21:22:45.086679: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2019-11-15 21:22:45.096304: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2600030000 Hz\n",
      "2019-11-15 21:22:45.099089: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562e31424970 executing computations on platform Host. Devices:\n",
      "2019-11-15 21:22:45.099127: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.\n",
      "OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info\n",
      "OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-31\n",
      "OMP: Info #156: KMP_AFFINITY: 32 available OS procs\n",
      "OMP: Info #157: KMP_AFFINITY: Uniform topology\n",
      "OMP: Info #179: KMP_AFFINITY: 2 packages x 8 cores/pkg x 2 threads/core (16 total cores)\n",
      "OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:\n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 7 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 7 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 1 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 24 maps to package 1 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 1 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 25 maps to package 1 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 1 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 26 maps to package 1 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 1 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 27 maps to package 1 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 1 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 28 maps to package 1 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 1 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 29 maps to package 1 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 1 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 30 maps to package 1 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 1 core 7 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 31 maps to package 1 core 7 thread 1 \n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23217 thread 0 bound to OS proc set 0\n",
      "2019-11-15 21:22:45.102065: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "W1115 21:22:45.133584 140336167892800 deprecation.py:323] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:75: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.random.categorical` instead.\n",
      "W1115 21:22:45.149364 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:86: The name tf.random_normal is deprecated. Please use tf.random.normal instead.\n",
      "\n",
      "W1115 21:22:45.170744 140336167892800 deprecation.py:323] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:91: Normal.__init__ (from tensorflow.python.ops.distributions.normal) is deprecated and will be removed after 2019-01-01.\n",
      "Instructions for updating:\n",
      "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n",
      "W1115 21:22:45.173031 140336167892800 deprecation.py:323] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/tensorflow/python/ops/distributions/normal.py:160: Distribution.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.\n",
      "Instructions for updating:\n",
      "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n",
      "W1115 21:22:45.178669 140336167892800 deprecation.py:323] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:100: Multinomial.__init__ (from tensorflow.python.ops.distributions.multinomial) is deprecated and will be removed after 2019-01-01.\n",
      "Instructions for updating:\n",
      "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n",
      "W1115 21:22:45.196078 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:115: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
      "\n",
      "W1115 21:22:45.223397 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:121: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "W1115 21:22:45.324239 140336167892800 deprecation.py:323] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py:286: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "W1115 21:22:45.355055 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:129: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.\n",
      "\n",
      "W1115 21:22:45.356424 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:130: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.\n",
      "\n",
      "W1115 21:22:45.357331 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:136: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.\n",
      "\n",
      "2019-11-15 21:22:45.376625: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1115 21:22:45.385093 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:163: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
      "\n",
      "  0%|                                                | 0/140000 [00:00<?, ?it/s]W1115 21:22:45.399310 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:172: The name tf.RunOptions is deprecated. Please use tf.compat.v1.RunOptions instead.\n",
      "\n",
      "W1115 21:22:45.399477 140336167892800 deprecation_wrapper.py:119] From /home/mfedarko/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:174: The name tf.RunMetadata is deprecated. Please use tf.compat.v1.RunMetadata instead.\n",
      "\n",
      "2019-11-15 21:22:45.636025: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23355 thread 1 bound to OS proc set 1\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23356 thread 2 bound to OS proc set 2\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23359 thread 4 bound to OS proc set 4\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23358 thread 3 bound to OS proc set 3\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23360 thread 5 bound to OS proc set 5\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23361 thread 6 bound to OS proc set 6\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23363 thread 8 bound to OS proc set 8\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23362 thread 7 bound to OS proc set 7\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23364 thread 9 bound to OS proc set 9\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23365 thread 10 bound to OS proc set 10\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23367 thread 12 bound to OS proc set 12\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23366 thread 11 bound to OS proc set 11\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23368 thread 13 bound to OS proc set 13\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23369 thread 14 bound to OS proc set 14\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23370 thread 15 bound to OS proc set 15\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23371 thread 16 bound to OS proc set 16\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23372 thread 17 bound to OS proc set 17\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23373 thread 18 bound to OS proc set 18\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23374 thread 19 bound to OS proc set 19\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23375 thread 20 bound to OS proc set 20\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23376 thread 21 bound to OS proc set 21\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23377 thread 22 bound to OS proc set 22\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23378 thread 23 bound to OS proc set 23\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23379 thread 24 bound to OS proc set 24\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23380 thread 25 bound to OS proc set 25\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23381 thread 26 bound to OS proc set 26\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23382 thread 27 bound to OS proc set 27\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23383 thread 28 bound to OS proc set 28\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23384 thread 29 bound to OS proc set 29\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23385 thread 30 bound to OS proc set 30\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23386 thread 31 bound to OS proc set 31\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23387 thread 32 bound to OS proc set 0\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23388 thread 33 bound to OS proc set 1\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23389 thread 34 bound to OS proc set 2\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23390 thread 35 bound to OS proc set 3\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23391 thread 36 bound to OS proc set 4\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23392 thread 37 bound to OS proc set 5\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23394 thread 39 bound to OS proc set 7\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23393 thread 38 bound to OS proc set 6\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23395 thread 40 bound to OS proc set 8\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23396 thread 41 bound to OS proc set 9\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23397 thread 42 bound to OS proc set 10\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23398 thread 43 bound to OS proc set 11\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23399 thread 44 bound to OS proc set 12\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23400 thread 45 bound to OS proc set 13\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23401 thread 46 bound to OS proc set 14\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23402 thread 47 bound to OS proc set 15\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23403 thread 48 bound to OS proc set 16\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23404 thread 49 bound to OS proc set 17\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23405 thread 50 bound to OS proc set 18\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23406 thread 51 bound to OS proc set 19\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23407 thread 52 bound to OS proc set 20\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23408 thread 53 bound to OS proc set 21\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23409 thread 54 bound to OS proc set 22\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23410 thread 55 bound to OS proc set 23\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23411 thread 56 bound to OS proc set 24\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23412 thread 57 bound to OS proc set 25\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23413 thread 58 bound to OS proc set 26\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23414 thread 59 bound to OS proc set 27\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23416 thread 61 bound to OS proc set 29\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23415 thread 60 bound to OS proc set 28\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23417 thread 62 bound to OS proc set 30\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23418 thread 63 bound to OS proc set 31\n",
      "OMP: Info #250: KMP_AFFINITY: pid 23217 tid 23419 thread 64 bound to OS proc set 0\n",
      "  0%|                                    | 215/140000 [00:00<1:42:19, 22.77it/s]2019-11-15 21:22:46.401572: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  0%|▏                                    | 666/140000 [00:01<08:31, 272.51it/s]2019-11-15 21:22:47.401558: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  1%|▎                                   | 1216/140000 [00:02<04:17, 538.89it/s]2019-11-15 21:22:48.403511: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  1%|▍                                   | 1791/140000 [00:03<04:07, 557.46it/s]2019-11-15 21:22:49.402691: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  2%|▌                                   | 2368/140000 [00:04<04:00, 572.15it/s]2019-11-15 21:22:50.403705: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  2%|▊                                   | 2947/140000 [00:05<04:01, 567.13it/s]2019-11-15 21:22:51.404286: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  2%|▉                                   | 3466/140000 [00:06<04:06, 553.24it/s]2019-11-15 21:22:52.405777: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  3%|█                                   | 4045/140000 [00:07<04:03, 557.35it/s]2019-11-15 21:22:53.406247: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  3%|█▏                                  | 4614/140000 [00:08<04:01, 561.50it/s]2019-11-15 21:22:54.406609: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  4%|█▎                                  | 5195/140000 [00:09<03:54, 573.92it/s]2019-11-15 21:22:55.408476: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  4%|█▍                                  | 5781/140000 [00:10<03:54, 573.12it/s]2019-11-15 21:22:56.408567: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  5%|█▌                                  | 6306/140000 [00:11<03:54, 569.68it/s]2019-11-15 21:22:57.409663: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  5%|█▊                                  | 6876/140000 [00:12<03:57, 560.24it/s]2019-11-15 21:22:58.409837: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  5%|█▉                                  | 7445/140000 [00:13<03:54, 565.24it/s]2019-11-15 21:22:59.411186: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  6%|██                                  | 8019/140000 [00:14<03:55, 559.74it/s]2019-11-15 21:23:00.412103: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  6%|██▏                                 | 8603/140000 [00:16<03:59, 548.36it/s]2019-11-15 21:23:01.412188: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  6%|██▎                                 | 9097/140000 [00:16<04:06, 530.38it/s]2019-11-15 21:23:02.412308: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  7%|██▍                                 | 9668/140000 [00:17<03:50, 565.64it/s]2019-11-15 21:23:03.413072: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  7%|██▌                                | 10235/140000 [00:18<03:53, 555.57it/s]2019-11-15 21:23:04.414713: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  8%|██▋                                | 10819/140000 [00:19<03:44, 575.48it/s]2019-11-15 21:23:05.415609: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  8%|██▊                                | 11400/140000 [00:20<03:43, 576.50it/s]2019-11-15 21:23:06.417179: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  9%|██▉                                | 11983/140000 [00:22<03:44, 571.41it/s]2019-11-15 21:23:07.417832: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  9%|███                                | 12497/140000 [00:22<03:46, 562.47it/s]2019-11-15 21:23:08.419108: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "  9%|███▎                               | 13065/140000 [00:23<03:49, 553.27it/s]2019-11-15 21:23:09.420174: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 10%|███▍                               | 13632/140000 [00:24<03:46, 558.72it/s]2019-11-15 21:23:10.420532: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 10%|███▌                               | 14200/140000 [00:25<03:42, 566.58it/s]2019-11-15 21:23:11.421083: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 11%|███▋                               | 14770/140000 [00:26<03:44, 558.55it/s]2019-11-15 21:23:12.421125: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 11%|███▊                               | 15352/140000 [00:28<03:35, 577.92it/s]2019-11-15 21:23:13.421666: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 11%|███▉                               | 15867/140000 [00:28<03:42, 557.21it/s]2019-11-15 21:23:14.422526: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 12%|████                               | 16435/140000 [00:29<03:37, 568.65it/s]2019-11-15 21:23:15.425686: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 12%|████▎                              | 17010/140000 [00:30<03:34, 572.58it/s]2019-11-15 21:23:16.424886: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 13%|████▍                              | 17591/140000 [00:31<03:35, 567.52it/s]2019-11-15 21:23:17.426551: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 13%|████▌                              | 18167/140000 [00:32<03:33, 569.91it/s]2019-11-15 21:23:18.427849: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 13%|████▋                              | 18739/140000 [00:34<03:31, 572.61it/s]2019-11-15 21:23:19.428170: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 14%|████▊                              | 19249/140000 [00:34<03:36, 556.52it/s]2019-11-15 21:23:20.429615: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 14%|████▉                              | 19828/140000 [00:35<03:30, 570.11it/s]2019-11-15 21:23:21.430966: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 15%|█████                              | 20394/140000 [00:36<03:40, 541.83it/s]2019-11-15 21:23:22.432202: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 15%|█████▏                             | 20971/140000 [00:38<03:30, 565.04it/s]2019-11-15 21:23:23.434097: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 15%|█████▍                             | 21534/140000 [00:39<03:33, 554.46it/s]2019-11-15 21:23:24.434277: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 16%|█████▌                             | 22045/140000 [00:39<03:32, 554.47it/s]2019-11-15 21:23:25.435411: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 16%|█████▋                             | 22616/140000 [00:40<03:29, 560.07it/s]2019-11-15 21:23:26.435743: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 17%|█████▊                             | 23195/140000 [00:41<03:22, 577.46it/s]2019-11-15 21:23:27.436444: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 17%|█████▉                             | 23776/140000 [00:42<03:25, 566.10it/s]2019-11-15 21:23:28.438109: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 17%|██████                             | 24354/140000 [00:44<03:27, 558.59it/s]2019-11-15 21:23:29.439204: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 18%|██████▏                            | 24928/140000 [00:45<03:22, 568.93it/s]2019-11-15 21:23:30.440975: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 18%|██████▎                            | 25451/140000 [00:45<03:19, 575.21it/s]2019-11-15 21:23:31.441478: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 19%|██████▌                            | 26028/140000 [00:46<03:24, 557.22it/s]2019-11-15 21:23:32.442140: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 19%|██████▋                            | 26602/140000 [00:47<03:20, 566.49it/s]2019-11-15 21:23:33.442403: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 19%|██████▊                            | 27161/140000 [00:49<03:25, 549.65it/s]2019-11-15 21:23:34.442932: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 20%|██████▉                            | 27722/140000 [00:50<03:20, 559.35it/s]2019-11-15 21:23:35.442984: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 20%|███████                            | 28290/140000 [00:51<03:21, 555.60it/s]2019-11-15 21:23:36.444616: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 21%|███████▏                           | 28796/140000 [00:51<03:25, 541.02it/s]2019-11-15 21:23:37.446262: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 21%|███████▎                           | 29345/140000 [00:52<03:23, 544.66it/s]2019-11-15 21:23:38.447678: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 21%|███████▍                           | 29910/140000 [00:53<03:16, 560.11it/s]2019-11-15 21:23:39.448617: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 22%|███████▌                           | 30481/140000 [00:55<03:13, 566.48it/s]2019-11-15 21:23:40.449521: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 22%|███████▊                           | 31065/140000 [00:56<03:08, 576.64it/s]2019-11-15 21:23:41.450449: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 23%|███████▉                           | 31639/140000 [00:57<03:10, 568.94it/s]2019-11-15 21:23:42.450681: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 23%|████████                           | 32160/140000 [00:57<03:12, 559.96it/s]2019-11-15 21:23:43.451323: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 23%|████████▏                          | 32723/140000 [00:58<03:12, 558.39it/s]2019-11-15 21:23:44.452738: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 24%|████████▎                          | 33290/140000 [01:00<03:16, 542.35it/s]2019-11-15 21:23:45.453859: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 24%|████████▍                          | 33858/140000 [01:01<03:06, 567.73it/s]2019-11-15 21:23:46.455534: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 25%|████████▌                          | 34433/140000 [01:02<03:05, 569.50it/s]2019-11-15 21:23:47.456000: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 25%|████████▋                          | 34945/140000 [01:02<03:07, 559.45it/s]2019-11-15 21:23:48.457205: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 25%|████████▉                          | 35517/140000 [01:03<03:03, 569.42it/s]2019-11-15 21:23:49.457737: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 26%|█████████                          | 36096/140000 [01:04<03:03, 566.19it/s]2019-11-15 21:23:50.458097: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 26%|█████████▏                         | 36675/140000 [01:06<02:59, 575.43it/s]2019-11-15 21:23:51.458209: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 27%|█████████▎                         | 37251/140000 [01:07<03:01, 567.42it/s]2019-11-15 21:23:52.459639: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 27%|█████████▍                         | 37829/140000 [01:08<02:59, 569.91it/s]2019-11-15 21:23:53.459786: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 27%|█████████▌                         | 38405/140000 [01:09<02:55, 579.20it/s]2019-11-15 21:23:54.461166: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 28%|█████████▋                         | 38926/140000 [01:09<03:00, 559.30it/s]2019-11-15 21:23:55.462402: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 28%|█████████▊                         | 39493/140000 [01:10<02:56, 568.70it/s]2019-11-15 21:23:56.462593: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 29%|██████████                         | 40068/140000 [01:12<02:55, 568.92it/s]2019-11-15 21:23:57.462674: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 29%|██████████▏                        | 40637/140000 [01:13<02:55, 566.14it/s]2019-11-15 21:23:58.463148: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 29%|██████████▎                        | 41217/140000 [01:14<02:54, 566.47it/s]2019-11-15 21:23:59.464597: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 30%|██████████▍                        | 41795/140000 [01:15<02:51, 572.38it/s]2019-11-15 21:24:00.466111: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 30%|██████████▌                        | 42311/140000 [01:15<02:52, 567.07it/s]2019-11-15 21:24:01.466325: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 31%|██████████▋                        | 42878/140000 [01:16<02:51, 567.76it/s]2019-11-15 21:24:02.467008: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 31%|██████████▊                        | 43441/140000 [01:18<02:53, 556.93it/s]2019-11-15 21:24:03.467286: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 31%|███████████                        | 44011/140000 [01:19<02:50, 561.59it/s]2019-11-15 21:24:04.467388: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 32%|███████████▏                       | 44592/140000 [01:20<02:49, 562.24it/s]2019-11-15 21:24:05.468734: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 32%|███████████▎                       | 45161/140000 [01:21<02:48, 564.29it/s]2019-11-15 21:24:06.470134: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 33%|███████████▍                       | 45677/140000 [01:21<02:49, 556.10it/s]2019-11-15 21:24:07.470365: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 33%|███████████▌                       | 46253/140000 [01:23<02:48, 556.62it/s]2019-11-15 21:24:08.471004: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 33%|███████████▋                       | 46817/140000 [01:24<02:48, 552.91it/s]2019-11-15 21:24:09.471821: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 34%|███████████▊                       | 47379/140000 [01:25<02:47, 553.49it/s]2019-11-15 21:24:10.473314: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 34%|███████████▉                       | 47947/140000 [01:26<02:45, 556.55it/s]2019-11-15 21:24:11.474183: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 35%|████████████▏                      | 48519/140000 [01:27<02:42, 561.67it/s]2019-11-15 21:24:12.475979: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 35%|████████████▎                      | 49037/140000 [01:27<02:44, 554.11it/s]2019-11-15 21:24:13.476647: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 35%|████████████▍                      | 49606/140000 [01:29<02:40, 564.72it/s]2019-11-15 21:24:14.477312: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 36%|████████████▌                      | 50181/140000 [01:30<02:39, 562.10it/s]2019-11-15 21:24:15.478499: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 36%|████████████▋                      | 50749/140000 [01:31<02:40, 554.63it/s]2019-11-15 21:24:16.479132: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 37%|████████████▊                      | 51319/140000 [01:32<02:41, 549.74it/s]2019-11-15 21:24:17.480566: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 37%|████████████▉                      | 51822/140000 [01:32<02:40, 548.19it/s]2019-11-15 21:24:18.481673: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 37%|█████████████                      | 52388/140000 [01:34<02:37, 554.82it/s]2019-11-15 21:24:19.482253: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 38%|█████████████▏                     | 52957/140000 [01:35<02:38, 549.38it/s]2019-11-15 21:24:20.482468: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 38%|█████████████▍                     | 53510/140000 [01:36<02:37, 548.72it/s]2019-11-15 21:24:21.482918: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 39%|█████████████▌                     | 54073/140000 [01:37<02:34, 555.59it/s]2019-11-15 21:24:22.482981: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 39%|█████████████▋                     | 54640/140000 [01:38<02:33, 554.47it/s]2019-11-15 21:24:23.484705: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 39%|█████████████▊                     | 55211/140000 [01:39<02:29, 566.19it/s]2019-11-15 21:24:24.485154: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 40%|█████████████▉                     | 55724/140000 [01:40<02:32, 552.60it/s]2019-11-15 21:24:25.485163: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 40%|██████████████                     | 56292/140000 [01:41<02:29, 559.71it/s]2019-11-15 21:24:26.485857: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 41%|██████████████▏                    | 56858/140000 [01:42<02:31, 550.10it/s]2019-11-15 21:24:27.486805: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 41%|██████████████▎                    | 57433/140000 [01:43<02:25, 568.42it/s]2019-11-15 21:24:28.487779: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 41%|██████████████▌                    | 58009/140000 [01:44<02:25, 563.34it/s]2019-11-15 21:24:29.488630: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 42%|██████████████▋                    | 58522/140000 [01:44<02:27, 553.52it/s]2019-11-15 21:24:30.489527: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 42%|██████████████▊                    | 59102/140000 [01:46<02:22, 568.54it/s]2019-11-15 21:24:31.490969: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 43%|██████████████▉                    | 59672/140000 [01:47<02:23, 559.96it/s]2019-11-15 21:24:32.492523: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 43%|███████████████                    | 60244/140000 [01:48<02:20, 569.64it/s]2019-11-15 21:24:33.493333: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 43%|███████████████▏                   | 60820/140000 [01:49<02:18, 571.22it/s]2019-11-15 21:24:34.493482: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 44%|███████████████▎                   | 61400/140000 [01:50<02:16, 574.35it/s]2019-11-15 21:24:35.493939: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 44%|███████████████▍                   | 61917/140000 [01:51<02:19, 560.25it/s]2019-11-15 21:24:36.495280: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 45%|███████████████▌                   | 62485/140000 [01:52<02:20, 550.63it/s]2019-11-15 21:24:37.496601: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 45%|███████████████▊                   | 63055/140000 [01:53<02:16, 565.53it/s]2019-11-15 21:24:38.497985: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 45%|███████████████▉                   | 63623/140000 [01:54<02:15, 563.70it/s]2019-11-15 21:24:39.499027: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 46%|████████████████                   | 64196/140000 [01:55<02:13, 567.44it/s]2019-11-15 21:24:40.500227: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 46%|████████████████▏                  | 64770/140000 [01:56<02:12, 566.58it/s]2019-11-15 21:24:41.501393: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 47%|████████████████▎                  | 65339/140000 [01:57<02:13, 560.83it/s]2019-11-15 21:24:42.502289: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 47%|████████████████▍                  | 65851/140000 [01:58<02:10, 568.79it/s]2019-11-15 21:24:43.502399: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 47%|████████████████▌                  | 66425/140000 [01:59<02:10, 565.80it/s]2019-11-15 21:24:44.503470: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 48%|████████████████▊                  | 67007/140000 [02:00<02:06, 576.70it/s]2019-11-15 21:24:45.504014: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 48%|████████████████▉                  | 67585/140000 [02:01<02:08, 565.13it/s]2019-11-15 21:24:46.505659: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 49%|█████████████████                  | 68152/140000 [02:02<02:10, 550.19it/s]2019-11-15 21:24:47.505868: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 49%|█████████████████▏                 | 68728/140000 [02:03<02:04, 571.51it/s]2019-11-15 21:24:48.506704: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 50%|█████████████████▎                 | 69318/140000 [02:04<02:02, 578.95it/s]2019-11-15 21:24:49.507740: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 50%|█████████████████▍                 | 69843/140000 [02:05<02:01, 577.53it/s]2019-11-15 21:24:50.508614: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 50%|█████████████████▌                 | 70421/140000 [02:06<02:02, 565.70it/s]2019-11-15 21:24:51.509813: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 51%|█████████████████▋                 | 70998/140000 [02:07<02:02, 561.68it/s]2019-11-15 21:24:52.510694: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 51%|█████████████████▉                 | 71571/140000 [02:08<02:01, 562.30it/s]2019-11-15 21:24:53.511473: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 52%|██████████████████                 | 72146/140000 [02:09<01:58, 572.32it/s]2019-11-15 21:24:54.512373: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 52%|██████████████████▏                | 72723/140000 [02:10<01:58, 569.52it/s]2019-11-15 21:24:55.513931: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 52%|██████████████████▎                | 73305/140000 [02:11<01:55, 579.25it/s]2019-11-15 21:24:56.515361: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 53%|██████████████████▍                | 73834/140000 [02:12<01:55, 571.85it/s]2019-11-15 21:24:57.516873: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 53%|██████████████████▌                | 74417/140000 [02:13<01:53, 577.83it/s]2019-11-15 21:24:58.517308: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 54%|██████████████████▋                | 74988/140000 [02:14<01:57, 553.21it/s]2019-11-15 21:24:59.517347: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 54%|██████████████████▉                | 75564/140000 [02:15<01:55, 557.14it/s]2019-11-15 21:25:00.518431: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 54%|███████████████████                | 76077/140000 [02:16<02:00, 532.52it/s]2019-11-15 21:25:01.519687: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 55%|███████████████████▏               | 76643/140000 [02:17<01:53, 556.11it/s]2019-11-15 21:25:02.519857: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 55%|███████████████████▎               | 77222/140000 [02:18<01:50, 569.27it/s]2019-11-15 21:25:03.520186: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 56%|███████████████████▍               | 77791/140000 [02:19<01:52, 553.77it/s]2019-11-15 21:25:04.521580: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 56%|███████████████████▌               | 78373/140000 [02:20<01:49, 563.42it/s]2019-11-15 21:25:05.522068: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 56%|███████████████████▋               | 78944/140000 [02:21<01:49, 556.61it/s]2019-11-15 21:25:06.522355: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 57%|███████████████████▊               | 79459/140000 [02:22<01:46, 566.26it/s]2019-11-15 21:25:07.523949: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 57%|████████████████████               | 80031/140000 [02:23<01:46, 565.02it/s]2019-11-15 21:25:08.525210: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 58%|████████████████████▏              | 80608/140000 [02:24<01:45, 560.66it/s]2019-11-15 21:25:09.526814: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 58%|████████████████████▎              | 81191/140000 [02:25<01:41, 580.87it/s]2019-11-15 21:25:10.527223: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 58%|████████████████████▍              | 81778/140000 [02:26<01:42, 570.42it/s]2019-11-15 21:25:11.527874: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 59%|████████████████████▌              | 82300/140000 [02:27<01:42, 562.05it/s]2019-11-15 21:25:12.528110: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 59%|████████████████████▋              | 82870/140000 [02:28<01:42, 557.59it/s]2019-11-15 21:25:13.528273: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 60%|████████████████████▊              | 83446/140000 [02:29<01:37, 578.65it/s]2019-11-15 21:25:14.529087: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 60%|█████████████████████              | 84016/140000 [02:30<01:41, 551.88it/s]2019-11-15 21:25:15.530624: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 60%|█████████████████████▏             | 84598/140000 [02:31<01:36, 576.67it/s]2019-11-15 21:25:16.531452: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 61%|█████████████████████▎             | 85182/140000 [02:32<01:35, 576.47it/s]2019-11-15 21:25:17.532931: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 61%|█████████████████████▍             | 85705/140000 [02:33<01:34, 573.81it/s]2019-11-15 21:25:18.534376: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 62%|█████████████████████▌             | 86282/140000 [02:34<01:34, 566.41it/s]2019-11-15 21:25:19.535231: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 62%|█████████████████████▋             | 86858/140000 [02:35<01:33, 570.46it/s]2019-11-15 21:25:20.536650: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 62%|█████████████████████▊             | 87440/140000 [02:36<01:33, 563.56it/s]2019-11-15 21:25:21.537827: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 63%|██████████████████████             | 88003/140000 [02:37<01:34, 548.30it/s]2019-11-15 21:25:22.539809: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 63%|██████████████████████▏            | 88570/140000 [02:38<01:31, 559.49it/s]2019-11-15 21:25:23.540222: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 64%|██████████████████████▎            | 89089/140000 [02:39<01:30, 561.68it/s]2019-11-15 21:25:24.540762: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 64%|██████████████████████▍            | 89665/140000 [02:40<01:27, 573.08it/s]2019-11-15 21:25:25.541820: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 64%|██████████████████████▌            | 90245/140000 [02:41<01:25, 578.96it/s]2019-11-15 21:25:26.542935: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 65%|██████████████████████▋            | 90824/140000 [02:42<01:25, 572.92it/s]2019-11-15 21:25:27.544802: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 65%|██████████████████████▊            | 91396/140000 [02:43<01:26, 563.84it/s]2019-11-15 21:25:28.545491: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 66%|██████████████████████▉            | 91976/140000 [02:44<01:23, 572.49it/s]2019-11-15 21:25:29.545602: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 66%|███████████████████████▏           | 92557/140000 [02:45<01:21, 579.34it/s]2019-11-15 21:25:30.545657: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 66%|███████████████████████▎           | 93075/140000 [02:46<01:23, 565.01it/s]2019-11-15 21:25:31.546607: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 67%|███████████████████████▍           | 93658/140000 [02:47<01:21, 568.08it/s]2019-11-15 21:25:32.547982: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 67%|███████████████████████▌           | 94232/140000 [02:48<01:22, 555.82it/s]2019-11-15 21:25:33.548527: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 68%|███████████████████████▋           | 94808/140000 [02:49<01:18, 577.82it/s]2019-11-15 21:25:34.550103: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 68%|███████████████████████▊           | 95393/140000 [02:50<01:17, 573.47it/s]2019-11-15 21:25:35.550950: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 69%|███████████████████████▉           | 95967/140000 [02:51<01:17, 565.52it/s]2019-11-15 21:25:36.551335: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 69%|████████████████████████           | 96482/140000 [02:52<01:18, 554.00it/s]2019-11-15 21:25:37.552467: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 69%|████████████████████████▎          | 97054/140000 [02:53<01:15, 569.66it/s]2019-11-15 21:25:38.552624: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 70%|████████████████████████▍          | 97634/140000 [02:54<01:14, 569.75it/s]2019-11-15 21:25:39.553268: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 70%|████████████████████████▌          | 98218/140000 [02:55<01:11, 581.58it/s]2019-11-15 21:25:40.554723: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 71%|████████████████████████▋          | 98800/140000 [02:56<01:12, 567.39it/s]2019-11-15 21:25:41.555250: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 71%|████████████████████████▊          | 99308/140000 [02:57<01:14, 547.43it/s]2019-11-15 21:25:42.555972: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 71%|████████████████████████▉          | 99864/140000 [02:58<01:12, 551.27it/s]2019-11-15 21:25:43.556095: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 72%|████████████████████████▍         | 100437/140000 [02:59<01:09, 567.23it/s]2019-11-15 21:25:44.557094: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 72%|████████████████████████▌         | 101016/140000 [03:00<01:09, 560.91it/s]2019-11-15 21:25:45.557530: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 73%|████████████████████████▋         | 101586/140000 [03:01<01:07, 565.26it/s]2019-11-15 21:25:46.558068: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 73%|████████████████████████▊         | 102161/140000 [03:02<01:06, 567.52it/s]2019-11-15 21:25:47.558620: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 73%|████████████████████████▉         | 102680/140000 [03:03<01:05, 565.75it/s]2019-11-15 21:25:48.559216: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 74%|█████████████████████████         | 103252/140000 [03:04<01:05, 565.13it/s]2019-11-15 21:25:49.560481: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 74%|█████████████████████████▏        | 103831/140000 [03:05<01:03, 568.97it/s]2019-11-15 21:25:50.560832: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 75%|█████████████████████████▎        | 104412/140000 [03:06<01:01, 574.77it/s]2019-11-15 21:25:51.561345: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 75%|█████████████████████████▍        | 104984/140000 [03:07<01:02, 559.81it/s]2019-11-15 21:25:52.562692: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 75%|█████████████████████████▋        | 105554/140000 [03:08<01:01, 560.47it/s]2019-11-15 21:25:53.562873: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 76%|█████████████████████████▊        | 106073/140000 [03:09<00:59, 570.51it/s]2019-11-15 21:25:54.564574: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 76%|█████████████████████████▉        | 106650/140000 [03:10<00:58, 565.60it/s]2019-11-15 21:25:55.565924: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 77%|██████████████████████████        | 107215/140000 [03:11<00:58, 561.35it/s]2019-11-15 21:25:56.566692: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 77%|██████████████████████████▏       | 107796/140000 [03:12<00:56, 566.41it/s]2019-11-15 21:25:57.567333: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 77%|██████████████████████████▎       | 108370/140000 [03:13<00:55, 574.84it/s]2019-11-15 21:25:58.568937: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 78%|██████████████████████████▍       | 108957/140000 [03:14<00:54, 573.33it/s]2019-11-15 21:25:59.569779: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 78%|██████████████████████████▌       | 109483/140000 [03:15<00:53, 568.98it/s]2019-11-15 21:26:00.570964: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 79%|██████████████████████████▋       | 110076/140000 [03:16<00:52, 569.33it/s]2019-11-15 21:26:01.571255: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 79%|██████████████████████████▊       | 110647/140000 [03:17<00:52, 563.05it/s]2019-11-15 21:26:02.572743: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 79%|███████████████████████████       | 111217/140000 [03:18<00:50, 565.42it/s]2019-11-15 21:26:03.573132: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 80%|███████████████████████████▏      | 111799/140000 [03:19<00:49, 573.68it/s]2019-11-15 21:26:04.573527: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 80%|███████████████████████████▎      | 112379/140000 [03:20<00:48, 572.62it/s]2019-11-15 21:26:05.574429: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 81%|███████████████████████████▍      | 112905/140000 [03:21<00:47, 574.62it/s]2019-11-15 21:26:06.575639: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 81%|███████████████████████████▌      | 113484/140000 [03:22<00:46, 570.64it/s]2019-11-15 21:26:07.576441: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 81%|███████████████████████████▋      | 114064/140000 [03:23<00:45, 571.05it/s]2019-11-15 21:26:08.576589: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 82%|███████████████████████████▊      | 114646/140000 [03:24<00:44, 569.47it/s]2019-11-15 21:26:09.577608: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 82%|███████████████████████████▉      | 115218/140000 [03:25<00:43, 564.12it/s]2019-11-15 21:26:10.577986: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 83%|████████████████████████████      | 115795/140000 [03:26<00:42, 568.57it/s]2019-11-15 21:26:11.579779: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 83%|████████████████████████████▏     | 116309/140000 [03:27<00:42, 563.30it/s]2019-11-15 21:26:12.580379: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 83%|████████████████████████████▍     | 116884/140000 [03:28<00:41, 563.69it/s]2019-11-15 21:26:13.582483: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 84%|████████████████████████████▌     | 117458/140000 [03:29<00:39, 571.40it/s]2019-11-15 21:26:14.582999: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 84%|████████████████████████████▋     | 118023/140000 [03:30<00:39, 559.07it/s]2019-11-15 21:26:15.583724: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 85%|████████████████████████████▊     | 118600/140000 [03:31<00:37, 571.01it/s]2019-11-15 21:26:16.584486: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 85%|████████████████████████████▉     | 119177/140000 [03:32<00:36, 562.87it/s]2019-11-15 21:26:17.585259: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 85%|█████████████████████████████     | 119691/140000 [03:33<00:36, 562.70it/s]2019-11-15 21:26:18.585598: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 86%|█████████████████████████████▏    | 120261/140000 [03:34<00:34, 565.34it/s]2019-11-15 21:26:19.587196: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 86%|█████████████████████████████▎    | 120846/140000 [03:35<00:33, 573.00it/s]2019-11-15 21:26:20.588279: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 87%|█████████████████████████████▍    | 121418/140000 [03:36<00:33, 560.71it/s]2019-11-15 21:26:21.588328: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 87%|█████████████████████████████▋    | 121989/140000 [03:37<00:32, 546.38it/s]2019-11-15 21:26:22.589728: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 88%|█████████████████████████████▊    | 122502/140000 [03:38<00:31, 564.32it/s]2019-11-15 21:26:23.590566: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 88%|█████████████████████████████▉    | 123069/140000 [03:39<00:30, 560.40it/s]2019-11-15 21:26:24.591396: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 88%|██████████████████████████████    | 123648/140000 [03:40<00:28, 565.13it/s]2019-11-15 21:26:25.592811: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 89%|██████████████████████████████▏   | 124229/140000 [03:41<00:27, 570.49it/s]2019-11-15 21:26:26.594496: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 89%|██████████████████████████████▎   | 124810/140000 [03:42<00:26, 574.24it/s]2019-11-15 21:26:27.596985: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 90%|██████████████████████████████▍   | 125380/140000 [03:43<00:26, 558.82it/s]2019-11-15 21:26:28.597496: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 90%|██████████████████████████████▌   | 125957/140000 [03:44<00:24, 578.05it/s]2019-11-15 21:26:29.598843: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 90%|██████████████████████████████▋   | 126473/140000 [03:45<00:24, 553.20it/s]2019-11-15 21:26:30.599558: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 91%|██████████████████████████████▊   | 127046/140000 [03:46<00:22, 567.86it/s]2019-11-15 21:26:31.600886: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 91%|██████████████████████████████▉   | 127628/140000 [03:47<00:21, 579.34it/s]2019-11-15 21:26:32.601898: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 92%|███████████████████████████████▏  | 128212/140000 [03:48<00:20, 571.86it/s]2019-11-15 21:26:33.602137: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 92%|███████████████████████████████▎  | 128790/140000 [03:49<00:19, 570.82it/s]2019-11-15 21:26:34.603720: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 92%|███████████████████████████████▍  | 129371/140000 [03:50<00:18, 567.48it/s]2019-11-15 21:26:35.604125: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 93%|███████████████████████████████▌  | 129950/140000 [03:51<00:17, 575.65it/s]2019-11-15 21:26:36.605282: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 93%|███████████████████████████████▋  | 130467/140000 [03:52<00:16, 570.65it/s]2019-11-15 21:26:37.606487: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 94%|███████████████████████████████▊  | 131042/140000 [03:53<00:15, 572.96it/s]2019-11-15 21:26:38.606989: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 94%|███████████████████████████████▉  | 131620/140000 [03:54<00:14, 573.72it/s]2019-11-15 21:26:39.607628: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 94%|████████████████████████████████  | 132203/140000 [03:55<00:13, 574.05it/s]2019-11-15 21:26:40.607648: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 95%|████████████████████████████████▏ | 132780/140000 [03:56<00:12, 566.04it/s]2019-11-15 21:26:41.608181: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 95%|████████████████████████████████▍ | 133360/140000 [03:57<00:11, 563.39it/s]2019-11-15 21:26:42.609516: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 96%|████████████████████████████████▌ | 133872/140000 [03:58<00:10, 566.65it/s]2019-11-15 21:26:43.611266: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 96%|████████████████████████████████▋ | 134452/140000 [03:59<00:09, 574.10it/s]2019-11-15 21:26:44.612611: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 96%|████████████████████████████████▊ | 135033/140000 [04:00<00:09, 550.98it/s]2019-11-15 21:26:45.612685: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 97%|████████████████████████████████▉ | 135609/140000 [04:01<00:07, 569.08it/s]2019-11-15 21:26:46.613651: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 97%|█████████████████████████████████ | 136185/140000 [04:02<00:06, 565.09it/s]2019-11-15 21:26:47.615523: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 98%|█████████████████████████████████▏| 136754/140000 [04:03<00:05, 558.30it/s]2019-11-15 21:26:48.616579: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 98%|█████████████████████████████████▎| 137271/140000 [04:04<00:04, 558.53it/s]2019-11-15 21:26:49.616622: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 98%|█████████████████████████████████▍| 137846/140000 [04:05<00:03, 571.46it/s]2019-11-15 21:26:50.618148: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 99%|█████████████████████████████████▌| 138417/140000 [04:06<00:02, 556.01it/s]2019-11-15 21:26:51.619354: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      " 99%|█████████████████████████████████▊| 138991/140000 [04:07<00:01, 561.18it/s]2019-11-15 21:26:52.621052: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "100%|█████████████████████████████████▉| 139561/140000 [04:08<00:00, 554.56it/s]2019-11-15 21:26:53.621948: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "100%|██████████████████████████████████| 140000/140000 [04:08<00:00, 562.29it/s]\n",
      "\u001b[32mSaved FeatureData[Differential] to: songbird-differentials.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[SongbirdStats] to: songbird-regression-stats.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults % Properties('biplot') to: songbird-regression-biplot.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime songbird multinomial \\\n",
    "    --i-table table.qza \\\n",
    "    --m-metadata-file $INPUT_SAMPLE_METADATA_PATH \\\n",
    "    --p-formula \"C(sample_type_body_site, Treatment('sea water'))\" \\\n",
    "    --p-epochs 10000 \\\n",
    "    --p-learning-rate 0.0001 \\\n",
    "    --p-num-random-test-examples 40 \\\n",
    "    --p-batch-size 40 \\\n",
    "    --p-summary-interval 1 \\\n",
    "    --verbose \\\n",
    "    --o-differentials songbird-differentials.qza \\\n",
    "    --o-regression-stats songbird-regression-stats.qza \\\n",
    "    --o-regression-biplot songbird-regression-biplot.qza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Visualize Songbird model fitting statistics\n",
    "For more information about how to interpret this visualization, check out [this section of the Songbird README](https://github.com/biocore/songbird#interpreting-model-fitting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: songbird-regression-summary.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime songbird summarize-single \\\n",
    "    --i-regression-stats songbird-regression-stats.qza \\\n",
    "    --o-visualization songbird-regression-summary.qzv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Run Qurro!\n",
    "We're doing this using a currently-in-development version of Qurro.\n",
    "\n",
    "Note that the version of the \"mackerel demo\" up on [Qurro's website](https://biocore.github.io/qurro) will be updated as Qurro itself is updated (so although the underlying data should remain the same, the visualization interface might look a bit different/contain a few more features in a few months). To make a long story short, this is because this dataset (the filtered BIOM table, the Songbird differentials produced above, the input sample metadata, and the taxonomic classifications produced above) is actually *included in Qurro's test suite*: so every time Qurro's tests are run, we also rerun Qurro on this dataset and check that a visualization is produced properly. This also automatically updates the mackerel demo on Qurro's website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22268 feature(s) in the BIOM table were not present in the feature rankings.\n",
      "These feature(s) have been removed from the visualization.\n",
      "1248 sample(s) in the sample metadata file were not present in the BIOM table.\n",
      "These sample(s) have been removed from the visualization.\n",
      "\u001b[32mSaved Visualization to: qurro-plot.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime qurro differential-plot \\\n",
    "    --i-table table.qza \\\n",
    "    --i-ranks songbird-differentials.qza \\\n",
    "    --m-sample-metadata-file $INPUT_SAMPLE_METADATA_PATH \\\n",
    "    --m-feature-metadata-file taxonomy.qza \\\n",
    "    --verbose \\\n",
    "    --o-visualization qurro-plot.qzv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}