dog-breed-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Conda/Miniconda\n",
    "[Download](https://conda.io/miniconda.html) and install Miniconda. Select the Python 3.7 version or later. Don't select the Python 2.x version.\n",
    "\n",
    "### AzureML Python SDK\n",
    "Install the Python SDK:  make sure to install notebook, and contrib\n",
    "```\n",
    "conda create -n azureml -y Python=3.6 ipywidgets nb_conda\n",
    "conda activate azureml\n",
    "pip install --upgrade azureml-sdk[notebooks,contrib] scikit-image tensorflow tensorboardX --user \n",
    "jupyter nbextension install --py --user azureml.widgets\n",
    "jupyter nbextension enable azureml.widgets --user --py\n",
    "```\n",
    "\n",
    "Install PyTorch:\n",
    "\n",
    "On MacOS: \n",
    "\n",
    "    conda install pytorch torchvision -c pytorch\n",
    "\n",
    "On Windows\n",
    "\n",
    "    conda install pytorch -c pytorch\n",
    "    pip install torchvision\n",
    "\n",
    "You will need to restart jupyter after this\n",
    "Detailed instructions are here: https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python \n",
    "\n",
    "### (Optional) Install VS Code and the VS Code extension \n",
    "[Download](https://code.visualstudio.com/) and install Visual Studio Code then the [Azure Machine Learning Extension](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.vscode-ai)\n",
    "Make sure it has a recent version of the Python SDK -- remove the folder ~/.azureml/envs if there are issuse. A current SDK will be installed when you first use AML from VSCode.\n",
    "\n",
    "### Clone this repository\n",
    "```\n",
    "git clone https://github.com/danielsc/dogbreeds\n",
    "jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Check core SDK version number\n",
    "import azureml.core\n",
    "\n",
    "print(\"SDK version:\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Dog breed classification using Pytorch Estimators on Azure Machine Learning service\n",
    "\n",
    "Have you ever seen a dog and not been able to tell the breed? Some dogs look so similar, that it can be nearly impossible to tell. For instance these are a few breeds that are difficult to tell apart:\n",
    "\n",
    "#### Alaskan Malamutes vs Siberian Huskies\n",
    "![Image of Alaskan Malamute vs Siberian Husky](http://cdn.akc.org/content/article-body-image/malamutehusky.jpg)\n",
    "\n",
    "#### Whippet vs Italian Greyhound \n",
    "![Image of Whippet vs Italian Greyhound](http://cdn.akc.org/content/article-body-image/whippetitalian.jpg)\n",
    "\n",
    "There are sites like http://what-dog.net, which use Microsoft Cognitive Services to be able to make this easier. \n",
    "\n",
    "In this tutorial, you will learn how to train a Pytorch image classification model using transfer learning with the Azure Machine Learning service. The Azure Machine Learning python SDK's [PyTorch estimator](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-train-pytorch) enables you to easily submit PyTorch training jobs for both single-node and distributed runs on Azure compute. The model is trained to classify dog breeds using the [Stanford Dog dataset](http://vision.stanford.edu/aditya86/ImageNetDogs/) and it is based on a pretrained ResNet18 model. This ResNet18 model has been built using images and annotation from ImageNet. The Stanford Dog dataset contains 120 classes (i.e. dog breeds), to save time however, for most of the tutorial, we will only use a subset of this dataset which includes only 10 dog breeds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Azure Machine Learning service?\n",
    "Azure Machine Learning service is a cloud service that you can use to develop and deploy machine learning models. Using Azure Machine Learning service, you can track your models as you build, train, deploy, and manage them, all at the broad scale that the cloud provides.\n",
    "![](aml-overview.png)\n",
    "\n",
    "\n",
    "## How can we use it for training image classification models?\n",
    "Training machine learning models, particularly deep neural networks, is often a time- and compute-intensive task. Once you've finished writing your training script and running on a small subset of data on your local machine, you will likely want to scale up your workload.\n",
    "\n",
    "To facilitate training, the Azure Machine Learning Python SDK provides a high-level abstraction, the estimator class, which allows users to easily train their models in the Azure ecosystem. You can create and use an Estimator object to submit any training code you want to run on remote compute, whether it's a single-node run or distributed training across a GPU cluster. For PyTorch and TensorFlow jobs, Azure Machine Learning also provides respective custom PyTorch and TensorFlow estimators to simplify using these frameworks.\n",
    "\n",
    "### Steps to train with a Pytorch Estimator:\n",
    "In this tutorial, we will:\n",
    "- Connect to an Azure Machine Learning service Workspace \n",
    "- Create a remote compute target\n",
    "- Upload your training data (Optional)\n",
    "- Create your training script\n",
    "- Create an Estimator object\n",
    "- Submit your training job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create workspace\n",
    "\n",
    "In the next step, you will create your own Workspace to use in this tutorial.\n",
    "\n",
    "**You will be asked to login during this step. Please use your Microsoft AAD credentials.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "Make sure you have access to an Azure subscription. Your group's admin should have added you to your team's subscription. The subscriptions are listed [here](https://microsoft-my.sharepoint.com/:x:/p/suhon/EURF8-YwP1lFmG2aYKemomgBX8NqTN1Qhr9fzHobhN2CvA?e=qQa7UL) (MS FTE credentials required) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## need to update the following 2 values based on the sub we are using\n",
    "subscription_id=                           # <<please retrieve from URL at above paragraph>>\n",
    "container_registry_name=                   # <<please retrieve from URL at above paragraph>>\n",
    "workspace_name=                            # <<a Unique Name -- use your alias>>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "location='westeurope'\n",
    "resource_group='AMLworkshop'\n",
    "container_registry=f'/subscriptions/{subscription_id}/resourcegroups/{resource_group}/providers/microsoft.containerregistry/registries/{container_registry_name}'\n",
    "\n",
    "ws = Workspace.create(name = workspace_name,\n",
    "                      subscription_id = subscription_id,\n",
    "                      resource_group = resource_group, \n",
    "                      location = location,\n",
    "                      container_registry = container_registry,\n",
    "                      exist_ok = True)\n",
    "\n",
    "ws.write_config()\n",
    "\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Subscription id: ' + ws.subscription_id, \n",
    "      'Resource group: ' + ws.resource_group, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will take a few minutes, so let's talk about what a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) is while it is being created.\n",
    "![](aml-workspace.png)\n",
    "\n",
    "\n",
    "## Create a remote compute target\n",
    "For this tutorial, we will create an AML Compute cluster with a NC6s_v2, P100 GPU machines, created to use as the [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) to execute your training script on. \n",
    "\n",
    "**Creation of the cluster takes approximately 5 minutes, but we will not wait for it to complete** \n",
    "\n",
    "If the cluster is already in your workspace this code will skip the cluster creation process. Note that the code is not waiting for completion of the cluster creation. If needed you can call `compute_target.wait_for_completion(show_output=True)`, which will block you notebook until the compute target is provisioned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
    "\n",
    "# choose a name for your cluster\n",
    "cluster_name = \"p100cluster\"\n",
    "\n",
    "try:\n",
    "    compute_target = ws.compute_targets[cluster_name]\n",
    "    print('Found existing compute target.')\n",
    "except KeyError:\n",
    "    print('Creating a new compute target...')\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_NC6s_v2', \n",
    "                                                           idle_seconds_before_scaledown=1800,\n",
    "                                                           min_nodes=0, \n",
    "                                                           max_nodes=4)\n",
    "\n",
    "    # create the cluster\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "    compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can ask for the provisioning state to see whether the cluster is up already of if there were errors\n",
    "compute_target.status.serialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attach the blobstore with the training data to the workspace\n",
    "While the cluster is still creating, let's attach some data to our workspace.\n",
    "\n",
    "The dataset we will use consists of ~150 images per class. Some breeds have more, while others have less. Each class has about 100 training images each for dog breeds, with ~50 validation images for each class. We will look at 10 classes in this tutorial.\n",
    "\n",
    "To make the data accessible for remote training, you will need to keep the data in the cloud. AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data, and interact with it from your remote compute targets. It is an abstraction over Azure Storage. The datastore can reference either an Azure Blob container or Azure file share as the underlying storage. \n",
    "\n",
    "You can view the subset of the data used [here](https://github.com/heatherbshapiro/pycon-canada/tree/master/breeds-10). Or download it from [here](https://github.com/heatherbshapiro/pycon-canada/master/breeds-10.zip) as a zip file. \n",
    "\n",
    "We already copied the data to an Azure blob storage container. To attach this blob container as a data store to your workspace, you use the `Datastore.register_azure_blob_container` function. You can copy the statement with the secrets filled in from [here](https://microsoft.sharepoint.com/teams/azuremlnursery/_layouts/OneNote.aspx?id=%2Fteams%2Fazuremlnursery%2FSiteAssets%2FAzure%20ML%20Nursery%20Notebook&wd=target%28Workshop.one%7C265D85D5-44C8-9D40-B556-A31FA098E708%2FDogbreeds%7C62F09F92-105C-7849-AF84-905BEE9F9588%2F%29) (requires Microsoft Employee login).\n",
    "\n",
    "**If you already have the breeds datstore attached you can skip the next cell**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "ds = Datastore.register_azure_blob_container(workspace=ws, \n",
    "                                             datastore_name='breeds', \n",
    "                                             container_name=<<add container name>>,\n",
    "                                             account_name=<<add account name name>>, \n",
    "                                             account_key=<<add account key ending with ==>>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get a reference to the path on the datastore with the training data. We can do so using the `path` method. In the next section, we can then pass this reference to our training script's `--data_dir` argument. We will start with the 10 classes dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "# working around bug https://msdata.visualstudio.com/Vienna/_workitems/edit/324147\n",
    "# istead of \n",
    "# ds = ws.datastores[\"breeds\"]\n",
    "# need to create the datastore object like this\n",
    "ds = Datastore(ws,'breeds')\n",
    "path_on_datastore = 'breeds-10'\n",
    "ds_data = ds.path(path_on_datastore)\n",
    "print(ds_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the Data\n",
    "\n",
    "If you are interested in downloading the data locally, you can run `ds.download(\".\", 'breeds-10')`. This might take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds.download('.', 'breeds-10', show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare training script\n",
    "Now you will need to create your training script. In this tutorial, the training script is already provided for you at `pytorch_train.py`. In practice, you should be able to take any custom training script as is and run it with AML without having to modify your code.\n",
    "\n",
    "However, if you would like to use AML's [tracking and metrics](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#metrics) capabilities, you will have to add a small amount of AML code inside your training script. \n",
    "\n",
    "In `pytorch_train.py`, we will log some metrics to our AML run. To do so, we will access the AML run object within the script:\n",
    "```Python\n",
    "from azureml.core.run import Run\n",
    "run = Run.get_context()\n",
    "```\n",
    "Further within `pytorch_train.py`, we log the learning rate and momentum parameters, the best validation accuracy the model achieves, and the number of classes in the model:\n",
    "```Python\n",
    "run.log('lr', np.float(learning_rate))\n",
    "run.log('momentum', np.float(momentum))\n",
    "run.log('num_classes', num_classes)\n",
    "\n",
    "run.log('best_val_acc', np.float(best_acc))\n",
    "```\n",
    "\n",
    "If you downloaded the data, you can start to train the model locally (note that it will take long if you don't have a GPU -- 21 min. on a Core i7 CPU).\n",
    "\n",
    "**This step requires Pytorch to be installed locally -- find instructions [here](https://pytorch.org/#pip-install-pytorch)**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir outputs\n",
    "!python pytorch_train.py --data_dir breeds-10 --num_epochs 10 --output_dir outputs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model on the remote compute\n",
    "Now that you have your data and training script prepared, you are ready to train on your remote compute cluster. You can take advantage of Azure compute to leverage GPUs to cut down your training time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an experiment\n",
    "Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this transfer learning PyTorch tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "experiment_name = 'pytorch-dogs' \n",
    "experiment = Experiment(ws, name=experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a PyTorch estimator\n",
    "The AML SDK's PyTorch estimator enables you to easily submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch). The following code will define a single-node PyTorch job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "##BATCH AI\n",
    "from azureml.train.dnn import PyTorch\n",
    "\n",
    "script_params = {\n",
    "    '--data_dir': ds_data.as_mount(),\n",
    "    '--num_epochs': 10,\n",
    "    '--output_dir': './outputs',\n",
    "    '--log_dir': './logs',\n",
    "    '--mode': 'fine_tune'\n",
    "}\n",
    "\n",
    "estimator10 = PyTorch(source_directory='.', \n",
    "                    script_params=script_params,\n",
    "                    compute_target=compute_target, \n",
    "                    entry_script='pytorch_train.py',\n",
    "                    pip_packages=['tensorboardX'],\n",
    "                    use_gpu=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `script_params` parameter is a dictionary containing the command-line arguments to your training script `entry_script`. Please note the following:\n",
    "- We passed our training data reference `ds_data` to our script's `--data_dir` argument. This will 1) mount our datastore on the remote compute and 2) provide the path to the training data `breeds` on our datastore.\n",
    "- We specified the output directory as `./outputs`. The `outputs` directory is specially treated by AML in that all the content in this directory gets uploaded to your workspace as part of your run history. The files written to this directory are therefore accessible even once your remote run is over. In this tutorial, we will save our trained model to this output directory.\n",
    "\n",
    "To leverage the Azure VM's GPU for training, we set `use_gpu=True`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit job\n",
    "Run your experiment by submitting your estimator object. Note that this call is asynchronous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run10 = experiment.submit(estimator10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor your run\n",
    "You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run10).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What happens during a run?\n",
    "If you are running this for the first time, the compute target will need to pull the docker image, which will take about 2 minutes. This gives us the time to go over how a **Run** is executed in Azure Machine Learning. \n",
    "\n",
    "Note: had we not created the workspace with an existing ACR, we would have also had to wait for the image creation to be performed -- that takes and extra 10-20 minutes for big GPU images like this one. This is a one-time cost for a given python configuration, and subsequent runs will then be faster. We are working on ways to make this image creation faster.\n",
    "\n",
    "![](aml-run.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Tensorboard\n",
    "Tensorboard is a popular Deep Learning Training visualization tool. It is part of TensorFlow framework, but can be used from PyTorch as well by using TensorboadX python package. TensorboardX allows PyTorch to write metrics and log information in Tensorboard format.\n",
    "\n",
    "In `pytorch_train.py`, we will log metrics to \"magical\" `logs` directory. The content of this special directory is automatically streamed by Azue ML service. The logging into Tensorboard event format is performed by SummaryWriter object:\n",
    "```Python\n",
    "from tensorboardX import SummaryWriter\n",
    "writer = SummaryWriter(f'./logs/{run.id}')\n",
    "```\n",
    "Within the script we write more detailed mini-batch loss and accuracy, as well as epoch level accuracy to Tensorboard:\n",
    "```Python\n",
    "writer.add_scalar(f'{phase}/Loss', loss.item(), niter)\n",
    "writer.add_scalar(f'{phase}/Epoch_accuracy', epoch_acc, (epoch+1) * len(dataloaders[phase]))\n",
    "```\n",
    "Finally, to ensure that TensorboardX python package is installed in Python environment during the training run, we add it using pip_packages parameter of the Estimator object:\n",
    "```Python\n",
    "estimator = PyTorch(...\n",
    "                    pip_packages=['tensorboardX']\n",
    "                    ...)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing and launching Tensorboard\n",
    "Azure ML SDK provides built-in integration with Tensorboard in package `azureml-contrib-tensorboard`, installed as part of the contrib extras package in prerequisites. In addition, you will need to pip install tensorboard, which we also did as part of the prerequisites.\n",
    "\n",
    "While the run is in progress (or after it has completed), we just need to start Tensorboard with the run as its target, and it will begin streaming logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.tensorboard import Tensorboard\n",
    "\n",
    "# The Tensorboard constructor takes an array of runs, so be sure and pass it in as a single-element array here\n",
    "tb = Tensorboard([run10])\n",
    "\n",
    "# If successful, start() returns a string with the URI of the instance.\n",
    "tb.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Tensorboard\n",
    "\n",
    "When you're done, make sure to call the `stop()` method of the Tensorboard object, or it will stay running even after your job completes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tb.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed training\n",
    "\n",
    "Now that the setup is working, we can go to the full dataset with 120 classes. We just need to point to a different path on the datastore. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_dataset = ds.path('breeds')\n",
    "print(full_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## AML Compute\n",
    "from azureml.train.dnn import PyTorch\n",
    "\n",
    "script_params = {\n",
    "    '--data_dir': full_dataset.as_mount(),\n",
    "    '--num_epochs': 25,\n",
    "    '--output_dir': './outputs',\n",
    "    '--log_dir': './logs',\n",
    "    '--mode': 'fine_tune'\n",
    "}\n",
    "\n",
    "estimator120 = PyTorch(source_directory='.', \n",
    "                        script_params=script_params,\n",
    "                        compute_target=compute_target, \n",
    "                        entry_script='pytorch_train.py',\n",
    "                        pip_packages=['tensorboardX'],\n",
    "                        node_count=1,\n",
    "                        use_gpu=True)\n",
    "\n",
    "run120 = experiment.submit(estimator120)\n",
    "\n",
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run120).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But now training takes very long (1.5 hours), so let's see if we can run this job on multiple GPUs to cut down on training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first let's cancel the above job\n",
    "run120.cancel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the model on multiple nodes is simple (in this case using Horovod MPI-based algorithm running on 4 nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## AML Compute\n",
    "from azureml.train.dnn import PyTorch\n",
    "\n",
    "script_params = {\n",
    "    '--data_dir': full_dataset.as_mount(),\n",
    "    '--num_epochs': 25,\n",
    "    '--output_dir': './outputs',\n",
    "    '--log_dir': './logs',\n",
    "    '--mode': 'fine_tune'\n",
    "}\n",
    "\n",
    "estimator120 = PyTorch(source_directory='.', \n",
    "                        script_params=script_params,\n",
    "                        compute_target=compute_target, \n",
    "                        pip_packages=['tensorboardX'],\n",
    "                        entry_script='pytorch_train_horovod.py',\n",
    "                        node_count=4,\n",
    "                        distributed_backend='mpi',\n",
    "                        use_gpu=True)\n",
    "\n",
    "run120 = experiment.submit(estimator120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run120).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.tensorboard import Tensorboard\n",
    "\n",
    "# The Tensorboard constructor takes an array of runs, so be sure and pass it in as a single-element array here\n",
    "tb = Tensorboard([run120])\n",
    "\n",
    "# If successful, start() returns a string with the URI of the instance.\n",
    "tb.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tb.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on 4 nodes completes in about 10 minutes and achieves 76% accuracy, which is similar to accuracy produced by single node training. This is great improvement of training time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning\n",
    " Now that you have trained an initial model, you can tune the hyperparameters of this model to optimize model performance. Azure ML allows you to automate this tuning, in an efficient manner via early termination of poorly performing runs.\n",
    "\n",
    "You can configure your Hyperparamter Tuning experiment by specifying the following info -\n",
    "- Define the hyparparameter space - specify ranges, distribution and sampling\n",
    "- Early Termination policy\n",
    "- Optimization metric\n",
    "- Resource / Compute budget\n",
    "- Desired concurrency\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "from azureml.train.hyperdrive import *\n",
    "\n",
    "ps = RandomParameterSampling(\n",
    "    {\n",
    "        '--momentum': uniform(0.6,0.99),\n",
    "        '--learning_rate': loguniform(-6, -3)\n",
    "    }\n",
    ")\n",
    "\n",
    "policy = BanditPolicy(evaluation_interval=2, slack_factor=0.2)\n",
    "\n",
    "\n",
    "hdc = HyperDriveRunConfig(estimator=estimator10, \n",
    "                          hyperparameter_sampling=ps, \n",
    "                          policy=policy, \n",
    "                          primary_metric_name='best_val_acc', \n",
    "                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
    "                          max_total_runs=50,\n",
    "                          max_concurrent_runs=4)\n",
    "\n",
    "hd_run = experiment.submit(hdc)\n",
    "RunDetails(hd_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the jobs is running, take a look at the [Hyperdrive documentation](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive?view=azure-ml-py) to see what other options Hyperdrive offers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# at any time, you can pull out the metrics generated so far from the runs\n",
    "hd_run.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Azure Machine Learning Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Azure Machine Learning Pipelines](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) enables data scientists to create and manage multiple simple and complex workflows concurrently. A typical pipeline would have multiple tasks to prepare data, train, deploy and evaluate models. Individual steps in the pipeline can make use of diverse compute options (for example: CPU for data preparation and GPU for training) and languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Step in a Pipeline\n",
    "A Step is a unit of execution. Step typically needs a target of execution (compute target), a script to execute, and may require script arguments and inputs, and can produce outputs. The step also could take a number of other parameters. Azure Machine Learning Pipelines provides the following built-in Steps:\n",
    "\n",
    "- [**PythonScriptStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py): Add a step to run a Python script in a Pipeline.\n",
    "- [**AdlaStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.adla_step.adlastep?view=azure-ml-py): Adds a step to run U-SQL script using Azure Data Lake Analytics.\n",
    "- [**DataTransferStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.data_transfer_step.datatransferstep?view=azure-ml-py): Transfers data between Azure Blob and Data Lake accounts.\n",
    "- [**DatabricksStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py): Adds a DataBricks notebook as a step in a Pipeline.\n",
    "- [**HyperDriveStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.hyper_drive_step.hyperdrivestep?view=azure-ml-py): Creates a Hyper Drive step for Hyper Parameter Tuning in a Pipeline.\n",
    "- [**EstimatorStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.estimator_step.estimatorstep?view=azure-ml-py): Adds a step to run Estimator in a Pipeline.\n",
    "- [**MpiStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.mpi_step.mpistep?view=azure-ml-py): Adds a step to run a MPI job in a Pipeline.\n",
    "- [**HyperDriveStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.hyper_drive_step.hyperdrivestep?view=azure-ml-py): Creates a HyperDrive step in a Pipeline.\n",
    "\n",
    "The following code will create a PythonScriptStep to be executed in the Azure Machine Learning Compute we created above using train.py, one of the files already made available in the project folder.\n",
    "\n",
    "A **PythonScriptStep** is a basic, built-in step to run a Python Script on a compute target. It takes a script name and optionally other parameters like arguments for the script, compute target, inputs and outputs. If no compute target is specified, default compute target for the workspace is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.steps import PythonScriptStep\n",
    "\n",
    "# All steps use files already available in the project_folder (local directory)\n",
    "project_folder = '.'\n",
    "\n",
    "# Uses default values for PythonScriptStep construct.\n",
    "\n",
    "# Syntax\n",
    "# PythonScriptStep(\n",
    "#     script_name, \n",
    "#     name=None, \n",
    "#     arguments=None, \n",
    "#     compute_target=None, \n",
    "#     runconfig=None, \n",
    "#     inputs=None, \n",
    "#     outputs=None, \n",
    "#     params=None, \n",
    "#     source_directory=None, \n",
    "#     allow_reuse=True, \n",
    "#     version=None, \n",
    "#     hash_paths=None)\n",
    "# This returns a Step\n",
    "trainStep = PythonScriptStep(name=\"train_step\",\n",
    "                         script_name=\"train.py\", \n",
    "                         compute_target=compute_target, \n",
    "                         source_directory=project_folder,\n",
    "                         allow_reuse=True)\n",
    "print(\"trainStep created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** In the above call to PythonScriptStep(), the flag *allow_reuse* determines whether the step should reuse previous results when run with the same settings/inputs. This flag's default value is *True*; the default is set to *True* because, when inputs and parameters have not changed, we typically do not want to re-run a given pipeline step. \n",
    "\n",
    "If *allow_reuse* is set to *False*, a new run will always be generated for this step during pipeline execution. The *allow_reuse* flag can come in handy in situations where you do *not* want to re-run a pipeline step.\n",
    "\n",
    "### Running a few steps in parallel\n",
    "Here we are looking at a simple scenario where we are running a few steps (all involving PythonScriptStep)  in parallel. Running nodes in **parallel** is the default behavior for steps in a pipeline.\n",
    "\n",
    "We already have one step defined earlier. Let's define few more steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All steps use the same Azure Machine Learning compute target as well\n",
    "compareStep = PythonScriptStep(name=\"compare_step\",\n",
    "                         script_name=\"compare.py\", \n",
    "                         compute_target=compute_target, \n",
    "                         source_directory=project_folder)\n",
    "\n",
    "extractStep = PythonScriptStep(name=\"extract_step\",\n",
    "                         script_name=\"extract.py\", \n",
    "                         compute_target=compute_target, \n",
    "                         source_directory=project_folder)\n",
    "\n",
    "# list of steps to run\n",
    "steps = [trainStep, compareStep, extractStep]\n",
    "print(\"Step lists created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the pipeline\n",
    "Once we have the steps (or steps collection), we can build the [pipeline](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline.pipeline?view=azure-ml-py). By deafult, all these steps will run in **parallel** once we submit the pipeline for run.\n",
    "\n",
    "A pipeline is created with a list of steps and a workspace. Submit a pipeline using [submit](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment%28class%29?view=azure-ml-py#submit). When submit is called, a [PipelineRun](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinerun?view=azure-ml-py) is created which in turn creates [StepRun](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.steprun?view=azure-ml-py) objects for each step in the workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "# Syntax\n",
    "# Pipeline(workspace, \n",
    "#          steps, \n",
    "#          description=None, \n",
    "#          default_datastore_name=None, \n",
    "#          default_source_directory=None, \n",
    "#          resolve_closure=True, \n",
    "#          _workflow_provider=None, \n",
    "#          _service_endpoint=None)\n",
    "\n",
    "pipeline1 = Pipeline(workspace=ws, steps=steps)\n",
    "print (\"Pipeline is built\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the pipeline\n",
    "You have the option to [validate](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline.pipeline?view=azure-ml-py#validate) the pipeline prior to submitting for run. The platform runs validation steps such as checking for circular dependencies and parameter checks etc. even if you do not explicitly call validate method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline1.validate()\n",
    "print(\"Pipeline validation complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the pipeline\n",
    "[Submitting](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline.pipeline?view=azure-ml-py#submit) the pipeline involves creating an [Experiment](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment?view=azure-ml-py) object and providing the built pipeline for submission. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from azureml.core import Experiment\n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "exp_name = time_format + \"Hello_World1\"\n",
    "# Submit syntax\n",
    "# submit(experiment_name, \n",
    "#        pipeline_parameters=None, \n",
    "#        continue_on_node_failure=False, \n",
    "#        regenerate_outputs=False)\n",
    "\n",
    "pipeline_run1 = Experiment(ws, exp_name).submit(pipeline1, regenerate_outputs=False)\n",
    "print(\"Pipeline is submitted for execution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If regenerate_outputs is set to True, a new submit will always force generation of all step outputs, and disallow data reuse for any step of this run. Once this run is complete, however, subsequent runs may reuse the results of this run.\n",
    "\n",
    "### Examine the pipeline run\n",
    "\n",
    "#### Use RunDetails Widget\n",
    "We are going to use the RunDetails widget to examine the run of the pipeline. You can click each row below to get more details on the step runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(pipeline_run1).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use Pipeline SDK objects\n",
    "You can cycle through the node_run objects and examine job logs, stdout, and stderr of each of the steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "step_runs = pipeline_run1.get_children()\n",
    "for step_run in step_runs:\n",
    "    status = step_run.get_status()\n",
    "    print('Script:', step_run.name, 'status:', status)\n",
    "    \n",
    "    # Change this if you want to see details even if the Step has succeeded.\n",
    "    if status == \"Failed\":\n",
    "        joblog = step_run.get_job_log()\n",
    "        print('job log:', joblog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get additonal run details\n",
    "If you wait until the pipeline_run is finished, you may be able to get additional details on the run. **Since this is a blocking call, the following code is commented out.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pipeline_run1.wait_for_completion()\n",
    "#for step_run in pipeline_run1.get_children():\n",
    "#    print(\"{}: {}\".format(step_run.name, step_run.get_metrics()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a few steps in sequence\n",
    "Now let's see how we run a few steps in sequence. We already have three steps defined earlier. Let's *reuse* those steps for this part.\n",
    "\n",
    "We will reuse step1, step2, step3, but build the pipeline in such a way that we chain step3 after step2 and step2 after step1. Note that there is no explicit data dependency between these steps, but still steps can be made dependent by using the [run_after](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.builder.pipelinestep?view=azure-ml-py#run-after) construct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import StepSequence\n",
    "\n",
    "compareStep.run_after(trainStep)\n",
    "extractStep.run_after(compareStep)\n",
    "\n",
    "pipeline2 = Pipeline(workspace=ws, steps=[extractStep])\n",
    "print (\"Pipeline is built\")\n",
    "\n",
    "pipeline2.validate()\n",
    "print(\"Simple validation complete\")\n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "exp_name = time_format + \"Hello_World2\"\n",
    "\n",
    "pipeline_run2 = Experiment(ws, exp_name).submit(pipeline2)\n",
    "print(\"Pipeline is submitted for execution\")\n",
    "\n",
    "RunDetails(pipeline_run2).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Pipeline Steps with Inputs and Outputs\n",
    "As mentioned earlier, a step in the pipeline can take data as input. This data can be a data source that lives in one of the accessible data locations, or intermediate data produced by a previous step in the pipeline.\n",
    "\n",
    "### Datasources\n",
    "Datasource is represented by **[DataReference](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.data_reference.datareference?view=azure-ml-py)** object and points to data that lives in or is accessible from Datastore. DataReference could be a pointer to a file or a directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload data to your datastore\n",
    "Let's upload a file to the blob storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ds = Datastore(ws,'breeds')\n",
    "ds.upload_files([\"./20news.pkl\"], target_path=\"pipeline\", overwrite=True)\n",
    "\n",
    "print(\"Upload calls completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reference this data\n",
    "Reference the data you just uploaded using DataReference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data.data_reference import DataReference\n",
    "\n",
    "# Reference the data uploaded to blob storage using DataReference\n",
    "# Assign the datasource to blob_input_data variable\n",
    "\n",
    "# DataReference(datastore, \n",
    "#               data_reference_name=None, \n",
    "#               path_on_datastore=None, \n",
    "#               mode='mount', \n",
    "#               path_on_compute=None, \n",
    "#               overwrite=False)\n",
    "\n",
    "blob_input_data = DataReference(\n",
    "    datastore=ds,\n",
    "    data_reference_name=\"test_data\",\n",
    "    path_on_datastore=\"pipeline/20news.pkl\")\n",
    "print(\"DataReference object created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermediate/Output Data\n",
    "Intermediate data (or output of a Step) is represented by **[PipelineData](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py)** object. PipelineData can be produced by one step and consumed in another step by providing the PipelineData object as an output of one step and the input of one or more steps.\n",
    "\n",
    "#### Constructing PipelineData\n",
    "- **name:** [*Required*] Name of the data item within the pipeline graph\n",
    "- **datastore_name:** Name of the Datastore to write this output to\n",
    "- **output_name:** Name of the output\n",
    "- **output_mode:** Specifies \"upload\" or \"mount\" modes for producing output (default: mount)\n",
    "- **output_path_on_compute:** For \"upload\" mode, the path to which the module writes this output during execution\n",
    "- **output_overwrite:** Flag to overwrite pre-existing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import PipelineData\n",
    "\n",
    "# Define intermediate data using PipelineData\n",
    "# Syntax\n",
    "\n",
    "# PipelineData(name, \n",
    "#              datastore=None, \n",
    "#              output_name=None, \n",
    "#              output_mode='mount', \n",
    "#              output_path_on_compute=None, \n",
    "#              output_overwrite=None, \n",
    "#              data_type=None, \n",
    "#              is_directory=None)\n",
    "\n",
    "# Naming the intermediate data as processed_data1 and assigning it to the variable processed_data1.\n",
    "processed_data1 = PipelineData(\"processed_data1\",datastore=ds)\n",
    "print(\"PipelineData object created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipelines steps using datasources and intermediate data\n",
    "Machine learning pipelines can have many steps and these steps could use or reuse datasources and intermediate data. Here's how we construct such a pipeline:\n",
    "\n",
    "#### Define a Step that consumes a datasource and produces intermediate data.\n",
    "In this step, we define a step that consumes a datasource and produces intermediate data.\n",
    "\n",
    "**Open `train.py` in the local machine and examine the arguments, inputs, and outputs for the script. That will give you a good sense of why the script argument names used below are important.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainStep consumes the datasource (Datareference) in the previous step\n",
    "# and produces processed_data1\n",
    "trainStep = PythonScriptStep(\n",
    "    script_name=\"train.py\", \n",
    "    arguments=[\"--input_data\", blob_input_data, \"--output_train\", processed_data1],\n",
    "    inputs=[blob_input_data],\n",
    "    outputs=[processed_data1],\n",
    "    compute_target=compute_target, \n",
    "    source_directory=project_folder\n",
    ")\n",
    "print(\"trainStep created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a Step that consumes intermediate data and produces intermediate data\n",
    "In this step, we define a step that consumes an intermediate data and produces intermediate data.\n",
    "\n",
    "**Open `extract.py` in the local machine and examine the arguments, inputs, and outputs for the script. That will give you a good sense of why the script argument names used below are important.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extractStep to use the intermediate data produced by step4\n",
    "# This step also produces an output processed_data2\n",
    "processed_data2 = PipelineData(\"processed_data2\", datastore=ds)\n",
    "\n",
    "extractStep = PythonScriptStep(\n",
    "    script_name=\"extract.py\",\n",
    "    arguments=[\"--input_extract\", processed_data1, \"--output_extract\", processed_data2],\n",
    "    inputs=[processed_data1],\n",
    "    outputs=[processed_data2],\n",
    "    compute_target=compute_target, \n",
    "    source_directory=project_folder)\n",
    "print(\"extractStep created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a Step that consumes multiple intermediate data and produces intermediate data\n",
    "In this step, we define a step that consumes multiple intermediate data and produces intermediate data.\n",
    "\n",
    "**Open `compare.py` in the local machine and examine the arguments, inputs, and outputs for the script. That will give you a good sense of why the script argument names used below are important.**\n",
    "\n",
    "This step also has a [PipelineParameter](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.graph.pipelineparameter?view=azure-ml-py) argument that help with calling the REST endpoint of the published pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core.graph import PipelineParameter\n",
    "# We will use this later in publishing pipeline\n",
    "pipeline_param = PipelineParameter(name=\"pipeline_arg\", default_value=10)\n",
    "print(\"pipeline parameter created\")\n",
    "\n",
    "# Now define compareStep that takes two inputs (both intermediate data), and a pipeline parameter and produce an output\n",
    "processed_data3 = PipelineData(\"processed_data3\", datastore=ds)\n",
    "\n",
    "compareStep = PythonScriptStep(\n",
    "    script_name=\"compare.py\",\n",
    "    arguments=[\"--compare_data1\", processed_data1, \"--compare_data2\", processed_data2, \"--output_compare\", processed_data3, \"--pipeline_param\", pipeline_param],\n",
    "    inputs=[processed_data1, processed_data2],\n",
    "    outputs=[processed_data3],    \n",
    "    compute_target=compute_target, \n",
    "    source_directory=project_folder)\n",
    "print(\"compareStep created\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline1 = Pipeline(workspace=ws, steps=[compareStep])\n",
    "print (\"Pipeline is built\")\n",
    "\n",
    "pipeline1.validate()\n",
    "print(\"Simple validation complete\") \n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "exp_name = time_format + \"Data_dependency\"\n",
    "\n",
    "pipeline_run1 = Experiment(ws, exp_name).submit(pipeline1)\n",
    "print(\"Pipeline is submitted for execution\")\n",
    "\n",
    "RunDetails(pipeline_run1).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Publish the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import PublishedPipeline\n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "pipeline_name = time_format + \"Data_dependency\"\n",
    "\n",
    "published_pipeline1 = pipeline1.publish(name=pipeline_name, description=\"My_Published_Pipeline_Description\")\n",
    "print(published_pipeline1.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run published pipeline using its REST endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.authentication import AzureCliAuthentication\n",
    "import requests\n",
    "\n",
    "cli_auth = AzureCliAuthentication()\n",
    "aad_token = cli_auth.get_authentication_header()\n",
    "\n",
    "rest_endpoint1 = published_pipeline1.endpoint\n",
    "\n",
    "print(rest_endpoint1)\n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "exp_name = time_format + \"My_Pipeline\"\n",
    "\n",
    "\n",
    "# specify the param when running the pipeline\n",
    "response = requests.post(rest_endpoint1, \n",
    "                         headers=aad_token, \n",
    "                         json={\"ExperimentName\": exp_name,\n",
    "                               \"RunSource\": \"SDK\",\n",
    "                               \"ParameterAssignments\": {\"pipeline_arg\": 55}})\n",
    "run_id = response.json()[\"Id\"]\n",
    "\n",
    "print(run_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Transfer\n",
    "In certain cases, you will need to transfer data from one data location to another. For example, your data may be in Files storage and you may want to move it to Blob storage. Or, if your data is in an ADLS account and you want to make it available in the Blob storage. The built-in **DataTransferStep** class helps you transfer data in these situations.\n",
    "\n",
    "#### Register Datastores\n",
    "\n",
    "In the code cell below, you will need to fill in the appropriate values for the workspace name, datastore name, subscription id, resource group, store name, tenant id, client id, and client secret that are associated with your ADLS datastore. \n",
    "\n",
    "For background on registering your data store, consult this article:\n",
    "\n",
    "https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory\n",
    "\n",
    "To obtain the keys for the ADLS account, please go to [this oneNote](https://microsoft.sharepoint.com/teams/azuremlnursery/_layouts/15/WopiFrame.aspx?sourcedoc={4c394733-8bc8-4a24-9ba6-5200de206450}&wd=target%28Workshop.one%7C265d85d5-44c8-9d40-b556-a31fa098e708%2FPipeline%20Datatransfer%7C3b63da03-f100-4843-96bf-c9c4805ada5b%2F%29&wdorigin=703) and replace the following cell with it's content.\n",
    "\n",
    "#### Source of the data copy (ADLS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Need to register ADLS, get details from the shared OneNote.\n",
    "\n",
    "adl_datastore_name='MigratedADLS'\n",
    "\n",
    "# These are the details you get from your migrated ADLS account.\n",
    "subscription_id=  # subscription id of ADLS account\n",
    "resource_group=  # resource group of ADLS account\n",
    "store_name=  # ADLS account name\n",
    "tenant_id=  # tenant id of service principal\n",
    "client_id=  # client id of service principal\n",
    "client_secret=  # the secret of service principal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    adls_datastore = Datastore.get(ws, adl_datastore_name)\n",
    "    print(\"Found datastore with name: %s\" % adl_datastore_name)\n",
    "except:\n",
    "    adls_datastore = Datastore.register_azure_data_lake(\n",
    "        workspace=ws,\n",
    "        datastore_name=adl_datastore_name,\n",
    "        subscription_id=subscription_id, # subscription id of ADLS account\n",
    "        resource_group=resource_group, # resource group of ADLS account\n",
    "        store_name=store_name, # ADLS account name\n",
    "        tenant_id=tenant_id, # tenant id of service principal\n",
    "        client_id=client_id, # client id of service principal\n",
    "        client_secret=client_secret) # the secret of service principal\n",
    "    print(\"Registered datastore with name: %s\" % adl_datastore_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alds_data = DataReference(\n",
    "    datastore=adls_datastore,\n",
    "    data_reference_name=\"adlsdata\",\n",
    "    path_on_datastore=\"local/AMLTest/input.tsv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Destination of the data copy (Blob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blob_data = DataReference(datastore=ds,\n",
    "                       path_on_datastore=\"pipeline\",\n",
    "                       data_reference_name=\"blobdata\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copy data using DataTransferStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import DataFactoryCompute\n",
    "from azureml.pipeline.steps import DataTransferStep\n",
    "\n",
    "data_factory_name = 'adftest'\n",
    "\n",
    "try:\n",
    "    data_factory_compute = DataFactoryCompute(ws, data_factory_name)\n",
    "    print(\"Found existing data factory compute.\")\n",
    "except:\n",
    "    print(\"Creating new data factory compute\")\n",
    "    \n",
    "    provisioning_config = DataFactoryCompute.provisioning_configuration()\n",
    "    data_factory_compute = ComputeTarget.create(ws, \"adftest\", provisioning_config)\n",
    "    data_factory_compute.wait_for_completion(show_output=True)\n",
    "\n",
    "print(\"Azure Machine Learning Compute attached\")\n",
    "\n",
    "transfer_adls_to_blob = DataTransferStep(\n",
    "    name=\"Copy_from_ADLS_to_Blob\",\n",
    "    source_data_reference=alds_data,\n",
    "    destination_data_reference=blob_data,\n",
    "    source_reference_type='file',\n",
    "    compute_target=data_factory_compute)\n",
    "print(\"data transfer step created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build and submit the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline3 = Pipeline(\n",
    "    description=\"Data Transfer\",\n",
    "    workspace=ws, \n",
    "    steps=[transfer_adls_to_blob])\n",
    "\n",
    "date_object = datetime.now()\n",
    "time_format = date_object.strftime('%b%d_%H_%M_')\n",
    "exp_name = time_format + \"Data_Transfer\"\n",
    "\n",
    "pipeline_run = Experiment(ws, exp_name).submit(pipeline3)\n",
    "RunDetails(pipeline_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inferencing\n",
    "### Create scoring script\n",
    "\n",
    "First, we will create a scoring script that will be invoked by the web service call. Note that the scoring script must have two required functions:\n",
    "* `init()`: In this function, you typically load the model into a `global` object. This function is executed only once when the Docker container is started. \n",
    "* `run(input_data)`: In this function, the model is used to predict a value based on the input data. The input and output typically use JSON as serialization and deserialization format, but you are not limited to that.\n",
    "\n",
    "Refer to the scoring script `pytorch_score.py` for this tutorial. Our web service will use this file to predict the dog breed based on a 120 class model trained earlier, located in the folder `model`. We will test the scoring file locally first before you go and deploy the web service.\n",
    "\n",
    "1. Import the scoring script, so that init() and run() are accessible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import pytorch_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Call the init function -- this will load the model and the class_names from the model directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pytorch_score.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Add some helper functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, json, base64\n",
    "from io import BytesIO\n",
    "import matplotlib.pyplot as plt\n",
    "from skimage import io\n",
    "from PIL import Image\n",
    "import urllib.request\n",
    "import io, requests\n",
    "\n",
    "##Get random dog\n",
    "def get_random_dog():\n",
    "    r = requests.get(url =\"https://dog.ceo/api/breeds/image/random\")\n",
    "    URL= r.json()['message']\n",
    "    return URL\n",
    "\n",
    "def imgToBase64(img):\n",
    "    \"\"\"Convert pillow image to base64-encoded image\"\"\"\n",
    "    imgio = BytesIO()\n",
    "    img.save(imgio, 'JPEG')\n",
    "    img_str = base64.b64encode(imgio.getvalue())\n",
    "    return img_str.decode('utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Find an image of a dog and call the run() function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Get Random Dog Image\n",
    "URL = get_random_dog()\n",
    "\n",
    "with urllib.request.urlopen(URL) as url:\n",
    "    test_img = io.BytesIO(url.read())\n",
    "\n",
    "\n",
    "plt.imshow(Image.open(test_img))\n",
    "\n",
    "base64Img = imgToBase64(Image.open(test_img))\n",
    "\n",
    "result = pytorch_score.run(input_data=json.dumps({'data': base64Img}))\n",
    "print(URL)\n",
    "print(json.loads(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register the model\n",
    "Once the run completes, we can register the model that was created.\n",
    "\n",
    "**Please use a unique name for the model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "model = Model.register(ws, model_name='model', model_path = 'model', description='120 Dogbreeds')\n",
    "print(model.name, model.id, model.version, sep = '\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model as web service\n",
    "Once you have your trained model, you can deploy the model on Azure. You can deploy your trained model as a web service on Azure Container Instances (ACI), Azure Kubernetes Service (AKS), IoT edge device, or field programmable gate arrays (FPGAs)\n",
    "\n",
    "ACI is generally cheaper than AKS and can be set up in 4-6 lines of code. ACI is the perfect option for testing deployments. Later, when you're ready to use your models and web services for high-scale, production usage, you can deploy them to AKS.\n",
    "\n",
    "\n",
    "In this tutorial, we will deploy the model as a web service in [Azure Container Instances](https://docs.microsoft.com/en-us/azure/container-instances/) (ACI). \n",
    "\n",
    "\n",
    "For more information on deploying models using Azure ML, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-deploy-and-where)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create environment file\n",
    "Then, we will need to create an environment file (`myenv.yml`) that specifies all of the scoring script's package dependencies. This file is used to ensure that all of those dependencies are installed in the Docker image by AML. In this case, we need to specify `torch`, `torchvision`, `pillow`, and `azureml-sdk`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%writefile myenv.yml\n",
    "name: myenv\n",
    "channels:\n",
    "  - defaults\n",
    "dependencies:\n",
    "  - pip:\n",
    "    - torch\n",
    "    - torchvision\n",
    "    - pillow\n",
    "    - azureml-core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the container image\n",
    "Now configure the Docker image that you will use to build your ACI container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.image import ContainerImage\n",
    "\n",
    "image_config = ContainerImage.image_configuration(execution_script='pytorch_score.py', \n",
    "                                                  runtime='python', \n",
    "                                                  conda_file='myenv.yml',\n",
    "                                                  description='Image with dog breed model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the ACI container\n",
    "We are almost ready to deploy. Create a deployment configuration file to specify the number of CPUs and gigabytes of RAM needed for your ACI container. While it depends on your model, the default of `1` core and `1` gigabyte of RAM is usually sufficient for many models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n",
    "                                               memory_gb=1, \n",
    "                                               tags={'data': 'dog_breeds',  'method':'transfer learning', 'framework':'pytorch'},\n",
    "                                               description='Classify dog breeds using transfer learning with PyTorch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the registered model\n",
    "Finally, let's deploy a web service from our registered model. First, retrieve the model from your workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model = ws.models['model']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, deploy the web service using the ACI config and image config files created in the previous steps. We pass the `model` object in a list to the `models` parameter. If you would like to deploy more than one registered model, append the additional models to this list.\n",
    "\n",
    "** Please use a unique service name**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from azureml.core.webservice import Webservice\n",
    "\n",
    "service_name = 'dog120'\n",
    "service = Webservice.deploy_from_model(workspace=ws,\n",
    "                                       name=service_name,\n",
    "                                       models=[model],\n",
    "                                       image_config=image_config,\n",
    "                                       deployment_config=aciconfig,)\n",
    "\n",
    "service.wait_for_deployment(show_output=True)\n",
    "print(service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your deployment fails for any reason and you need to redeploy, make sure to delete the service before you do so: `service.delete()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip: If something goes wrong with the deployment, the first thing to look at is the logs from the service by running the following command:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "service.get_logs()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the web service's HTTP endpoint, which accepts REST client calls. This endpoint can be shared with anyone who wants to test the web service or integrate it into an application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(service.scoring_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the web service\n",
    "Finally, let's test our deployed web service. We will send the data as a JSON string to the web service hosted in ACI and use the SDK's `run` API to invoke the service. Here we will take an arbitrary image from online to predict on. This is the same as above, but now we are testing on our own trained model. You can use any dog image, but please remember we only trained on 10 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "##Get Random Dog Image\n",
    "URL = get_random_dog()\n",
    "\n",
    "with urllib.request.urlopen(URL) as url:\n",
    "    test_img = io.BytesIO(url.read())\n",
    "\n",
    "plt.imshow(Image.open(test_img))\n",
    "\n",
    "with urllib.request.urlopen(URL) as url:\n",
    "    test_img = io.BytesIO(url.read())\n",
    "\n",
    "plt.imshow(Image.open(test_img))\n",
    "print(URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def imgToBase64(img):\n",
    "    \"\"\"Convert pillow image to base64-encoded image\"\"\"\n",
    "    imgio = BytesIO()\n",
    "    img.save(imgio, 'JPEG')\n",
    "    img_str = base64.b64encode(imgio.getvalue())\n",
    "    return img_str.decode('utf-8')\n",
    "\n",
    "base64Img = imgToBase64(Image.open(test_img))\n",
    "\n",
    "result = service.run(input_data=json.dumps({'data': base64Img}))\n",
    "print(json.loads(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete web service\n",
    "Once you no longer need the web service, you should delete it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "service.delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}