17 - Monitor Data Drift.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitoring Data Drift\n",
    "\n",
    "Over time, models can become less effective at predicting accurately due to changing trends in feature data. This phenomenon is known as *data drift*, and it's important to monitor your machine learning solution to detect it so you can retrain your models if necessary.\n",
    "\n",
    "In this lab, you'll configure data drift monitoring for datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the DataDriftDetector module\n",
    "\n",
    "To define a data drift monitor, you'll need to ensure that you have the latest version of the Azure ML SDK installed, and install the **datadrift** module; so run the following cell to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade azureml-sdk azureml-datadrift"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to your workspace\n",
    "\n",
    "With the latest version of the SDK installed, now you're ready to connect to your workspace.\n",
    "\n",
    "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to work with', ws.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a *baseline* dataset\n",
    "\n",
    "To monitor a dataset for data drift, you must register a *baseline* dataset (usually the dataset used to train your model) to use as a point of comparison with data collected in the future. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore, Dataset\n",
    "\n",
    "\n",
    "# Upload the baseline data\n",
    "default_ds = ws.get_default_datastore()\n",
    "default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'],\n",
    "                       target_path='diabetes-baseline',\n",
    "                       overwrite=True, \n",
    "                       show_progress=True)\n",
    "\n",
    "# Create and register the baseline dataset\n",
    "print('Registering baseline dataset...')\n",
    "baseline_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-baseline/*.csv'))\n",
    "baseline_data_set = baseline_data_set.register(workspace=ws, \n",
    "                           name='diabetes baseline',\n",
    "                           description='diabetes baseline data',\n",
    "                           tags = {'format':'CSV'},\n",
    "                           create_new_version=True)\n",
    "\n",
    "print('Baseline dataset registered!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a *target* dataset\n",
    "\n",
    "Over time, you can collect new data with the same features as your baseline training data. To compare this new data to the baseline data, you must define a target dataset that includes the features you want to analyze for data drift as well as a timestamp field that indicates the point in time when the new data was current -this enables you to measure data drift over temporal intervals. The timestamp can either be a field in the dataset itself, or derived from the folder and filename pattern used to store the data. For example, you might store new data in a folder hierarchy that consists of a folder for the year, containing a folder for the month, which in turn contains a folder for the day; or you might just encode the year, month, and day in the file name like this: *data_2020-01-29.csv*; which is the approach taken in the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import pandas as pd\n",
    "\n",
    "print('Generating simulated data...')\n",
    "\n",
    "# Load the smaller of the two data files\n",
    "data = pd.read_csv('data/diabetes2.csv')\n",
    "\n",
    "# We'll generate data for the past 6 weeks\n",
    "weeknos = reversed(range(6))\n",
    "\n",
    "file_paths = []\n",
    "for weekno in weeknos:\n",
    "    \n",
    "    # Get the date X weeks ago\n",
    "    data_date = dt.date.today() - dt.timedelta(weeks=weekno)\n",
    "    \n",
    "    # Modify data to ceate some drift\n",
    "    data['Pregnancies'] = data['Pregnancies'] + 1\n",
    "    data['Age'] = round(data['Age'] * 1.2).astype(int)\n",
    "    data['BMI'] = data['BMI'] * 1.1\n",
    "    \n",
    "    # Save the file with the date encoded in the filename\n",
    "    file_path = 'data/diabetes_{}.csv'.format(data_date.strftime(\"%Y-%m-%d\"))\n",
    "    data.to_csv(file_path)\n",
    "    file_paths.append(file_path)\n",
    "\n",
    "# Upload the files\n",
    "path_on_datastore = 'diabetes-target'\n",
    "default_ds.upload_files(files=file_paths,\n",
    "                       target_path=path_on_datastore,\n",
    "                       overwrite=True,\n",
    "                       show_progress=True)\n",
    "\n",
    "# Use the folder partition format to define a dataset with a 'date' timestamp column\n",
    "partition_format = path_on_datastore + '/diabetes_{date:yyyy-MM-dd}.csv'\n",
    "target_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, path_on_datastore + '/*.csv'),\n",
    "                                                       partition_format=partition_format)\n",
    "\n",
    "# Register the target dataset\n",
    "print('Registering target dataset...')\n",
    "target_data_set = target_data_set.with_timestamp_columns('date').register(workspace=ws,\n",
    "                                                                          name='diabetes target',\n",
    "                                                                          description='diabetes target data',\n",
    "                                                                          tags = {'format':'CSV'},\n",
    "                                                                          create_new_version=True)\n",
    "\n",
    "print('Target dataset registered!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a data drift monitor\n",
    "\n",
    "Now you're ready to create a data drift monitor for the diabetes data. The data drift monitor will run periodicaly or on-demand to compare the baseline dataset with the target dataset, to which new data will be added over time.\n",
    "\n",
    "### Create a compute target\n",
    "\n",
    "To run the data drift monitor, you'll need a compute target. Run the following cell to specify a compute cluster (if it doesn't exist, it will be created).\n",
    "\n",
    "> **Important**: Change *your-compute-cluster* to the name of your compute cluster in the code below before running it! Cluster names must be globally unique names between 2 to 16 characters in length. Valid characters are letters, digits, and the - character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "cluster_name = \"your-compute-cluster\"\n",
    "\n",
    "try:\n",
    "    # Check for existing compute target\n",
    "    training_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    # If it doesn't already exist, create it\n",
    "    try:\n",
    "        compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS11_V2', max_nodes=2)\n",
    "        training_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "        training_cluster.wait_for_completion(show_output=True)\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the data drift monitor\n",
    "\n",
    "Now you're ready to use a **DataDriftDetector** class to define the data drift monitor for your data. You can specify the features you want to monitor for data drift, the name of the compute target to be used to run the monitoring process, the frequency at which the data should be compared, the data drift threshold above which an alert should be triggered, and the latency (in hours) to allow for data collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.datadrift import DataDriftDetector\n",
    "\n",
    "# set up feature list\n",
    "features = ['Pregnancies', 'Age', 'BMI']\n",
    "\n",
    "# set up data drift detector\n",
    "monitor = DataDriftDetector.create_from_datasets(ws, 'mslearn-diabates-drift', baseline_data_set, target_data_set,\n",
    "                                                      compute_target=cluster_name, \n",
    "                                                      frequency='Week', \n",
    "                                                      feature_list=features, \n",
    "                                                      drift_threshold=.3, \n",
    "                                                      latency=24)\n",
    "monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backfill the data drift monitor\n",
    "\n",
    "You have a baseline dataset and a target dataset that includes simulated weekly data collection for six weeks. You can use this to backfill the monitor so that it can analyze data drift between the original baseline and the target data.\n",
    "\n",
    "> **Note** This may take some time to run, as the compute target must be started to run the backfill analysis. The widget may not always update to show the status, so click the link to observe the experiment status in Azure Machine Learning studio!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "\n",
    "backfill = monitor.backfill(dt.datetime.now() - dt.timedelta(weeks=6), dt.datetime.now())\n",
    "\n",
    "RunDetails(backfill).show()\n",
    "backfill.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze data drift\n",
    "\n",
    "You can use the following code to examine data drift for the points in time collected in the backfill run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_metrics = backfill.get_metrics()\n",
    "for metric in drift_metrics:\n",
    "    print(metric, drift_metrics[metric])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also visualize the data drift metrics in [Azure Machine Learning studio](https://ml.azure.com) by following these steps:\n",
    "\n",
    "1. On the **Datasets** page, view the **Dataset monitors** tab.\n",
    "2. Click the data drift monitor you want to view.\n",
    "3. Select the date range over which you want to view data drift metrics (if the column chart does not show multiple weeks of data, wait a minute or so and click **Refresh**).\n",
    "4. Examine the charts in the **Drift overview** section at the top, which show overall drift magnitude and the drift contribution per feature.\n",
    "5. Explore the charts in the **Feature detail** section at the bottom, which enable you to see various measures of drift for individual features.\n",
    "\n",
    "> **Note**: For help understanding the data drift metrics, see the [How to monitor datasets](https://docs.microsoft.com/azure/machine-learning/how-to-monitor-datasets#understanding-data-drift-results) in the Azure Machine Learning documentation.\n",
    "\n",
    "## Explore further\n",
    "\n",
    "This lab is designed to introduce you to the concepts and principles of data drift monitoring. To learn more about monitoring data drift using datasets, see the [Detect data drift on datasets](https://docs.microsoft.com/azure/machine-learning/how-to-monitor-datasets) in the Azure machine Learning documentation.\n",
    "\n",
    "You can also collect data from published services and use it as a target dataset for datadrift monitoring. See [Collect data from models in production](https://docs.microsoft.com/azure/machine-learning/how-to-enable-data-collection) for details.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}