notebooks/demo-adult-income.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explaining XGBoost Model trained on Adult Income Data\n",
    "\n",
    "This notebook contains a user guide on how to use TE2Rules to explain a XGBoost binary classification model trained using scikit-learn. TE2Rules explains a Tree Ensemble model using rules. This notebook contains different levers that can be used to control the faithfulness and interpretability of the extracted rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Load Python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import metrics\n",
    "\n",
    "# TE2Rules supports tree ensemble models from scikit-learn and xgboost   \n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "import te2rules\n",
    "from te2rules.explainer import ModelExplainer\n",
    "\n",
    "print(\"Using te2rules version: \" + str(te2rules.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load pre-processed training and testing data\n",
    "\n",
    "Adult Income data can be found in the [UCI repository](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/). It contains various numerical and categorical features like age, hours of work, capital-gain, education-level, marital-status, race, sex etc., to predict if a person's annual income is above 50K USD or below 50K USD. \n",
    "\n",
    "The pre-processed data used in this notebook can be generated by running ```python3 data_prep/data_prep_adult.py```. This script downloads the adult income data, cleans missing values and encodes categorical features with one-hot encoding. The records with income values above 50K USD are labeled as positives and the rest are labeled as negatives.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(123)\n",
    "\n",
    "training_path = \"../data/adult/train.csv\"\n",
    "testing_path = \"../data/adult/test.csv\"\n",
    "\n",
    "data_train = pd.read_csv(training_path)\n",
    "data_test = pd.read_csv(testing_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = list(data_train.columns)\n",
    "feature_names = cols[:-1]\n",
    "label_name = cols[-1]\n",
    "\n",
    "data_train = data_train.to_numpy()\n",
    "data_test = data_test.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = data_train[:, :-1]\n",
    "y_train = data_train[:, -1]\n",
    "\n",
    "x_test = data_test[:, :-1]\n",
    "y_test = data_test[:, -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a XGBoost model using scikit-learn or xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scikit-Learn Model\n",
    "model = GradientBoostingClassifier(n_estimators=10, max_depth=3)\n",
    "model.fit(x_train, y_train)\n",
    "\n",
    "# XGBoost Model\n",
    "# model = XGBClassifier(n_estimators=10, max_depth=3)\n",
    "# model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = model.predict(x_train)\n",
    "y_train_pred_score = model.predict_proba(x_train)[:, 1]\n",
    "\n",
    "y_test_pred = model.predict(x_test)\n",
    "y_test_pred_score = model.predict_proba(x_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model.score(x_test, y_test)\n",
    "print(\"Accuracy\")\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_test_pred_score)\n",
    "auc = metrics.auc(fpr, tpr)\n",
    "print(\"AUC\")\n",
    "print(auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explain the XGBoost model using TE2Rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_explainer = ModelExplainer(\n",
    "    model=model, \n",
    "    feature_names=feature_names\n",
    ")\n",
    "\n",
    "rules = model_explainer.explain(\n",
    "    X=x_train, y=y_train_pred,\n",
    "    num_stages = 10,               # stages can be between 1 and max_depth \n",
    "    min_precision = 0.95,          # higher min_precision can result in rules with more terms overfit on training data \n",
    "    jaccard_threshold = 0.4        # lower jaccard_threshold speeds up the rule exploration, but can miss some good rules\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretability: Inspect the rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(str(len(rules)) + \" rules found:\")\n",
    "print()\n",
    "for i in range(len(rules)):\n",
    "    print(\"Rule \" + str(i) + \": \" + str(rules[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Faithfulness: Fidelity of the rules\n",
    "\n",
    "If the fidelity on positives is not high enough, try running with more `num_stages` and higher `jaccard_threshold`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fidelity, positive_fidelity, negative_fidelity = model_explainer.get_fidelity()\n",
    "\n",
    "print(\"The rules explain \" + str(round(fidelity*100, 2)) + \"% of the overall predictions of the model\" )\n",
    "print(\"The rules explain \" + str(round(positive_fidelity*100, 2)) + \"% of the positive predictions of the model\" )\n",
    "print(\"The rules explain \" + str(round(negative_fidelity*100, 2)) + \"% of the negative predictions of the model\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All possible explanations\n",
    "\n",
    "TE2Rules provides one possible set of explanations to explain the positive model predictions. TE2Rules finds all possible explanations from a model and then shortlists a small subset of these rules such that they explain most of the positives. However, these rules are not the only possible way to explain the model.\n",
    "\n",
    "TE2Rules can also show all possible explanations to explain the model prediction. From these longer set of possible rules, a domain expert using their domain knowledge can choose a smaller set of rules that closely aligns with the decision-making process in their domain. These shortlisted rules can be used as an alternative of the default subset of rules selected by TE2Rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules = model_explainer.longer_rules\n",
    "print(str(len(rules)) + \" rules found:\")\n",
    "print()\n",
    "\n",
    "for i in range(len(rules)):\n",
    "    print(\"Rule \" + str(i) + \": \" + str(rules[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local Instance-Level Explanations\n",
    "\n",
    "For a given input with positive model prediction, TE2Rules can be used to show different possible reasons for why the model assigned it a positive class prediction. A domain expert can choose the most plausible explanation out of the different possible reasons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from util import display_input\n",
    "\n",
    "explanations = model_explainer.explain_instance_with_rules(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Local Explanations of a particular model decision\")\n",
    "print()\n",
    "for i in range(140, 155):\n",
    "    if(y_test_pred[i] == 1):\n",
    "        print(\"Index:\", i)\n",
    "        print()\n",
    "        print(\"Model Input:\")\n",
    "        display_input(x_test[i], feature_names)\n",
    "        print()\n",
    "        print(\"Model Prediction:\", y_test_pred[i])\n",
    "        print()\n",
    "        print(\"Possible Reasons:\")\n",
    "        rules = explanations[i]\n",
    "        for j in range(len(rules)):\n",
    "            print(\"Rule\", j+1, \":\", rules[j])   \n",
    "        print(\"--------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}