intro_to_ai_ethics/05-model-cards.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "517b48f6",
   "metadata": {
    "papermill": {
     "duration": 0.013073,
     "end_time": "2022-03-06T22:49:16.734041",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.720968",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [AI Ethics](https://www.kaggle.com/learn/ai-ethics) course.  You can reference the tutorial at [this link](https://www.kaggle.com/var0101/model-cards).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "777f8359",
   "metadata": {
    "papermill": {
     "duration": 0.011328,
     "end_time": "2022-03-06T22:49:16.757633",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.746305",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In the tutorial, you learned how to use model cards.  In this exercise, you'll sharpen your understanding of model cards by engaging with them in a couple of scenarios.\n",
    "\n",
    "# Introduction\n",
    "\n",
    "Run the next code cell to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "65f3d43f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T22:49:16.791116Z",
     "iopub.status.busy": "2022-03-06T22:49:16.790401Z",
     "iopub.status.idle": "2022-03-06T22:49:16.828543Z",
     "shell.execute_reply": "2022-03-06T22:49:16.827811Z",
     "shell.execute_reply.started": "2022-03-06T22:35:07.756155Z"
    },
    "papermill": {
     "duration": 0.059548,
     "end_time": "2022-03-06T22:49:16.828722",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.769174",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ethics.ex5 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aed566d",
   "metadata": {
    "papermill": {
     "duration": 0.010889,
     "end_time": "2022-03-06T22:49:16.851153",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.840264",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Scenario A\n",
    "\n",
    "You are the creator of the _Simple Zoom_ video editing tool, which uses AI to automatically zoom the video camera in on a presenter as they walk across a room during a presentation. You are launching the _Simple Zoom_ tool and releasing a model card along with the tool, in the interest of transparency.\n",
    "\n",
    "# 1) Audience\n",
    "\n",
    "Which audiences should you write the model card for? Once you have thought about your answer, check it against the solution below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d50d76d1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T22:49:16.878033Z",
     "iopub.status.busy": "2022-03-06T22:49:16.877007Z",
     "iopub.status.idle": "2022-03-06T22:49:16.889021Z",
     "shell.execute_reply": "2022-03-06T22:49:16.889599Z",
     "shell.execute_reply.started": "2022-03-06T22:35:41.563683Z"
    },
    "papermill": {
     "duration": 0.027193,
     "end_time": "2022-03-06T22:49:16.889789",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.862596",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1_ScenarioA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Solution:</span> Model cards should be written for the groups that are most likely to read it. For *Simple Zoom*, such groups probably include people using the tool to record videos, organizations seeking to adopt the tool, IT and audio-visual teams and agencies, computer vision researchers, policymakers and developers of similar AI systems. Given how broad this group is, your team can only assume a basic knowledge of video recording terms throughout the model card."
      ],
      "text/plain": [
       "Solution: Model cards should be written for the groups that are most likely to read it. For *Simple Zoom*, such groups probably include people using the tool to record videos, organizations seeking to adopt the tool, IT and audio-visual teams and agencies, computer vision researchers, policymakers and developers of similar AI systems. Given how broad this group is, your team can only assume a basic knowledge of video recording terms throughout the model card."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a87a553d",
   "metadata": {
    "papermill": {
     "duration": 0.012927,
     "end_time": "2022-03-06T22:49:16.916337",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.903410",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Scenario B\n",
    "\n",
    "You are the product manager for _Presenter Pro_, a popular video and audio recording product for people delivering talks and presentations. As a new feature based on customer demand, your team has been planning to add the AI-powered ability for a single video camera to automatically track a presenter, focusing on them as they walk across the room or stage, zooming in and out automatically and continuously adjusting lighting and focus within the video frame. \n",
    "\n",
    "You are hoping to incorporate a different company’s AI tool (called _Simple Zoom_) into your product (_Presenter Pro_). To determine whether _Simple Zoom_ is a good fit for _Presenter Pro_, you are reviewing *Simple Zoom*’s model card.\n",
    "\n",
    "# 2) Intended Use\n",
    "\n",
    "The **Intended Use** section of the model card includes the following bullets:\n",
    "* _Simple Zoom_ is intended to be used for automatic zoom, focus, and lighting adjustment in the real-time video recording of individual presenters by a single camera\n",
    "* _Simple Zoom_ is not suitable for presentations in which there is more than one presenter or for presentations in which the presenter is partially or fully hidden at any time\n",
    "\n",
    "As a member of the team evaluating _Simple Zoom_ for potential integration into _Presenter Pro_, you are aware that _Presenter Pro_ only supports one presenter per video.\n",
    "\n",
    "However, you are also aware that in some _Presenter Pro_ customers use large props in their presentation videos. Given the information in the Intended Use section of the model card, what problem do you foresee for these customers if you integrate _Simple Zoom_ into _Presenter Pro_? What are some ways in which you could address this issue?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "134ee2ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T22:49:16.948911Z",
     "iopub.status.busy": "2022-03-06T22:49:16.947813Z",
     "iopub.status.idle": "2022-03-06T22:49:16.953035Z",
     "shell.execute_reply": "2022-03-06T22:49:16.953635Z",
     "shell.execute_reply.started": "2022-03-06T22:40:26.674269Z"
    },
    "papermill": {
     "duration": 0.02422,
     "end_time": "2022-03-06T22:49:16.953829",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.929609",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 4, \"questionId\": \"2_ScenarioB\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> There is a risk that the large props could partially or fully hide the presenter."
      ],
      "text/plain": [
       "Hint: There is a risk that the large props could partially or fully hide the presenter."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run this code cell to receive a hint\n",
    "q_2.hint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "faa0a154",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T22:49:16.991110Z",
     "iopub.status.busy": "2022-03-06T22:49:16.990042Z",
     "iopub.status.idle": "2022-03-06T22:49:16.995794Z",
     "shell.execute_reply": "2022-03-06T22:49:16.996411Z",
     "shell.execute_reply.started": "2022-03-06T22:40:36.941722Z"
    },
    "papermill": {
     "duration": 0.027558,
     "end_time": "2022-03-06T22:49:16.996607",
     "exception": false,
     "start_time": "2022-03-06T22:49:16.969049",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"2_ScenarioB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Solution:</span> Since *Simple Zoom* is not suitable for presentations in which the presenter is partially or fully hidden at any time, it might not work well in a presentation in which the presenter uses a large object, because the object could partially or fully hide the presenter. There are many potential ways to address this issue. For example, your team could reach out to the *Simple Zoom* team to assess the potential risks and harms of using *Simple Zoom* with props. As another example, your team could eventually add a message in the Presenter Pro user interface explaining that the *Simple Zoom* feature should not be used in presentations that use props."
      ],
      "text/plain": [
       "Solution: Since *Simple Zoom* is not suitable for presentations in which the presenter is partially or fully hidden at any time, it might not work well in a presentation in which the presenter uses a large object, because the object could partially or fully hide the presenter. There are many potential ways to address this issue. For example, your team could reach out to the *Simple Zoom* team to assess the potential risks and harms of using *Simple Zoom* with props. As another example, your team could eventually add a message in the Presenter Pro user interface explaining that the *Simple Zoom* feature should not be used in presentations that use props."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_2.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f0f7ad8",
   "metadata": {
    "papermill": {
     "duration": 0.015644,
     "end_time": "2022-03-06T22:49:17.028488",
     "exception": false,
     "start_time": "2022-03-06T22:49:17.012844",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Factors, Evaluation Data, Metrics, and Quantitative Analyses\n",
    "\n",
    "We'll continue with **Scenario B**, where you are the product manager for _Presenter Pro_.  Four more sections of the model card for _Simple Zoom_ are described below.\n",
    "\n",
    "**Factors**: The model card lists the following factors as potentially relevant to model performance: \n",
    "* Group Factors\n",
    "  * Self-reported gender\n",
    "  * Skin tone\n",
    "  * Self-reported age\n",
    "* Other Factors\n",
    "  * Camera angle\n",
    "  * Presenter distance from camera\n",
    "  * Camera type\n",
    "  * Lighting\n",
    "\n",
    "**Evaluation Data**: To generate the performance metrics reported in the **Quantitative Analysis** section (discussed below), the _Simple Zoom_ team used an evaluation data set of 500 presentation videos, each between two and five minutes long. The videos included both regular and unusual presentation and recording scenarios and included presenters from various demographic backgrounds.\n",
    "\n",
    "**Metrics**: Since _Simple Zoom_ model performance is subjective (involving questions like whether a zoom is of appropriate speed or smoothness; or whether a lighting adjustment is well-executed), the _Simple Zoom_ team tested the tool’s performance by asking a diverse viewer group to view _Simple Zoom_’s output videos (using the evaluation dataset’s 500 videos as inputs). Each viewer was asked to rate the quality of video editing for each video on a scale of 1 to 10, and each video’s average rating was used as a proxy for _Simple Zoom_’s performance on that video.\n",
    "\n",
    "**Quantitative Analyses**: The quantitative analyses section of the model card includes a brief summary of performance results. According to the summary, the model generally performs equally well across all the listed demographic groups (gender, skin tone and age).\n",
    "\n",
    "The quantitative analyses section also includes interactive graphs, which allow you to view the performance of the _Simple Zoom_ tool by each factor and by intersections of ‘Group’ and ‘Other’ factors.\n",
    "\n",
    "As a member of the team evaluating _Simple Zoom_ for potential integration into _Presenter Pro_, what are some questions you might be interested in answering and exploring via the interactive graphs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7b16d1db",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-06T22:49:17.064405Z",
     "iopub.status.busy": "2022-03-06T22:49:17.063301Z",
     "iopub.status.idle": "2022-03-06T22:49:17.071958Z",
     "shell.execute_reply": "2022-03-06T22:49:17.072536Z",
     "shell.execute_reply.started": "2022-03-06T22:46:22.49637Z"
    },
    "papermill": {
     "duration": 0.028405,
     "end_time": "2022-03-06T22:49:17.072740",
     "exception": false,
     "start_time": "2022-03-06T22:49:17.044335",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"3_ScenarioB2\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Solution:</span> There are many possible answers to this question. For example, you may want to check that the model’s equal performance across demographic groups (gender, skin tone and age) remains equal across different camera angles, distances from camera, camera types and lighting conditions. As another example, you may want to know how well the model performs from the specific camera angles that _Production Pro_ customers most commonly use."
      ],
      "text/plain": [
       "Solution: There are many possible answers to this question. For example, you may want to check that the model’s equal performance across demographic groups (gender, skin tone and age) remains equal across different camera angles, distances from camera, camera types and lighting conditions. As another example, you may want to know how well the model performs from the specific camera angles that _Production Pro_ customers most commonly use."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_3.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288176cf",
   "metadata": {
    "papermill": {
     "duration": 0.017349,
     "end_time": "2022-03-06T22:49:17.107318",
     "exception": false,
     "start_time": "2022-03-06T22:49:17.089969",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Congratulations!  You have reached the end of the AI Ethics course.  Once you have viewed all of the tutorials and successfully answered the questions in the exercises, you will receive a completion certificate to celebrate your hard work.  \n",
    "\n",
    "If you enjoyed the AI Ethics course, we recommend the [**Machine Learning Explainability**](https://www.kaggle.com/learn/machine-learning-explainability) course as a next step.  To prepare for the content in this course, you should have some background in python (which you can get from the [**Python**](https://www.kaggle.com/learn/python) course) and machine learning (which you can get from the [**Intro to Machine Learning**](https://www.kaggle.com/learn/intro-to-machine-learning) course)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "552d9eff",
   "metadata": {
    "papermill": {
     "duration": 0.016987,
     "end_time": "2022-03-06T22:49:17.141646",
     "exception": false,
     "start_time": "2022-03-06T22:49:17.124659",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/230765) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 11.093739,
   "end_time": "2022-03-06T22:49:17.772339",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-03-06T22:49:06.678600",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}