Sample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure Document Intelligence Custom Template User Feedback Loop Experiment\n",
    "\n",
    "This experiment demonstrates how to replicate the functionality of the [Azure AI Document Intelligence](https://learn.microsoft.com/en-GB/azure/ai-services/document-intelligence/overview) Studio custom model training process. The aim is to showcase how you may implement a user feedback loop for improving the quality of document processing results. The feedback mechanism can be implemented to allow developers of custom models in Azure AI Document Intelligence to collect feedback from users to improve the model's performance.\n",
    "\n",
    "This notebook provides an interactive user feedback experience, enabling a user to analyze a document using a trained model, visualize the analysis results overlaid on the document, and correct any incorrectly identified or missing fields. This implementation could be replicated in any client application using your chosen framework's capabilities, such as React, Angular, or Vue.js.\n",
    "\n",
    "> **Note**: This notebook provides _one_ potential approach to user interaction, and can be interpreted in many ways based on your use case.\n",
    "\n",
    "## Pre-requisites\n",
    "\n",
    "> **Note**: Before continuing, please ensure that the [`Setup-Environment.ps1`](./Setup-Environment.ps1) script has been run to deploy the required infrastructure to Azure. This includes the Azure AI Document Intelligence resource and the Azure Storage account for creating a custom model.\n",
    "\n",
    "This notebook uses [Dev Containers](https://code.visualstudio.com/docs/remote/containers) to ensure that all the required dependencies are available in a consistent local development environment.\n",
    "\n",
    "The following are required to run this notebook:\n",
    "\n",
    "- [Visual Studio Code](https://code.visualstudio.com/)\n",
    "- [Docker Desktop](https://www.docker.com/products/docker-desktop)\n",
    "- [Remote - Containers extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)\n",
    "\n",
    "> **Note**: The Dev Container is pre-configured with the required dependencies and extensions. You can run this notebook outside of a Dev Container, but you will need to manually install the required dependencies including Poppler, Tesseract, and OpenCV.\n",
    "\n",
    "The Dev Container will include the following dependencies by default:\n",
    "\n",
    "- Debian 11 (Bullseye) base image\n",
    "- Python 3.12\n",
    "  - azure-ai-formrecognizer - for interacting with the Azure AI Document Intelligence service\n",
    "  - azure-core - for interacting with the Azure AI Document Intelligence service\n",
    "  - ipycanvas - for rendering the document and allowing the user to draw over it\n",
    "  - ipykernel - for running the notebook\n",
    "  - notebook - for running the notebook\n",
    "  - opencv-python-headless - for image processing\n",
    "  - pdf2image - for converting PDFs to images\n",
    "  - pytesseract - for performing OCR on the document\n",
    "- Poppler - used by pdf2image to convert PDFs to images\n",
    "- Tesseract OCR - used by pytesseract to perform OCR on the document\n",
    "- Python3 OpenCV - used for image processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Requirements\n",
    "\n",
    "The following code block imports the required dependencies for this notebook.\n",
    "\n",
    "It also configures the following:\n",
    "\n",
    "- Setup the local working directory.\n",
    "- Load local environment variables based on the output of the [`Setup-Environment.ps1`](./Setup-Environment.ps1) script run. The environment variables will be available in the [`.env`](./.env) file.\n",
    "- Initialize the credential that will be used to authentication with the Azure services.\n",
    "\n",
    "> **Note**: The [`Setup-Environment.ps1`](./Setup-Environment.ps1) script is not run as part of this notebook. It must be run separately, prior to running this notebook, to deploy the required infrastructure to Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from dotenv import dotenv_values\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from modules.app_settings import AppSettings\n",
    "from modules.model_training_client import ModelTrainingClient\n",
    "from modules.document_canvas import (DocumentCanvas)\n",
    "from modules.document_intelligence_label import DocumentIntelligenceLabel\n",
    "from modules.document_intelligence_result_formatter import DocumentIntelligenceResultFormatter\n",
    "\n",
    "working_dir = os.path.abspath('')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/config.env\"))\n",
    "azure_credential = DefaultAzureCredential(\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_visual_studio_code_credential=False,\n",
    "    exclude_cli_credential=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a custom extraction model in Azure AI Document Intelligence\n",
    "\n",
    "This experiment comes prepared with the data required to train a custom model. The data is located in the [`model_training`](./model_training/) directory and contains a set of invoices that will be used to create the initial custom model.\n",
    "\n",
    "The following code blocks will create a model training client, using the [`ModelTrainingClient`](./modules/model_training_client.py), and run it to upload the files to and Azure Storage blob container, and training the model using Azure AI Document Intelligence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of the model\n",
    "model_name = 'invoices' \n",
    "\n",
    "# The version of the model\n",
    "initial_model_version = '1.0.0'\n",
    "\n",
    "# The name of the model that will be registered in Azure AI Document Intelligence\n",
    "initial_model_id = f\"{model_name}-{initial_model_version}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_training_client = ModelTrainingClient(settings=settings, use_azure_credential=False, azure_credential=azure_credential)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resets the sample environment to only contain the initial training set. This is only necessary if the sample has been run previously.\n",
    "model_training_client.delete_training_data(\"Invoice_6\")\n",
    "\n",
    "# Uploads the initial training set to Azure Blob Storage and initiates model training using the uploaded data.\n",
    "model_training_client.upload_training_data(f\"{working_dir}/model_training\")\n",
    "invoice_model = model_training_client.create_model(model_name=initial_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demonstrate a user feedback loop experience for improving the model\n",
    "\n",
    "The user feedback loop is a mechanism that allows users of your model to provide feedback on the quality of the results generated by the model from interactions they have with it using their own data.\n",
    "\n",
    "The following code blocks emulates what a user experience flow may present itself within an intelligent application interfacing with Azure AI Document Intelligence.\n",
    "\n",
    "The steps include:\n",
    "\n",
    "- Analyzing a document using the custom model (this is required for providing the user feedback experience) and the prebuilt-layout model (this is required for the training of the custom model).\n",
    "- Visualizing the analysis results overlaid on the document.\n",
    "- Allowing the user to correct any incorrectly identified or missing fields.\n",
    "- Providing the corrected data to the model for retraining.\n",
    "- Using the retrained model to analyze a document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of the PDF file the user is providing.\n",
    "pdf_file_name = 'Invoice_6.pdf'\n",
    "\n",
    "# The directory containing the PDF file.\n",
    "pdf_dir = os.path.join(working_dir, 'pdfs')\n",
    "\n",
    "# The file path to the PDF file for loading.\n",
    "pdf_path = os.path.join(pdf_dir, pdf_file_name)\n",
    "\n",
    "# The file path to where the required JSON result from Azure AI Document Intelligence layout analysis will be stored.\n",
    "pdf_ocr_path = os.path.join(pdf_dir, f\"{pdf_file_name}.ocr.json\")\n",
    "\n",
    "# The file path to where the initial analysis of the user feedback document will be stored.\n",
    "pdf_feedback_path = os.path.join(pdf_dir, f\"{pdf_file_name}.ocr_{initial_model_version}.json\")\n",
    "\n",
    "# The file path to where the required JSON result for Azure AI Document Intelligence labels will be stored after user feedback.\n",
    "pdf_labels_path = os.path.join(pdf_dir, f\"{pdf_file_name}.labels.json\")\n",
    "\n",
    "# The file path to where the required document fields are, based on the original model training data.\n",
    "document_fields_path = os.path.join(working_dir, 'model_training', 'fields.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run layout analysis on the document using Azure AI Document Intelligence\n",
    "\n",
    "This step will use the Azure AI Document Intelligence service to perform layout analysis on the PDF document. When complete, the files will be saved to the `./pdfs` directory with the name format `<pdf_file_name>.ocr.json`.\n",
    "\n",
    "> **Note**: These specific steps do not need to be run every time. The layout analysis is only required to be run once to capture the initial state of the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retraining a model requires that the OCR result provided in the training data set is created using the 'prebuilt-layout' model.\n",
    "model_training_client.run_layout_analysis(pdf_path, pdf_ocr_path, 'prebuilt-layout')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For providing the feedback, the user would perform their analysis using your initial model.\n",
    "model_training_client.run_layout_analysis(pdf_path, pdf_feedback_path, initial_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the document in the notebook for user feedback\n",
    "\n",
    "This step will render the document inside the notebook for the user to interact with. This is only a visual representation for the purpose of this experiment, and in a real-world scenario, this would be implemented in a client application.\n",
    "\n",
    "The following code block will perform the following:\n",
    "\n",
    "1. Load a document and store each page as an image.\n",
    "1. Display the rendered image below as an interactive element in an output cell, rendering the output of the layout analysis over the image as label regions.\n",
    "1. Allow you to move, remove, and resize label regions on the rendered image, and add fields to correct any incorrectly identified or missing fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_canvas = DocumentCanvas(working_dir)\n",
    "\n",
    "canvases = doc_canvas.load_pdf(pdf_path, document_fields_path, pdf_feedback_path)\n",
    "for canvas in canvases:\n",
    "    display(canvas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process the user feedback into Document Intelligence labels format\n",
    "\n",
    "Once the user has corrected the document analysis, the following code will process the label regions into the labels JSON format used by the Azure AI Document Intelligence service. The files will be saved to the `./pdfs` directory with the name format `<pdf_file_name>.labels.json`.\n",
    "\n",
    "In a real-world scenario, labels may be presented alongside the rendered document UI, connected to the label regions, to allow the user to update the text and field, and then retrain the model using the updated labels and PDF documents.\n",
    "\n",
    "The following code blocks will render the label regions as UI inputs. The inputs will be pre-populated, and you can update the details of each label associated with the document before re-training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [DocumentIntelligenceLabel(label_region, doc_canvas.fields) for label_region in doc_canvas.get_document_labels()]\n",
    "    \n",
    "for label in labels:\n",
    "    display(label.render())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the labels JSON file\n",
    "\n",
    "Once the user has updated the labels and text associated with the label regions, the following code block will create the labels JSON file in the format required by the Azure AI Document Intelligence service. The file will be saved to the `./pdfs` directory with the name format `<pdf_file_name>.labels.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DocumentIntelligenceResultFormatter.save_to_labels_json(labels, pdf_file_name, pdf_labels_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrain the model using the updated labels and PDF documents\n",
    "\n",
    "The next step emulates a post-user feedback loop experience, where the updated labels and PDF documents are used to retrain the model using the Azure AI Document Intelligence service. \n",
    "\n",
    "This would typically be done by you, as the application developer, manually by reviewing your user's feedback, selecting the appropriate documents to retrain the model with, and then processing them through the Azure AI Document Intelligence service.\n",
    "\n",
    "The following code blocks will upload the updated labels and PDF documents to the Azure Storage blob container for the model training data set, and then retrain the model using the Azure AI Document Intelligence service. The updated model will then be used to analyze a document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The version of the updated model, in this example, a minor change by adding a new training document.\n",
    "updated_model_version = \"1.1.0\"\n",
    "\n",
    "# The name of the model that will be registered in Azure AI Document Intelligence\n",
    "updated_model_id = f\"{model_name}-{updated_model_version}\"\n",
    "\n",
    "# Uploads the updated user feedback documents to Azure Blob Storage and initiates model training using both the existing and new data.\n",
    "model_training_client.upload_training_data(pdf_dir)\n",
    "updated_model = model_training_client.create_model(model_name=updated_model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The file path to where the updated analysis of the user feedback document will be stored.\n",
    "pdf_updated_analysis_path = os.path.join(pdf_dir, f\"{pdf_file_name}.ocr_{updated_model_version}.json\")\n",
    "\n",
    "# Run layout analysis with the updated model\n",
    "model_training_client.run_layout_analysis(pdf_path, pdf_updated_analysis_path, updated_model_id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}