NVIDIA-AI-SQL/NVIDIA-RAG-with-resumes.ipynb

{
    "cells": [
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "507219d1-d713-4c41-86d5-e938bf69627c",
                "language": "sql"
            },
            "source": [
                "# Leveraging Azure SQL DB’s Native Vector Capabilities for Enhanced Resume Matching with NVIDIA AI and Azure Document Intelligence\n",
                "\n",
                "In this tutorial, we will explore how to leverage Azure SQL DB’s new vector data type to store embeddings and perform similarity searches using built-in vector functions, enabling advanced resume matching to identify the most suitable candidates. \n",
                "\n",
                "By extracting and chunking content from PDF resumes using Azure Document Intelligence, generating embeddings with NVIDIA AI, and storing these embeddings in Azure SQL DB, we can perform sophisticated vector similarity searches and retrieval-augmented generation (RAG) to identify the most suitable candidates based on their resumes.\n",
                "\n",
                "### **Tutorial Overview**\n",
                "\n",
                "- This Python notebook will teach you to:\n",
                "    1. **Chunk PDF Resumes**: Use **`Azure Document Intelligence`** to extract and chunk content from PDF resumes.\n",
                "    2. **Create Embeddings**: Generate embeddings from the chunked content using the **`NVIDIA AI Models`**.\n",
                "    3. **Vector Database Utilization**: Store embeddings in **`Azure SQL DB`** utilizing the **`new Vector Data Type`** and perform similarity searches using built-in vector functions to find the most suitable candidates.\n",
                "    4. **LLM Generation Augmentation**: Enhance language model generation with embeddings from a vector database. In this case, we use the embeddings to inform a llama-3.3-70b-instruct model, enabling it to provide rich, context-aware answers about candidates based on their resumes\n",
                "\n",
                "## Dataset\n",
                "\n",
                "We use a sample dataset from [Kaggle](https://www.kaggle.com/datasets/snehaanbhawal/resume-dataset) containing PDF resumes for this tutorial. For the purpose of this tutorial we will use 120 resumes from the **Information-Technology** folder\n",
                "\n",
                "## Prerequisites\n",
                "\n",
                "- **Azure Subscription**: [Create one for free](https://azure.microsoft.com/free/cognitive-services?azure-portal=true)\n",
                "- **Azure SQL Database**: [Set up your database for free](https://learn.microsoft.com/azure/azure-sql/database/free-offer?view=azuresql)\n",
                "- **Azure Document Intelligence** [Create a FreeAzure Doc Intelligence resource](https:/learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0)\n",
                "- SSMS or VS Code to manage database\n",
                "\n",
                "## Additional Requirements for Embedding Generation\n",
                "\n",
                "- **NVIDIA AI**: [NVIDIA AI Models](https://build.nvidia.com/)\n",
                "- **Python**: Version 3.7.1 or later from Python.org. (Sample has been tested with Python 3.11)\n",
                "- **Python Libraries**: Install the required libraries openai, num2words, pandas, tiktoken, and pyodbc.\n",
                "- **Jupyter Notebooks**: Visual Studio Code.\n",
                "\n",
                "\n",
                "## Getting Started\n",
                "\n",
                "1. **Database Setup**: Execute SQL commands from the `CreateTable.sql` script to create the necessary table in your database.\n",
                "2. **Model Deployment**: Deploy an embeddings model via NVIDIA AI\n",
                "\n",
                "![Deployed NVIDIA AI Model](../Assets/NVIDIA_embed.png)\n",
                "\n",
                "\n",
                "3. **Connection String**: Find your Azure SQL DB connection string in the Azure portal under your database settings.\n",
                "4. **Configuration**: Populate the `.env` file with your SQL server connection details , NVIDIA API keys and endpoints, Azure Document Intelligence key and endpoint values.\n",
                "\n",
                "You can retrieve the NVIDIA AI _endpoint_ and _key_:\n",
                "\n",
                "![NVIDIA AI Endpoint and Key](../Assets/NVIDIA_key_endpoint.png)\n",
                "\n",
                "\n",
                "You can [retrieve](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0#get-endpoint-url-and-keys) the Document Intelligence _endpoint_ and _key_:\n",
                "\n",
                "![Azure Document Intelligence Endpoint and Key](../Assets/docintelendpoint.png)\n",
                "\n",
                "## Running the Notebook\n",
                "Use Visual Studio Code\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "azdata_cell_guid": "fe37c601-5918-4055-badc-6c0ba90c68ce",
                "language": "python"
            },
            "outputs": [],
            "source": [
                "#Setup the python libraries required for this notebook\n",
                "#Please ensure that you navigate to the directory containing the `requirements.txt` file in your terminal\n",
                "%pip install -r requirements.txt"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {
                "azdata_cell_guid": "4c29709e-1c3a-495d-83ec-05737e220847",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "True"
                        ]
                    },
                    "execution_count": 4,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "#Load the env details\n",
                "from dotenv import load_dotenv\n",
                "load_dotenv()"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "4b543f05-9036-4887-8737-09aa9f865ec2",
                "language": "python"
            },
            "source": [
                "# **PART 1: Extracting and Chunking Text from PDF Resumes using Azure Document Intelligence**\n",
                "\n",
                "Create an instance of the [DocumentAnalysisClient](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0#get-endpoint-url-and-keys) using the endpoint and API key. \n",
                "\n",
                "[Azure Document Intelligence](https://learn.microsoft.com/azure/ai-services/document-intelligence/?view=doc-intel-4.0.0_)(previously known as Form Recognizer) is a Azure cloud service that uses machine learning to analyze text and structured data from your documents. This client will be used to send requests to the [Azure Document Intelligence](https://learn.microsoft.com/python/api/overview/azure/ai-formrecognizer-readme?view=azure-python) service and receive responses containing the extracted text from the PDF resumes."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "%pip install azure-ai-ml"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {
                "azdata_cell_guid": "b20cc66a-50ce-4486-b275-d4683f4ba545",
                "language": "python"
            },
            "outputs": [],
            "source": [
                "import os\n",
                "import re\n",
                "from azure.ai.formrecognizer import DocumentAnalysisClient\n",
                "from azure.core.credentials import AzureKeyCredential\n",
                "\n",
                "# Load environment variables\n",
                "endpoint = os.getenv(\"AZUREDOCINTELLIGENCE_ENDPOINT\")\n",
                "api_key = os.getenv(\"AZUREDOCINTELLIGENCE_API_KEY\")\n",
                "\n",
                "# Create a DocumentAnalysisClient\n",
                "document_analysis_client = DocumentAnalysisClient(\n",
                "    endpoint=endpoint,\n",
                "    credential=AzureKeyCredential(api_key)\n",
                ")\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "1ecc04b0-1a5f-4d48-819d-2cc06a070062",
                "language": "python"
            },
            "source": [
                "### **Analyze input documents using prebuilt model in Azure Document Intelligence**\n",
                "\n",
                "- DocumentAnalysisClient provides operations for analyzing input documents using prebuilt and custom models through the `begin_analyze_document` and `begin_analyze_document_from_url` APIs. In this tutorial we are using the [prebuilt-layout](https://learn.microsoft.com/python/api/overview/azure/ai-formrecognizer-readme?view=azure-python#using-prebuilt-models)\n",
                "    \n",
                "\n",
                "### **Split text into chunks of 500 tokens**\n",
                "\n",
                "- When faced with content that exceeds the embedding limit, we usually also chunk the content into smaller pieces and then embed those one at a time. Here we will use [tiktoken](https://github.com/openai/tiktoken?tab=readme-ov-file) to chunk the extracted text into token sizes of 500, as we will later pass the extracted chunks to to the `text-embedding-small` model for [generating text embeddings](https://learn.microsoft.com/azure/ai-services/openai/tutorials/embeddings?tabs=python-new%2Ccommand-line&pivots=programming-language-python) as this has a model input token limit of 8192.\n",
                "\n",
                "**Note**: You need to provide the location of the folder where the PDF files reside in the below script."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 23,
            "metadata": {
                "azdata_cell_guid": "0355f92c-0546-4eac-aea8-b55d8bbef194",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of PDF files in the directory: 120\n",
                        "Processing file 1/120: 10089434.pdf\n",
                        "Number of chunks for file 10089434.pdf: 3\n",
                        "File: 10089434.pdf, Chunk ID: 0, Unique Chunk ID: 0_0, Chunk Length: 3035, Chunk Text: INFORMATION TECHNOLOGY TECHNICIAN I Summary Versat...\n",
                        "File: 10089434.pdf, Chunk ID: 1, Unique Chunk ID: 0_1, Chunk Length: 2959, Chunk Text: Disaster Recovery plan and procedures  Researching...\n",
                        "File: 10089434.pdf, Chunk ID: 2, Unique Chunk ID: 0_2, Chunk Length: 2048, Chunk Text: Installing configuring and supporting McAfee antiv...\n",
                        "Processing file 2/120: 10247517.pdf\n",
                        "Number of chunks for file 10247517.pdf: 3\n",
                        "File: 10247517.pdf, Chunk ID: 0, Unique Chunk ID: 1_0, Chunk Length: 3191, Chunk Text: INFORMATION TECHNOLOGY MANAGER Professional Summar...\n",
                        "File: 10247517.pdf, Chunk ID: 1, Unique Chunk ID: 1_1, Chunk Length: 2744, Chunk Text: network which entailed changing software and LAN c...\n",
                        "File: 10247517.pdf, Chunk ID: 2, Unique Chunk ID: 1_2, Chunk Length: 729, Chunk Text: i4 City State 2015 Master of Science  Information ...\n",
                        "Processing file 3/120: 10265057.pdf\n",
                        "Number of chunks for file 10265057.pdf: 2\n",
                        "File: 10265057.pdf, Chunk ID: 0, Unique Chunk ID: 2_0, Chunk Length: 3024, Chunk Text: WORKING RF SYSTEMS ENGINEER Qualifications Microso...\n",
                        "File: 10265057.pdf, Chunk ID: 1, Unique Chunk ID: 2_1, Chunk Length: 1892, Chunk Text: surveys ElectricalValidation Engineer May 2011 to ...\n",
                        "Processing file 4/120: 10553553.pdf\n",
                        "Number of chunks for file 10553553.pdf: 2\n",
                        "File: 10553553.pdf, Chunk ID: 0, Unique Chunk ID: 3_0, Chunk Length: 2934, Chunk Text: INFORMATION TECHNOLOGY MANAGER Summary Dedicated I...\n",
                        "File: 10553553.pdf, Chunk ID: 1, Unique Chunk ID: 3_1, Chunk Length: 1917, Chunk Text: XP Vista and Mac operating systems  Responsible fo...\n",
                        "Processing file 5/120: 10641230.pdf\n",
                        "Number of chunks for file 10641230.pdf: 2\n",
                        "File: 10641230.pdf, Chunk ID: 0, Unique Chunk ID: 4_0, Chunk Length: 2854, Chunk Text: IT MANAGEMENT Career Overview Detailoriented profe...\n",
                        "File: 10641230.pdf, Chunk ID: 1, Unique Chunk ID: 4_1, Chunk Length: 2209, Chunk Text: to the device itself being recharged by a small so...\n",
                        "Processing file 6/120: 10839851.pdf\n",
                        "Number of chunks for file 10839851.pdf: 2\n",
                        "File: 10839851.pdf, Chunk ID: 0, Unique Chunk ID: 5_0, Chunk Length: 2928, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Professional Sum...\n",
                        "File: 10839851.pdf, Chunk ID: 1, Unique Chunk ID: 5_1, Chunk Length: 1336, Chunk Text: business and or technical problems Applies metrics...\n",
                        "Processing file 7/120: 10840430.pdf\n",
                        "Number of chunks for file 10840430.pdf: 3\n",
                        "File: 10840430.pdf, Chunk ID: 0, Unique Chunk ID: 6_0, Chunk Length: 3295, Chunk Text: BRANCH CHIEF INFORMATION TECHNOLOGY SPECIALIST Pro...\n",
                        "File: 10840430.pdf, Chunk ID: 1, Unique Chunk ID: 6_1, Chunk Length: 3179, Chunk Text: computer network operations plans including defens...\n",
                        "File: 10840430.pdf, Chunk ID: 2, Unique Chunk ID: 6_2, Chunk Length: 681, Chunk Text: 2009 DISA Action Officers Course 10Dec2009 DOD Inf...\n",
                        "Processing file 8/120: 11580408.pdf\n",
                        "Number of chunks for file 11580408.pdf: 2\n",
                        "File: 11580408.pdf, Chunk ID: 0, Unique Chunk ID: 7_0, Chunk Length: 2784, Chunk Text: INFORMATION TECHNOLOGY COORDINATOR Career Overview...\n",
                        "File: 11580408.pdf, Chunk ID: 1, Unique Chunk ID: 7_1, Chunk Length: 2741, Chunk Text: dispatch process to an automated delivery system v...\n",
                        "Processing file 9/120: 11584809.pdf\n",
                        "Number of chunks for file 11584809.pdf: 2\n",
                        "File: 11584809.pdf, Chunk ID: 0, Unique Chunk ID: 8_0, Chunk Length: 3013, Chunk Text: MANAGER  INFORMATION TECHNOLOGY AND BUILDING AUTOM...\n",
                        "File: 11584809.pdf, Chunk ID: 1, Unique Chunk ID: 8_1, Chunk Length: 1972, Chunk Text: and Operation New Dawn focusing on network securit...\n",
                        "Processing file 10/120: 11957080.pdf\n",
                        "Number of chunks for file 11957080.pdf: 3\n",
                        "File: 11957080.pdf, Chunk ID: 0, Unique Chunk ID: 9_0, Chunk Length: 3011, Chunk Text: LEAD INFORMATION TECHNOLOGY SUPPORT SPECIALIST Wor...\n",
                        "File: 11957080.pdf, Chunk ID: 1, Unique Chunk ID: 9_1, Chunk Length: 3067, Chunk Text: 20800 calls from user community per year  Analyze ...\n",
                        "File: 11957080.pdf, Chunk ID: 2, Unique Chunk ID: 9_2, Chunk Length: 2855, Chunk Text: web pages to the database and pull information fro...\n",
                        "Processing file 11/120: 12045067.pdf\n",
                        "Number of chunks for file 12045067.pdf: 3\n",
                        "File: 12045067.pdf, Chunk ID: 0, Unique Chunk ID: 10_0, Chunk Length: 3182, Chunk Text: INFORMATION TECHNOLOGY IT SPECIALIST Experience In...\n",
                        "File: 12045067.pdf, Chunk ID: 1, Unique Chunk ID: 10_1, Chunk Length: 2936, Chunk Text: Materials MILOMs as well as independent government...\n",
                        "File: 12045067.pdf, Chunk ID: 2, Unique Chunk ID: 10_2, Chunk Length: 2450, Chunk Text: Provided detailed engineering for the design and i...\n",
                        "Processing file 12/120: 12334140.pdf\n",
                        "Number of chunks for file 12334140.pdf: 5\n",
                        "File: 12334140.pdf, Chunk ID: 0, Unique Chunk ID: 11_0, Chunk Length: 3056, Chunk Text: PRODUCTION ASSOCIATE Summary Conclude your applica...\n",
                        "File: 12334140.pdf, Chunk ID: 1, Unique Chunk ID: 11_1, Chunk Length: 2793, Chunk Text: program software solve customer problems enforce d...\n",
                        "File: 12334140.pdf, Chunk ID: 2, Unique Chunk ID: 11_2, Chunk Length: 2762, Chunk Text: section of your cover letter  Heres how to space y...\n",
                        "File: 12334140.pdf, Chunk ID: 3, Unique Chunk ID: 11_3, Chunk Length: 2792, Chunk Text: formed diagnostics and troubleshooting of system i...\n",
                        "File: 12334140.pdf, Chunk ID: 4, Unique Chunk ID: 11_4, Chunk Length: 851, Chunk Text: application letter should describe what you have t...\n",
                        "Processing file 13/120: 12635195.pdf\n",
                        "Number of chunks for file 12635195.pdf: 1\n",
                        "File: 12635195.pdf, Chunk ID: 0, Unique Chunk ID: 12_0, Chunk Length: 2741, Chunk Text: Objective To obtain a position in the information ...\n",
                        "Processing file 14/120: 12763627.pdf\n",
                        "Number of chunks for file 12763627.pdf: 3\n",
                        "File: 12763627.pdf, Chunk ID: 0, Unique Chunk ID: 13_0, Chunk Length: 2677, Chunk Text: ASPNET WEB DEVELOPER Accomplishments  Won Associat...\n",
                        "File: 12763627.pdf, Chunk ID: 1, Unique Chunk ID: 13_1, Chunk Length: 2713, Chunk Text: Supplier Portal is designed in SharePoint 2010 to ...\n",
                        "File: 12763627.pdf, Chunk ID: 2, Unique Chunk ID: 13_2, Chunk Length: 519, Chunk Text: Mvc net Access Adp Application Software Aspnet 40 ...\n",
                        "Processing file 15/120: 13385306.pdf\n",
                        "Number of chunks for file 13385306.pdf: 2\n",
                        "File: 13385306.pdf, Chunk ID: 0, Unique Chunk ID: 14_0, Chunk Length: 2836, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Profile SUMMAR ...\n",
                        "File: 13385306.pdf, Chunk ID: 1, Unique Chunk ID: 14_1, Chunk Length: 2014, Chunk Text: 022005 Company Name  Provide network application a...\n",
                        "Processing file 16/120: 13405733.pdf\n",
                        "Number of chunks for file 13405733.pdf: 2\n",
                        "File: 13405733.pdf, Chunk ID: 0, Unique Chunk ID: 15_0, Chunk Length: 3300, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Professional Su...\n",
                        "File: 13405733.pdf, Chunk ID: 1, Unique Chunk ID: 15_1, Chunk Length: 2786, Chunk Text: formed risk assessment of stakeholders senior mana...\n",
                        "Processing file 17/120: 13477922.pdf\n",
                        "Number of chunks for file 13477922.pdf: 2\n",
                        "File: 13477922.pdf, Chunk ID: 0, Unique Chunk ID: 16_0, Chunk Length: 2964, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Professional Pro...\n",
                        "File: 13477922.pdf, Chunk ID: 1, Unique Chunk ID: 16_1, Chunk Length: 2098, Chunk Text: Install and maintain video conferencing systems wh...\n",
                        "Processing file 18/120: 13836471.pdf\n",
                        "Number of chunks for file 13836471.pdf: 2\n",
                        "File: 13836471.pdf, Chunk ID: 0, Unique Chunk ID: 17_0, Chunk Length: 2843, Chunk Text: INFORMATION TECHNOLOGY MANAGER Experience Informat...\n",
                        "File: 13836471.pdf, Chunk ID: 1, Unique Chunk ID: 17_1, Chunk Length: 1764, Chunk Text: branch offices  New installation and updates of So...\n",
                        "Processing file 19/120: 14789139.pdf\n",
                        "Number of chunks for file 14789139.pdf: 3\n",
                        "File: 14789139.pdf, Chunk ID: 0, Unique Chunk ID: 18_0, Chunk Length: 3003, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Professional Pr...\n",
                        "File: 14789139.pdf, Chunk ID: 1, Unique Chunk ID: 18_1, Chunk Length: 2825, Chunk Text: serve over 900 members of the Sheriffs Department ...\n",
                        "File: 14789139.pdf, Chunk ID: 2, Unique Chunk ID: 18_2, Chunk Length: 2519, Chunk Text: engineering drawing change solution which reduced ...\n",
                        "Processing file 20/120: 15118506.pdf\n",
                        "Number of chunks for file 15118506.pdf: 2\n",
                        "File: 15118506.pdf, Chunk ID: 0, Unique Chunk ID: 19_0, Chunk Length: 2836, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Executive Profi...\n",
                        "File: 15118506.pdf, Chunk ID: 1, Unique Chunk ID: 19_1, Chunk Length: 2335, Chunk Text: system DMS The DMS is similar to an ERP system for...\n",
                        "Processing file 21/120: 15297298.pdf\n",
                        "Number of chunks for file 15297298.pdf: 2\n",
                        "File: 15297298.pdf, Chunk ID: 0, Unique Chunk ID: 20_0, Chunk Length: 3248, Chunk Text: PRACTICE MANAGER Executive Summary Strategic motiv...\n",
                        "File: 15297298.pdf, Chunk ID: 1, Unique Chunk ID: 20_1, Chunk Length: 2516, Chunk Text: visits to ensure all field staff competencies in E...\n",
                        "Processing file 22/120: 15651486.pdf\n",
                        "Number of chunks for file 15651486.pdf: 3\n",
                        "File: 15651486.pdf, Chunk ID: 0, Unique Chunk ID: 21_0, Chunk Length: 2513, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Career Overview...\n",
                        "File: 15651486.pdf, Chunk ID: 1, Unique Chunk ID: 21_1, Chunk Length: 2801, Chunk Text: LAN VOIP problems and emergencies  Troubleshoot an...\n",
                        "File: 15651486.pdf, Chunk ID: 2, Unique Chunk ID: 21_2, Chunk Length: 1040, Chunk Text: and Cisco Switches  Installed repaired maintained ...\n",
                        "Processing file 23/120: 15791766.pdf\n",
                        "Number of chunks for file 15791766.pdf: 2\n",
                        "File: 15791766.pdf, Chunk ID: 0, Unique Chunk ID: 22_0, Chunk Length: 2876, Chunk Text: PROJECT MANAGER Summary Technical Support Professi...\n",
                        "File: 15791766.pdf, Chunk ID: 1, Unique Chunk ID: 22_1, Chunk Length: 1717, Chunk Text: Performed routine maintenance on workstations IP p...\n",
                        "Processing file 24/120: 15802627.pdf\n",
                        "Number of chunks for file 15802627.pdf: 3\n",
                        "File: 15802627.pdf, Chunk ID: 0, Unique Chunk ID: 23_0, Chunk Length: 3094, Chunk Text: SENIOR VICEPRESIDENT AND CHIEF INFORMATION OFFICER...\n",
                        "File: 15802627.pdf, Chunk ID: 1, Unique Chunk ID: 23_1, Chunk Length: 2964, Chunk Text: costs for developing print materials over traditio...\n",
                        "File: 15802627.pdf, Chunk ID: 2, Unique Chunk ID: 23_2, Chunk Length: 1849, Chunk Text: networks and maintain external and internal web pr...\n",
                        "Processing file 25/120: 16186411.pdf\n",
                        "Number of chunks for file 16186411.pdf: 4\n",
                        "File: 16186411.pdf, Chunk ID: 0, Unique Chunk ID: 24_0, Chunk Length: 2827, Chunk Text: DATABASE PROGRAMMERANALYST NET DEVELOPER Summary S...\n",
                        "File: 16186411.pdf, Chunk ID: 1, Unique Chunk ID: 24_1, Chunk Length: 2858, Chunk Text: from the class timing through Availability Grid an...\n",
                        "File: 16186411.pdf, Chunk ID: 2, Unique Chunk ID: 24_2, Chunk Length: 2821, Chunk Text: to interact with the Database Consumed ADONet Enti...\n",
                        "File: 16186411.pdf, Chunk ID: 3, Unique Chunk ID: 24_3, Chunk Length: 214, Chunk Text: System page pdf Programmer Programming progress qu...\n",
                        "Processing file 26/120: 16533554.pdf\n",
                        "Number of chunks for file 16533554.pdf: 2\n",
                        "File: 16533554.pdf, Chunk ID: 0, Unique Chunk ID: 25_0, Chunk Length: 2817, Chunk Text: INFORMATION TECHNOLOGY MANAGER  NETWORK ENGINEER P...\n",
                        "File: 16533554.pdf, Chunk ID: 1, Unique Chunk ID: 25_1, Chunk Length: 2194, Chunk Text: updated spam filtering Virus and Malware preventio...\n",
                        "Processing file 27/120: 16899268.pdf\n",
                        "Number of chunks for file 16899268.pdf: 2\n",
                        "File: 16899268.pdf, Chunk ID: 0, Unique Chunk ID: 26_0, Chunk Length: 3040, Chunk Text: INFORMATION TECHNOLOGY MANAGERANALYST Professional...\n",
                        "File: 16899268.pdf, Chunk ID: 1, Unique Chunk ID: 26_1, Chunk Length: 784, Chunk Text: Education Family Financial Planning Graduate Certi...\n",
                        "Processing file 28/120: 17111768.pdf\n",
                        "Number of chunks for file 17111768.pdf: 3\n",
                        "File: 17111768.pdf, Chunk ID: 0, Unique Chunk ID: 27_0, Chunk Length: 3090, Chunk Text: INFORMATION TECHNOLOGY PROJECT MANAGER SYSTEM ANAL...\n",
                        "File: 17111768.pdf, Chunk ID: 1, Unique Chunk ID: 27_1, Chunk Length: 2971, Chunk Text: process  Proactively manage the development of cus...\n",
                        "File: 17111768.pdf, Chunk ID: 2, Unique Chunk ID: 27_2, Chunk Length: 2621, Chunk Text: Hosting collaboration sessions dedicated to develo...\n",
                        "Processing file 29/120: 17641670.pdf\n",
                        "Number of chunks for file 17641670.pdf: 4\n",
                        "File: 17641670.pdf, Chunk ID: 0, Unique Chunk ID: 28_0, Chunk Length: 3109, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Professional Sum...\n",
                        "File: 17641670.pdf, Chunk ID: 1, Unique Chunk ID: 28_1, Chunk Length: 3012, Chunk Text: and guidance  Attend scheduled meetings and partic...\n",
                        "File: 17641670.pdf, Chunk ID: 2, Unique Chunk ID: 28_2, Chunk Length: 2945, Chunk Text: Failover F5 Load Balancers G2Sidewinders RouteRefl...\n",
                        "File: 17641670.pdf, Chunk ID: 3, Unique Chunk ID: 28_3, Chunk Length: 681, Chunk Text: et Help Desk Analyst July 2006 to May 2007  Handle...\n",
                        "Processing file 30/120: 17681064.pdf\n",
                        "Number of chunks for file 17681064.pdf: 2\n",
                        "File: 17681064.pdf, Chunk ID: 0, Unique Chunk ID: 29_0, Chunk Length: 2971, Chunk Text: INFORMATION TECHNOLOGY SENIOR MANAGER Summary  15 ...\n",
                        "File: 17681064.pdf, Chunk ID: 1, Unique Chunk ID: 29_1, Chunk Length: 2040, Chunk Text: within the Business Units  Our IT customer surveys...\n",
                        "Processing file 31/120: 17688766.pdf\n",
                        "Number of chunks for file 17688766.pdf: 2\n",
                        "File: 17688766.pdf, Chunk ID: 0, Unique Chunk ID: 30_0, Chunk Length: 2886, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Summary IT Dire...\n",
                        "File: 17688766.pdf, Chunk ID: 1, Unique Chunk ID: 30_1, Chunk Length: 1299, Chunk Text: management hardware configuration and troubleshoot...\n",
                        "Processing file 32/120: 17987433.pdf\n",
                        "Number of chunks for file 17987433.pdf: 3\n",
                        "File: 17987433.pdf, Chunk ID: 0, Unique Chunk ID: 31_0, Chunk Length: 2689, Chunk Text: EPIC INFORMATION TECHNOLOGY SPECIALIST Summary See...\n",
                        "File: 17987433.pdf, Chunk ID: 1, Unique Chunk ID: 31_1, Chunk Length: 2585, Chunk Text: techniques GC  Developed new analytical methods ba...\n",
                        "File: 17987433.pdf, Chunk ID: 2, Unique Chunk ID: 31_2, Chunk Length: 401, Chunk Text: i4 City India Bio Technology 38040 May 2010 Bachel...\n",
                        "Processing file 33/120: 18067556.pdf\n",
                        "Number of chunks for file 18067556.pdf: 4\n",
                        "File: 18067556.pdf, Chunk ID: 0, Unique Chunk ID: 32_0, Chunk Length: 2901, Chunk Text: MASTER DATA MANAGER Experience Master Data Manager...\n",
                        "File: 18067556.pdf, Chunk ID: 1, Unique Chunk ID: 32_1, Chunk Length: 2821, Chunk Text: 1000 cable runs racking 100 AP 300 cameras NVRs co...\n",
                        "File: 18067556.pdf, Chunk ID: 2, Unique Chunk ID: 32_2, Chunk Length: 2996, Chunk Text: hardware and migrating servers and databases this ...\n",
                        "File: 18067556.pdf, Chunk ID: 3, Unique Chunk ID: 32_3, Chunk Length: 1818, Chunk Text: data governance committee oversees the framework f...\n",
                        "Processing file 34/120: 18159866.pdf\n",
                        "Number of chunks for file 18159866.pdf: 3\n",
                        "File: 18159866.pdf, Chunk ID: 0, Unique Chunk ID: 33_0, Chunk Length: 3355, Chunk Text: SENIOR VICE PRESIDENT OF GLOBAL INFORMATION TECHNO...\n",
                        "File: 18159866.pdf, Chunk ID: 1, Unique Chunk ID: 33_1, Chunk Length: 3057, Chunk Text: and technology  Provided highperformance highavail...\n",
                        "File: 18159866.pdf, Chunk ID: 2, Unique Chunk ID: 33_2, Chunk Length: 1177, Chunk Text: domestic and overseas  Responsible for the plannin...\n",
                        "Processing file 35/120: 18176523.pdf\n",
                        "Number of chunks for file 18176523.pdf: 2\n",
                        "File: 18176523.pdf, Chunk ID: 0, Unique Chunk ID: 34_0, Chunk Length: 3056, Chunk Text: SENIOR INFORMATION TECHNOLOGY MANAGER Executive Su...\n",
                        "File: 18176523.pdf, Chunk ID: 1, Unique Chunk ID: 34_1, Chunk Length: 1516, Chunk Text: upgrades repairs configuration and troubleshooting...\n",
                        "Processing file 36/120: 18187364.pdf\n",
                        "Number of chunks for file 18187364.pdf: 3\n",
                        "File: 18187364.pdf, Chunk ID: 0, Unique Chunk ID: 35_0, Chunk Length: 3145, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST INFORMATION SECU...\n",
                        "File: 18187364.pdf, Chunk ID: 1, Unique Chunk ID: 35_1, Chunk Length: 2956, Chunk Text: Soft upgrade and AntiMoney Laundering projects  Re...\n",
                        "File: 18187364.pdf, Chunk ID: 2, Unique Chunk ID: 35_2, Chunk Length: 2318, Chunk Text: State  DBA for telesales signature verification an...\n",
                        "Processing file 37/120: 18301617.pdf\n",
                        "Number of chunks for file 18301617.pdf: 2\n",
                        "File: 18301617.pdf, Chunk ID: 0, Unique Chunk ID: 36_0, Chunk Length: 3286, Chunk Text: INFORMATION TECHNOLOGY MANAGER Summary Successful ...\n",
                        "File: 18301617.pdf, Chunk ID: 1, Unique Chunk ID: 36_1, Chunk Length: 2052, Chunk Text: and hardware compatibility and reliabilityManaged ...\n",
                        "Processing file 38/120: 18752129.pdf\n",
                        "Number of chunks for file 18752129.pdf: 4\n",
                        "File: 18752129.pdf, Chunk ID: 0, Unique Chunk ID: 37_0, Chunk Length: 2948, Chunk Text: INFORMATION TECHNOLOGY ANALYST Summary To pursue a...\n",
                        "File: 18752129.pdf, Chunk ID: 1, Unique Chunk ID: 37_1, Chunk Length: 2767, Chunk Text: makes a body strong I would encourage more of the ...\n",
                        "File: 18752129.pdf, Chunk ID: 2, Unique Chunk ID: 37_2, Chunk Length: 2668, Chunk Text: Train  Table Activity Popsicle sticks craft work m...\n",
                        "File: 18752129.pdf, Chunk ID: 3, Unique Chunk ID: 37_3, Chunk Length: 8, Chunk Text: armapuri...\n",
                        "Processing file 39/120: 19201175.pdf\n",
                        "Number of chunks for file 19201175.pdf: 2\n",
                        "File: 19201175.pdf, Chunk ID: 0, Unique Chunk ID: 38_0, Chunk Length: 3053, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary  Recent ...\n",
                        "File: 19201175.pdf, Chunk ID: 1, Unique Chunk ID: 38_1, Chunk Length: 2374, Chunk Text: ont  City State Skills  Hardware updates  Process ...\n",
                        "Processing file 40/120: 19796840.pdf\n",
                        "Number of chunks for file 19796840.pdf: 4\n",
                        "File: 19796840.pdf, Chunk ID: 0, Unique Chunk ID: 39_0, Chunk Length: 3019, Chunk Text: INFORMATION TECHNOLOGY AUDITOR Skills PeopleSoft H...\n",
                        "File: 19796840.pdf, Chunk ID: 1, Unique Chunk ID: 39_1, Chunk Length: 2960, Chunk Text: any risks to management  Obtain report approval fr...\n",
                        "File: 19796840.pdf, Chunk ID: 2, Unique Chunk ID: 39_2, Chunk Length: 3088, Chunk Text: planning audit schedules risk management complianc...\n",
                        "File: 19796840.pdf, Chunk ID: 3, Unique Chunk ID: 39_3, Chunk Length: 1249, Chunk Text: established reinforcement mechanisms and celebrati...\n",
                        "Processing file 41/120: 19850482.pdf\n",
                        "Number of chunks for file 19850482.pdf: 3\n",
                        "File: 19850482.pdf, Chunk ID: 0, Unique Chunk ID: 40_0, Chunk Length: 2930, Chunk Text: SENIOR VP  INFORMATION TECHNOLOGY Executive Profil...\n",
                        "File: 19850482.pdf, Chunk ID: 1, Unique Chunk ID: 40_1, Chunk Length: 3137, Chunk Text: New Orleans LA Implemented disaster recovery plan ...\n",
                        "File: 19850482.pdf, Chunk ID: 2, Unique Chunk ID: 40_2, Chunk Length: 855, Chunk Text: 1999 Montclair State University Computer Science S...\n",
                        "Processing file 42/120: 20001721.pdf\n",
                        "Number of chunks for file 20001721.pdf: 2\n",
                        "File: 20001721.pdf, Chunk ID: 0, Unique Chunk ID: 41_0, Chunk Length: 3213, Chunk Text: INFORMATION TECHNOLOGY STUDENT Career Overview Res...\n",
                        "File: 20001721.pdf, Chunk ID: 1, Unique Chunk ID: 41_1, Chunk Length: 337, Chunk Text: and various lesson plans to increase concentration...\n",
                        "Processing file 43/120: 20024870.pdf\n",
                        "Number of chunks for file 20024870.pdf: 1\n",
                        "File: 20024870.pdf, Chunk ID: 0, Unique Chunk ID: 42_0, Chunk Length: 2642, Chunk Text: INFORMATION TECHNOLOGY INTERNSHIP Summary MBA grad...\n",
                        "Processing file 44/120: 20237244.pdf\n",
                        "Number of chunks for file 20237244.pdf: 2\n",
                        "File: 20237244.pdf, Chunk ID: 0, Unique Chunk ID: 43_0, Chunk Length: 2927, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary Informat...\n",
                        "File: 20237244.pdf, Chunk ID: 1, Unique Chunk ID: 43_1, Chunk Length: 23, Chunk Text: Digital Media Collector...\n",
                        "Processing file 45/120: 20408458.pdf\n",
                        "Number of chunks for file 20408458.pdf: 4\n",
                        "File: 20408458.pdf, Chunk ID: 0, Unique Chunk ID: 44_0, Chunk Length: 2761, Chunk Text: INFORMATION TECHNOLOGY SUPPORT SPECIALISTNETWORK S...\n",
                        "File: 20408458.pdf, Chunk ID: 1, Unique Chunk ID: 44_1, Chunk Length: 2565, Chunk Text: for SunTrust servers Shared drive space up to 1000...\n",
                        "File: 20408458.pdf, Chunk ID: 2, Unique Chunk ID: 44_2, Chunk Length: 2755, Chunk Text: their new folder on the NAS  Use Remote Desktop Co...\n",
                        "File: 20408458.pdf, Chunk ID: 3, Unique Chunk ID: 44_3, Chunk Length: 52, Chunk Text: Analyst desktop support Troubleshoot type upgrade ...\n",
                        "Processing file 46/120: 20674668.pdf\n",
                        "Number of chunks for file 20674668.pdf: 2\n",
                        "File: 20674668.pdf, Chunk ID: 0, Unique Chunk ID: 45_0, Chunk Length: 2827, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST III DRUPAL DEV S...\n",
                        "File: 20674668.pdf, Chunk ID: 1, Unique Chunk ID: 45_1, Chunk Length: 2614, Chunk Text: Maintaining the NYSEDs existing websites using the...\n",
                        "Processing file 47/120: 20824105.pdf\n",
                        "Number of chunks for file 20824105.pdf: 3\n",
                        "File: 20824105.pdf, Chunk ID: 0, Unique Chunk ID: 46_0, Chunk Length: 2565, Chunk Text: INFORMATION TECHNOLOGY AND AWS ADMIN INTERN Experi...\n",
                        "File: 20824105.pdf, Chunk ID: 1, Unique Chunk ID: 46_1, Chunk Length: 2681, Chunk Text: Set up a VPC network on Amazon and created public ...\n",
                        "File: 20824105.pdf, Chunk ID: 2, Unique Chunk ID: 46_2, Chunk Length: 1120, Chunk Text: SPFMPLSFTPSMTPIPSec VLAN VPN  Network Tools Wiresh...\n",
                        "Processing file 48/120: 20879311.pdf\n",
                        "Number of chunks for file 20879311.pdf: 3\n",
                        "File: 20879311.pdf, Chunk ID: 0, Unique Chunk ID: 47_0, Chunk Length: 3077, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY AND ANALYTICS S...\n",
                        "File: 20879311.pdf, Chunk ID: 1, Unique Chunk ID: 47_1, Chunk Length: 3093, Chunk Text: fullest potential by providing challenging opportu...\n",
                        "File: 20879311.pdf, Chunk ID: 2, Unique Chunk ID: 47_2, Chunk Length: 1779, Chunk Text: in multisite environment  Maintained 85 node devel...\n",
                        "Processing file 49/120: 21283365.pdf\n",
                        "Number of chunks for file 21283365.pdf: 3\n",
                        "File: 21283365.pdf, Chunk ID: 0, Unique Chunk ID: 48_0, Chunk Length: 2734, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Summary I am a ...\n",
                        "File: 21283365.pdf, Chunk ID: 1, Unique Chunk ID: 48_1, Chunk Length: 2843, Chunk Text: was a new technology but my decision demonstrates ...\n",
                        "File: 21283365.pdf, Chunk ID: 2, Unique Chunk ID: 48_2, Chunk Length: 269, Chunk Text: Design  Implementation network policies processes ...\n",
                        "Processing file 50/120: 21780877.pdf\n",
                        "Number of chunks for file 21780877.pdf: 3\n",
                        "File: 21780877.pdf, Chunk ID: 0, Unique Chunk ID: 49_0, Chunk Length: 3052, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST GS11 Experience ...\n",
                        "File: 21780877.pdf, Chunk ID: 1, Unique Chunk ID: 49_1, Chunk Length: 2947, Chunk Text: center personnel  Set and adjust work priorities e...\n",
                        "File: 21780877.pdf, Chunk ID: 2, Unique Chunk ID: 49_2, Chunk Length: 63, Chunk Text: fundamentals of wiring a house for electric cable ...\n",
                        "Processing file 51/120: 22450718.pdf\n",
                        "Number of chunks for file 22450718.pdf: 3\n",
                        "File: 22450718.pdf, Chunk ID: 0, Unique Chunk ID: 50_0, Chunk Length: 3232, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Professional Pro...\n",
                        "File: 22450718.pdf, Chunk ID: 1, Unique Chunk ID: 50_1, Chunk Length: 3152, Chunk Text: performance parameters Systems Engineer May 2010 t...\n",
                        "File: 22450718.pdf, Chunk ID: 2, Unique Chunk ID: 50_2, Chunk Length: 1155, Chunk Text: Science  Computer Engineering Technology Spring 20...\n",
                        "Processing file 52/120: 22776912.pdf\n",
                        "Number of chunks for file 22776912.pdf: 3\n",
                        "File: 22776912.pdf, Chunk ID: 0, Unique Chunk ID: 51_0, Chunk Length: 3144, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY CHIEF TECHNOLOG...\n",
                        "File: 22776912.pdf, Chunk ID: 1, Unique Chunk ID: 51_1, Chunk Length: 3245, Chunk Text: ruit train and mentor team members  Developed the ...\n",
                        "File: 22776912.pdf, Chunk ID: 2, Unique Chunk ID: 51_2, Chunk Length: 2720, Chunk Text: providing sharable information and maintaining dat...\n",
                        "Processing file 53/120: 23527321.pdf\n",
                        "Number of chunks for file 23527321.pdf: 2\n",
                        "File: 23527321.pdf, Chunk ID: 0, Unique Chunk ID: 52_0, Chunk Length: 3016, Chunk Text: HEAD INFORMATION TECHNOLOGY AND INFORMATION CENTER...\n",
                        "File: 23527321.pdf, Chunk ID: 1, Unique Chunk ID: 52_1, Chunk Length: 2543, Chunk Text: Overall responsibility for leadership development ...\n",
                        "Processing file 54/120: 23666211.pdf\n",
                        "Number of chunks for file 23666211.pdf: 2\n",
                        "File: 23666211.pdf, Chunk ID: 0, Unique Chunk ID: 53_0, Chunk Length: 3390, Chunk Text: TRAINING MANAGER Executive Summary Qualified Train...\n",
                        "File: 23666211.pdf, Chunk ID: 1, Unique Chunk ID: 53_1, Chunk Length: 2376, Chunk Text: effectively modernize inventory cataloging process...\n",
                        "Processing file 55/120: 23864648.pdf\n",
                        "Number of chunks for file 23864648.pdf: 2\n",
                        "File: 23864648.pdf, Chunk ID: 0, Unique Chunk ID: 54_0, Chunk Length: 3185, Chunk Text: VICE PRESIDENT INFORMATION TECHNOLOGY INFRASTRUCTU...\n",
                        "File: 23864648.pdf, Chunk ID: 1, Unique Chunk ID: 54_1, Chunk Length: 2575, Chunk Text: to 072006  Responsible for administering Microsoft...\n",
                        "Processing file 56/120: 24020470.pdf\n",
                        "Number of chunks for file 24020470.pdf: 3\n",
                        "File: 24020470.pdf, Chunk ID: 0, Unique Chunk ID: 55_0, Chunk Length: 2828, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary Over twe...\n",
                        "File: 24020470.pdf, Chunk ID: 1, Unique Chunk ID: 55_1, Chunk Length: 2762, Chunk Text: laser and dot matrix printers  Responsible to coor...\n",
                        "File: 24020470.pdf, Chunk ID: 2, Unique Chunk ID: 55_2, Chunk Length: 1969, Chunk Text: I always completed the sale by walking customer to...\n",
                        "Processing file 57/120: 24038620.pdf\n",
                        "Number of chunks for file 24038620.pdf: 3\n",
                        "File: 24038620.pdf, Chunk ID: 0, Unique Chunk ID: 56_0, Chunk Length: 2874, Chunk Text: INFORMATION TECHNOLOGY DIRECTOR Experience Informa...\n",
                        "File: 24038620.pdf, Chunk ID: 1, Unique Chunk ID: 56_1, Chunk Length: 3130, Chunk Text: digital dictation Dropbox Microsoft Office Mobile ...\n",
                        "File: 24038620.pdf, Chunk ID: 2, Unique Chunk ID: 56_2, Chunk Length: 99, Chunk Text: Scanner strategic planning Technical Training phon...\n",
                        "Processing file 58/120: 24083609.pdf\n",
                        "Number of chunks for file 24083609.pdf: 1\n",
                        "File: 24083609.pdf, Chunk ID: 0, Unique Chunk ID: 57_0, Chunk Length: 1816, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST INFOSEC Summary ...\n",
                        "Processing file 59/120: 24230851.pdf\n",
                        "Number of chunks for file 24230851.pdf: 2\n",
                        "File: 24230851.pdf, Chunk ID: 0, Unique Chunk ID: 58_0, Chunk Length: 3053, Chunk Text: CUSTOMER SERVICE REPRESENTATIVE Summary Recognized...\n",
                        "File: 24230851.pdf, Chunk ID: 1, Unique Chunk ID: 58_1, Chunk Length: 1650, Chunk Text: Confer with customers by telephone or in person to...\n",
                        "Processing file 60/120: 24889109.pdf\n",
                        "Number of chunks for file 24889109.pdf: 2\n",
                        "File: 24889109.pdf, Chunk ID: 0, Unique Chunk ID: 59_0, Chunk Length: 2821, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary Security...\n",
                        "File: 24889109.pdf, Chunk ID: 1, Unique Chunk ID: 59_1, Chunk Length: 1754, Chunk Text: Command USEUCOM and other Department of Defense ac...\n",
                        "Processing file 61/120: 24913648.pdf\n",
                        "Number of chunks for file 24913648.pdf: 3\n",
                        "File: 24913648.pdf, Chunk ID: 0, Unique Chunk ID: 60_0, Chunk Length: 3117, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Professional Sum...\n",
                        "File: 24913648.pdf, Chunk ID: 1, Unique Chunk ID: 60_1, Chunk Length: 2946, Chunk Text: expanded network systems and their components Comp...\n",
                        "File: 24913648.pdf, Chunk ID: 2, Unique Chunk ID: 60_2, Chunk Length: 777, Chunk Text: data processing Dell servers desktops DHCP documen...\n",
                        "Processing file 62/120: 25207620.pdf\n",
                        "Number of chunks for file 25207620.pdf: 3\n",
                        "File: 25207620.pdf, Chunk ID: 0, Unique Chunk ID: 61_0, Chunk Length: 3056, Chunk Text: INFORMATION TECHNOLOGY CERTIFIED TECHNICIAN Summar...\n",
                        "File: 25207620.pdf, Chunk ID: 1, Unique Chunk ID: 61_1, Chunk Length: 2772, Chunk Text: om It May Concern The intent of this letter is to ...\n",
                        "File: 25207620.pdf, Chunk ID: 2, Unique Chunk ID: 61_2, Chunk Length: 137, Chunk Text: Network Networking Novell SAP Software Documentati...\n",
                        "Processing file 63/120: 25857360.pdf\n",
                        "Number of chunks for file 25857360.pdf: 2\n",
                        "File: 25857360.pdf, Chunk ID: 0, Unique Chunk ID: 62_0, Chunk Length: 3224, Chunk Text: STAFF ASSISTANT Professional Summary Highly organi...\n",
                        "File: 25857360.pdf, Chunk ID: 1, Unique Chunk ID: 62_1, Chunk Length: 3014, Chunk Text: or office  Obtained signatures for financial docum...\n",
                        "Processing file 64/120: 25905275.pdf\n",
                        "Number of chunks for file 25905275.pdf: 3\n",
                        "File: 25905275.pdf, Chunk ID: 0, Unique Chunk ID: 63_0, Chunk Length: 3213, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Career Overview ...\n",
                        "File: 25905275.pdf, Chunk ID: 1, Unique Chunk ID: 63_1, Chunk Length: 3002, Chunk Text: 40 hrs per week Heidelberg Germany 40000 per year ...\n",
                        "File: 25905275.pdf, Chunk ID: 2, Unique Chunk ID: 63_2, Chunk Length: 919, Chunk Text: Disabled Veteran  Schedule A Hiring Authority Skil...\n",
                        "Processing file 65/120: 25959103.pdf\n",
                        "Number of chunks for file 25959103.pdf: 2\n",
                        "File: 25959103.pdf, Chunk ID: 0, Unique Chunk ID: 64_0, Chunk Length: 3251, Chunk Text: ADMINISTRATOR OF INFORMATION TECHNOLOGY Summary Ad...\n",
                        "File: 25959103.pdf, Chunk ID: 1, Unique Chunk ID: 64_1, Chunk Length: 1965, Chunk Text: support of Capitol Correspond SQL database  Provid...\n",
                        "Processing file 66/120: 25990239.pdf\n",
                        "Number of chunks for file 25990239.pdf: 2\n",
                        "File: 25990239.pdf, Chunk ID: 0, Unique Chunk ID: 65_0, Chunk Length: 2862, Chunk Text: INFORMATION TECHNOLOGY INSTRUCTOR Summary Seventee...\n",
                        "File: 25990239.pdf, Chunk ID: 1, Unique Chunk ID: 65_1, Chunk Length: 2189, Chunk Text: Provide remote control support for customers throu...\n",
                        "Processing file 67/120: 26480367.pdf\n",
                        "Number of chunks for file 26480367.pdf: 2\n",
                        "File: 26480367.pdf, Chunk ID: 0, Unique Chunk ID: 66_0, Chunk Length: 3074, Chunk Text: IT TECHNOLOGY SPECIALIST Professional Summary Anal...\n",
                        "File: 26480367.pdf, Chunk ID: 1, Unique Chunk ID: 66_1, Chunk Length: 1783, Chunk Text: to computer hardware and software issues  Identify...\n",
                        "Processing file 68/120: 26746496.pdf\n",
                        "Number of chunks for file 26746496.pdf: 3\n",
                        "File: 26746496.pdf, Chunk ID: 0, Unique Chunk ID: 67_0, Chunk Length: 2819, Chunk Text: DATABASE PROGRAMMERANALYST NET DEVELOPER Summary S...\n",
                        "File: 26746496.pdf, Chunk ID: 1, Unique Chunk ID: 67_1, Chunk Length: 2926, Chunk Text: for the classes that heshe enrolled and gives avai...\n",
                        "File: 26746496.pdf, Chunk ID: 2, Unique Chunk ID: 67_2, Chunk Length: 2281, Chunk Text: DAL Data Access Layer to interact with the Databas...\n",
                        "Processing file 69/120: 26768723.pdf\n",
                        "Number of chunks for file 26768723.pdf: 2\n",
                        "File: 26768723.pdf, Chunk ID: 0, Unique Chunk ID: 68_0, Chunk Length: 2961, Chunk Text: SUPPORT  NETWORK SERVICES INTERN Professional Summ...\n",
                        "File: 26768723.pdf, Chunk ID: 1, Unique Chunk ID: 68_1, Chunk Length: 1935, Chunk Text: room checks to test equipment and replace faulty c...\n",
                        "Processing file 70/120: 26801767.pdf\n",
                        "Number of chunks for file 26801767.pdf: 3\n",
                        "File: 26801767.pdf, Chunk ID: 0, Unique Chunk ID: 69_0, Chunk Length: 3229, Chunk Text: DIRECTOR INFORMATION TECHNOLOGY Professional Summa...\n",
                        "File: 26801767.pdf, Chunk ID: 1, Unique Chunk ID: 69_1, Chunk Length: 3272, Chunk Text: protected and secure against viruses outside intru...\n",
                        "File: 26801767.pdf, Chunk ID: 2, Unique Chunk ID: 69_2, Chunk Length: 2272, Chunk Text: that students were respected equally with faculty ...\n",
                        "Processing file 71/120: 27058381.pdf\n",
                        "Number of chunks for file 27058381.pdf: 5\n",
                        "File: 27058381.pdf, Chunk ID: 0, Unique Chunk ID: 70_0, Chunk Length: 1785, Chunk Text: SYSTEM ADMINISTRATOR Experience 032009 Company Nam...\n",
                        "File: 27058381.pdf, Chunk ID: 1, Unique Chunk ID: 70_1, Chunk Length: 2738, Chunk Text: ARTA FISMA Establishing Security Metrics 031406 18...\n",
                        "File: 27058381.pdf, Chunk ID: 2, Unique Chunk ID: 70_2, Chunk Length: 2783, Chunk Text: I also file out the Server Compliance checklist so...\n",
                        "File: 27058381.pdf, Chunk ID: 3, Unique Chunk ID: 70_3, Chunk Length: 2517, Chunk Text: analyses and determine where there programming err...\n",
                        "File: 27058381.pdf, Chunk ID: 4, Unique Chunk ID: 70_4, Chunk Length: 1348, Chunk Text: Project Management 121015 27225 SKSBS Troubleshoot...\n",
                        "Processing file 72/120: 27295996.pdf\n",
                        "Number of chunks for file 27295996.pdf: 3\n",
                        "File: 27295996.pdf, Chunk ID: 0, Unique Chunk ID: 71_0, Chunk Length: 3258, Chunk Text: IT DIRECTOR Accomplishments  CXA 2061  Citrix XenA...\n",
                        "File: 27295996.pdf, Chunk ID: 1, Unique Chunk ID: 71_1, Chunk Length: 3020, Chunk Text: Configure Manage Provide expertise and support for...\n",
                        "File: 27295996.pdf, Chunk ID: 2, Unique Chunk ID: 71_2, Chunk Length: 623, Chunk Text: 9 Zaporizhzhya Pedagogical College 14 City  Ukrain...\n",
                        "Processing file 73/120: 27372171.pdf\n",
                        "Number of chunks for file 27372171.pdf: 4\n",
                        "File: 27372171.pdf, Chunk ID: 0, Unique Chunk ID: 72_0, Chunk Length: 3234, Chunk Text: INFORMATION TECHNOLOGY SPECIALISTSYSTEM ANALYSIS S...\n",
                        "File: 27372171.pdf, Chunk ID: 1, Unique Chunk ID: 72_1, Chunk Length: 3217, Chunk Text: new or revised information security policy standar...\n",
                        "File: 27372171.pdf, Chunk ID: 2, Unique Chunk ID: 72_2, Chunk Length: 3201, Chunk Text: standard procedure to ship and package of electron...\n",
                        "File: 27372171.pdf, Chunk ID: 3, Unique Chunk ID: 72_3, Chunk Length: 708, Chunk Text: forms to be mailed to appropriate site  Plan and s...\n",
                        "Processing file 74/120: 27485716.pdf\n",
                        "Number of chunks for file 27485716.pdf: 2\n",
                        "File: 27485716.pdf, Chunk ID: 0, Unique Chunk ID: 73_0, Chunk Length: 3386, Chunk Text: CORPORATE PROJECT MANAGER Career Overview Seasoned...\n",
                        "File: 27485716.pdf, Chunk ID: 1, Unique Chunk ID: 73_1, Chunk Length: 2432, Chunk Text: 012008 to 012014 Director of Information Technolog...\n",
                        "Processing file 75/120: 27536013.pdf\n",
                        "Number of chunks for file 27536013.pdf: 2\n",
                        "File: 27536013.pdf, Chunk ID: 0, Unique Chunk ID: 74_0, Chunk Length: 3215, Chunk Text: EXPERIENCED INFORMATION TECHNOLOGY MANAGER Experie...\n",
                        "File: 27536013.pdf, Chunk ID: 1, Unique Chunk ID: 74_1, Chunk Length: 2497, Chunk Text: communicate project status and created and maintai...\n",
                        "Processing file 76/120: 27770859.pdf\n",
                        "Number of chunks for file 27770859.pdf: 2\n",
                        "File: 27770859.pdf, Chunk ID: 0, Unique Chunk ID: 75_0, Chunk Length: 3006, Chunk Text: INFORMATION TECHNOLOGY ADMINISTRATOR Professional ...\n",
                        "File: 27770859.pdf, Chunk ID: 1, Unique Chunk ID: 75_1, Chunk Length: 2222, Chunk Text: Wise to Microsoft Exchange Company Name City State...\n",
                        "Processing file 77/120: 28035460.pdf\n",
                        "Number of chunks for file 28035460.pdf: 3\n",
                        "File: 28035460.pdf, Chunk ID: 0, Unique Chunk ID: 76_0, Chunk Length: 2999, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Experience Infor...\n",
                        "File: 28035460.pdf, Chunk ID: 1, Unique Chunk ID: 76_1, Chunk Length: 3150, Chunk Text: cost reduction efforts for the production units  M...\n",
                        "File: 28035460.pdf, Chunk ID: 2, Unique Chunk ID: 76_2, Chunk Length: 4, Chunk Text: NULL...\n",
                        "Processing file 78/120: 28126340.pdf\n",
                        "Number of chunks for file 28126340.pdf: 3\n",
                        "File: 28126340.pdf, Chunk ID: 0, Unique Chunk ID: 77_0, Chunk Length: 2930, Chunk Text: INFORMATION TECHNOLOGY COORDINATOR Professional Pr...\n",
                        "File: 28126340.pdf, Chunk ID: 1, Unique Chunk ID: 77_1, Chunk Length: 2991, Chunk Text: the research and cost benefit analysis provided to...\n",
                        "File: 28126340.pdf, Chunk ID: 2, Unique Chunk ID: 77_2, Chunk Length: 2506, Chunk Text: requirements of the program instead of immediately...\n",
                        "Processing file 79/120: 28672970.pdf\n",
                        "Number of chunks for file 28672970.pdf: 3\n",
                        "File: 28672970.pdf, Chunk ID: 0, Unique Chunk ID: 78_0, Chunk Length: 3217, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Executive Profi...\n",
                        "File: 28672970.pdf, Chunk ID: 1, Unique Chunk ID: 78_1, Chunk Length: 3223, Chunk Text: for over 350 users  Provided strategic and tactica...\n",
                        "File: 28672970.pdf, Chunk ID: 2, Unique Chunk ID: 78_2, Chunk Length: 4, Chunk Text: NULL...\n",
                        "Processing file 80/120: 28697203.pdf\n",
                        "Number of chunks for file 28697203.pdf: 2\n",
                        "File: 28697203.pdf, Chunk ID: 0, Unique Chunk ID: 79_0, Chunk Length: 2993, Chunk Text: INFORMATION TECHNOLOGY PROJECT MANAGER Summary SUM...\n",
                        "File: 28697203.pdf, Chunk ID: 1, Unique Chunk ID: 79_1, Chunk Length: 2213, Chunk Text: and managed project risks defined opportunities fo...\n",
                        "Processing file 81/120: 28897981.pdf\n",
                        "Number of chunks for file 28897981.pdf: 3\n",
                        "File: 28897981.pdf, Chunk ID: 0, Unique Chunk ID: 80_0, Chunk Length: 3003, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST WEB GS11 Career ...\n",
                        "File: 28897981.pdf, Chunk ID: 1, Unique Chunk ID: 80_1, Chunk Length: 2875, Chunk Text: istered SharePoint sites  Provide group and oneono...\n",
                        "File: 28897981.pdf, Chunk ID: 2, Unique Chunk ID: 80_2, Chunk Length: 375, Chunk Text: 2016 New Horizons City State USA Bachelor of Scien...\n",
                        "Processing file 82/120: 29051656.pdf\n",
                        "Number of chunks for file 29051656.pdf: 3\n",
                        "File: 29051656.pdf, Chunk ID: 0, Unique Chunk ID: 81_0, Chunk Length: 2756, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary An organ...\n",
                        "File: 29051656.pdf, Chunk ID: 1, Unique Chunk ID: 81_1, Chunk Length: 2697, Chunk Text: and ACFS file systems  Migrated multiple standalon...\n",
                        "File: 29051656.pdf, Chunk ID: 2, Unique Chunk ID: 81_2, Chunk Length: 1272, Chunk Text: 10g on ASM file system  Planned and applied PSUCPU...\n",
                        "Processing file 83/120: 29075857.pdf\n",
                        "Number of chunks for file 29075857.pdf: 4\n",
                        "File: 29075857.pdf, Chunk ID: 0, Unique Chunk ID: 82_0, Chunk Length: 3105, Chunk Text: ADJUNCT INSTRUCTOR Skill Highlights Quality Engine...\n",
                        "File: 29075857.pdf, Chunk ID: 1, Unique Chunk ID: 82_1, Chunk Length: 3117, Chunk Text: by using approved PM tools including peer reviews ...\n",
                        "File: 29075857.pdf, Chunk ID: 2, Unique Chunk ID: 82_2, Chunk Length: 2804, Chunk Text: less within 12 months leveraging onshore and offsh...\n",
                        "File: 29075857.pdf, Chunk ID: 3, Unique Chunk ID: 82_3, Chunk Length: 26, Chunk Text: User Acceptance Visio Word...\n",
                        "Processing file 84/120: 29975124.pdf\n",
                        "Number of chunks for file 29975124.pdf: 2\n",
                        "File: 29975124.pdf, Chunk ID: 0, Unique Chunk ID: 83_0, Chunk Length: 2977, Chunk Text: CONSULTANT Experience Consultant  052017 to Curren...\n",
                        "File: 29975124.pdf, Chunk ID: 1, Unique Chunk ID: 83_1, Chunk Length: 2835, Chunk Text: against Oracle 8 database Information Technology C...\n",
                        "Processing file 85/120: 30223363.pdf\n",
                        "Number of chunks for file 30223363.pdf: 4\n",
                        "File: 30223363.pdf, Chunk ID: 0, Unique Chunk ID: 84_0, Chunk Length: 3180, Chunk Text: BUSINESS SYSTEMS ANALYSTI Qualifications TECHNICAL...\n",
                        "File: 30223363.pdf, Chunk ID: 1, Unique Chunk ID: 84_1, Chunk Length: 3566, Chunk Text: program specifications and business requirements P...\n",
                        "File: 30223363.pdf, Chunk ID: 2, Unique Chunk ID: 84_2, Chunk Length: 3342, Chunk Text: quality  Accountable for determining and requestin...\n",
                        "File: 30223363.pdf, Chunk ID: 3, Unique Chunk ID: 84_3, Chunk Length: 501, Chunk Text: User Access Writing Functional C Data Warehouse Fr...\n",
                        "Processing file 86/120: 31111279.pdf\n",
                        "Number of chunks for file 31111279.pdf: 3\n",
                        "File: 31111279.pdf, Chunk ID: 0, Unique Chunk ID: 85_0, Chunk Length: 3104, Chunk Text: MANAGER INFORMATION TECHNOLOGY PROJECT MANAGEMENT ...\n",
                        "File: 31111279.pdf, Chunk ID: 1, Unique Chunk ID: 85_1, Chunk Length: 2845, Chunk Text: influx of projects in the pipeline  Help PMO Train...\n",
                        "File: 31111279.pdf, Chunk ID: 2, Unique Chunk ID: 85_2, Chunk Length: 1960, Chunk Text: UNIX IDX Hospital Software package  Oversee all re...\n",
                        "Processing file 87/120: 31243710.pdf\n",
                        "Number of chunks for file 31243710.pdf: 3\n",
                        "File: 31243710.pdf, Chunk ID: 0, Unique Chunk ID: 86_0, Chunk Length: 2673, Chunk Text: IT MANAGER Summary Ten years of management experie...\n",
                        "File: 31243710.pdf, Chunk ID: 1, Unique Chunk ID: 86_1, Chunk Length: 2759, Chunk Text: 2012 HyperV installation and support Windows XP mo...\n",
                        "File: 31243710.pdf, Chunk ID: 2, Unique Chunk ID: 86_2, Chunk Length: 1834, Chunk Text: 2003  Supervised all Computer Information Systems ...\n",
                        "Processing file 88/120: 32959732.pdf\n",
                        "Number of chunks for file 32959732.pdf: 3\n",
                        "File: 32959732.pdf, Chunk ID: 0, Unique Chunk ID: 87_0, Chunk Length: 3215, Chunk Text: SENIOR DIRECTOR INFORMATION TECHNOLOGY Executive P...\n",
                        "File: 32959732.pdf, Chunk ID: 1, Unique Chunk ID: 87_1, Chunk Length: 2902, Chunk Text: support based on new policies and procedures I cre...\n",
                        "File: 32959732.pdf, Chunk ID: 2, Unique Chunk ID: 87_2, Chunk Length: 2475, Chunk Text: the complete Office suite and A Onedrive for indiv...\n",
                        "Processing file 89/120: 33241454.pdf\n",
                        "Number of chunks for file 33241454.pdf: 2\n",
                        "File: 33241454.pdf, Chunk ID: 0, Unique Chunk ID: 88_0, Chunk Length: 3090, Chunk Text: INFORMATION TECHNOLOGY SUPERVISOR Summary Seeking ...\n",
                        "File: 33241454.pdf, Chunk ID: 1, Unique Chunk ID: 88_1, Chunk Length: 861, Chunk Text: manual and automated data products  Disseminated t...\n",
                        "Processing file 90/120: 33381211.pdf\n",
                        "Number of chunks for file 33381211.pdf: 3\n",
                        "File: 33381211.pdf, Chunk ID: 0, Unique Chunk ID: 89_0, Chunk Length: 3038, Chunk Text: VICE PRESIDENT INFORMATION TECHNOLOGY  SOFTWARE EN...\n",
                        "File: 33381211.pdf, Chunk ID: 1, Unique Chunk ID: 89_1, Chunk Length: 2679, Chunk Text: and promote innovation in leading the change and a...\n",
                        "File: 33381211.pdf, Chunk ID: 2, Unique Chunk ID: 89_2, Chunk Length: 2862, Chunk Text: 2013 Belron Exceptional Service Award  Customer Se...\n",
                        "Processing file 91/120: 35325329.pdf\n",
                        "Number of chunks for file 35325329.pdf: 2\n",
                        "File: 35325329.pdf, Chunk ID: 0, Unique Chunk ID: 90_0, Chunk Length: 3287, Chunk Text: ASSISTANT INFORMATION TECHNOLOGY IT DIRECTOR Summa...\n",
                        "File: 35325329.pdf, Chunk ID: 1, Unique Chunk ID: 90_1, Chunk Length: 3151, Chunk Text: July 2005 to December 2006 Company Name City State...\n",
                        "Processing file 92/120: 36434348.pdf\n",
                        "Number of chunks for file 36434348.pdf: 2\n",
                        "File: 36434348.pdf, Chunk ID: 0, Unique Chunk ID: 91_0, Chunk Length: 3120, Chunk Text: INFORMATION TECHNOLOGY MANAGER Qualifications  Str...\n",
                        "File: 36434348.pdf, Chunk ID: 1, Unique Chunk ID: 91_1, Chunk Length: 2284, Chunk Text: resource usage of an internal service group to the...\n",
                        "Processing file 93/120: 36856210.pdf\n",
                        "Number of chunks for file 36856210.pdf: 2\n",
                        "File: 36856210.pdf, Chunk ID: 0, Unique Chunk ID: 92_0, Chunk Length: 3169, Chunk Text: INFORMATION TECHNOLOGY Summary Dedicated Informati...\n",
                        "File: 36856210.pdf, Chunk ID: 1, Unique Chunk ID: 92_1, Chunk Length: 1410, Chunk Text: 30 minutes as well as allow for more robust data t...\n",
                        "Processing file 94/120: 37242217.pdf\n",
                        "Number of chunks for file 37242217.pdf: 3\n",
                        "File: 37242217.pdf, Chunk ID: 0, Unique Chunk ID: 93_0, Chunk Length: 3216, Chunk Text: INFORMATION TECHNOLOGY CONSULTANT Career Overview ...\n",
                        "File: 37242217.pdf, Chunk ID: 1, Unique Chunk ID: 93_1, Chunk Length: 3366, Chunk Text: AdWords and social media Created and maintained pr...\n",
                        "File: 37242217.pdf, Chunk ID: 2, Unique Chunk ID: 93_2, Chunk Length: 394, Chunk Text: MySQL network networking Operating Systems OS OS 7...\n",
                        "Processing file 95/120: 37764298.pdf\n",
                        "Number of chunks for file 37764298.pdf: 3\n",
                        "File: 37764298.pdf, Chunk ID: 0, Unique Chunk ID: 94_0, Chunk Length: 2883, Chunk Text: INFORMATION TECHNOLOGY CONSULTANT Highlights Softw...\n",
                        "File: 37764298.pdf, Chunk ID: 1, Unique Chunk ID: 94_1, Chunk Length: 3097, Chunk Text: agencies and national accreditation agency ACCET A...\n",
                        "File: 37764298.pdf, Chunk ID: 2, Unique Chunk ID: 94_2, Chunk Length: 479, Chunk Text: PowerPoint MS Windows Windows 2000 2000 MS Windows...\n",
                        "Processing file 96/120: 38753827.pdf\n",
                        "Number of chunks for file 38753827.pdf: 4\n",
                        "File: 38753827.pdf, Chunk ID: 0, Unique Chunk ID: 95_0, Chunk Length: 3029, Chunk Text: VICE PRESIDENT INFORMATION TECHNOLOGY Executive Pr...\n",
                        "File: 38753827.pdf, Chunk ID: 1, Unique Chunk ID: 95_1, Chunk Length: 3060, Chunk Text: and reviews  Consistent record of delivering major...\n",
                        "File: 38753827.pdf, Chunk ID: 2, Unique Chunk ID: 95_2, Chunk Length: 2693, Chunk Text: September 1999 to October 2006 Company Name i14 Ci...\n",
                        "File: 38753827.pdf, Chunk ID: 3, Unique Chunk ID: 95_3, Chunk Length: 44, Chunk Text: Certified Administrator CCA Technical Skills...\n",
                        "Processing file 97/120: 39413067.pdf\n",
                        "Number of chunks for file 39413067.pdf: 2\n",
                        "File: 39413067.pdf, Chunk ID: 0, Unique Chunk ID: 96_0, Chunk Length: 3343, Chunk Text: FREELANCE IT CONSULTANT Career Overview Highly ski...\n",
                        "File: 39413067.pdf, Chunk ID: 1, Unique Chunk ID: 96_1, Chunk Length: 1892, Chunk Text: in an Active Directory environment Information Tec...\n",
                        "Processing file 98/120: 39718499.pdf\n",
                        "Number of chunks for file 39718499.pdf: 1\n",
                        "File: 39718499.pdf, Chunk ID: 0, Unique Chunk ID: 97_0, Chunk Length: 2286, Chunk Text: ASSISTANT FOOTBALL COACH Summary Enthusiastic reli...\n",
                        "Processing file 99/120: 40018190.pdf\n",
                        "Number of chunks for file 40018190.pdf: 2\n",
                        "File: 40018190.pdf, Chunk ID: 0, Unique Chunk ID: 98_0, Chunk Length: 2833, Chunk Text: IT SUPPORT TECHNICIAN Education Bachelor of Scienc...\n",
                        "File: 40018190.pdf, Chunk ID: 1, Unique Chunk ID: 98_1, Chunk Length: 87, Chunk Text: lete of the Year  Academic AllConference  IC3 Inte...\n",
                        "Processing file 100/120: 41344156.pdf\n",
                        "Number of chunks for file 41344156.pdf: 3\n",
                        "File: 41344156.pdf, Chunk ID: 0, Unique Chunk ID: 99_0, Chunk Length: 2918, Chunk Text: VP OF INFORMATION TECHNOLOGY Executive Profile A r...\n",
                        "File: 41344156.pdf, Chunk ID: 1, Unique Chunk ID: 99_1, Chunk Length: 3029, Chunk Text: Reporting to the CEO responsible for all informati...\n",
                        "File: 41344156.pdf, Chunk ID: 2, Unique Chunk ID: 99_2, Chunk Length: 361, Chunk Text: 072001 Company Name City State Deployed and mainta...\n",
                        "Processing file 101/120: 46260230.pdf\n",
                        "Number of chunks for file 46260230.pdf: 3\n",
                        "File: 46260230.pdf, Chunk ID: 0, Unique Chunk ID: 100_0, Chunk Length: 3136, Chunk Text: INFORMATION TECHNOLOGY SPECIALISTDISCOUNTPCFIX Sum...\n",
                        "File: 46260230.pdf, Chunk ID: 1, Unique Chunk ID: 100_1, Chunk Length: 3225, Chunk Text: equipment purchases  Updated or installed software...\n",
                        "File: 46260230.pdf, Chunk ID: 2, Unique Chunk ID: 100_2, Chunk Length: 2563, Chunk Text: defect prevention and continuous improvement  Used...\n",
                        "Processing file 102/120: 48037995.pdf\n",
                        "Number of chunks for file 48037995.pdf: 2\n",
                        "File: 48037995.pdf, Chunk ID: 0, Unique Chunk ID: 101_0, Chunk Length: 3161, Chunk Text: STAFF PHARMACIST Summary Patient and professional ...\n",
                        "File: 48037995.pdf, Chunk ID: 1, Unique Chunk ID: 101_1, Chunk Length: 2501, Chunk Text: Strictly maintained customer and patient confident...\n",
                        "Processing file 103/120: 51363762.pdf\n",
                        "Number of chunks for file 51363762.pdf: 2\n",
                        "File: 51363762.pdf, Chunk ID: 0, Unique Chunk ID: 102_0, Chunk Length: 2998, Chunk Text: AMC COMPUTER SPECIALIST AND INTERN Summary Informa...\n",
                        "File: 51363762.pdf, Chunk ID: 1, Unique Chunk ID: 102_1, Chunk Length: 2786, Chunk Text: aircraft under of the orders the PMEWRSTA PM Colon...\n",
                        "Processing file 104/120: 51639418.pdf\n",
                        "Number of chunks for file 51639418.pdf: 2\n",
                        "File: 51639418.pdf, Chunk ID: 0, Unique Chunk ID: 103_0, Chunk Length: 2832, Chunk Text: INFORMATION TECHNOLOGY BUREAU DEPUTY DIRECTOR Prof...\n",
                        "File: 51639418.pdf, Chunk ID: 1, Unique Chunk ID: 103_1, Chunk Length: 1946, Chunk Text: 2003 to August 2004 Information Technology Busines...\n",
                        "Processing file 105/120: 52246737.pdf\n",
                        "Number of chunks for file 52246737.pdf: 3\n",
                        "File: 52246737.pdf, Chunk ID: 0, Unique Chunk ID: 104_0, Chunk Length: 3039, Chunk Text: INFORMATION TECHNOLOGY PROVISIONING TECHNICIAN Car...\n",
                        "File: 52246737.pdf, Chunk ID: 1, Unique Chunk ID: 104_1, Chunk Length: 3289, Chunk Text: for director level and above Active Directory admi...\n",
                        "File: 52246737.pdf, Chunk ID: 2, Unique Chunk ID: 104_2, Chunk Length: 1144, Chunk Text: expenditure policies for the enterprise Administer...\n",
                        "Processing file 106/120: 52618188.pdf\n",
                        "Number of chunks for file 52618188.pdf: 3\n",
                        "File: 52618188.pdf, Chunk ID: 0, Unique Chunk ID: 105_0, Chunk Length: 3259, Chunk Text: INFORMATION TECHNOLOGY HELP DESK SPECIALIST Highli...\n",
                        "File: 52618188.pdf, Chunk ID: 1, Unique Chunk ID: 105_1, Chunk Length: 3290, Chunk Text: to successfully work and interact with all levels ...\n",
                        "File: 52618188.pdf, Chunk ID: 2, Unique Chunk ID: 105_2, Chunk Length: 862, Chunk Text: BBA  Business Administration 2015 Sam Houston Stat...\n",
                        "Processing file 107/120: 57002858.pdf\n",
                        "Number of chunks for file 57002858.pdf: 2\n",
                        "File: 57002858.pdf, Chunk ID: 0, Unique Chunk ID: 106_0, Chunk Length: 2900, Chunk Text: INFORMATION TECHNOLOGY MANAGER Summary Experienced...\n",
                        "File: 57002858.pdf, Chunk ID: 1, Unique Chunk ID: 106_1, Chunk Length: 2088, Chunk Text: additional employees and responsible for all issue...\n",
                        "Processing file 108/120: 64017585.pdf\n",
                        "Number of chunks for file 64017585.pdf: 1\n",
                        "File: 64017585.pdf, Chunk ID: 0, Unique Chunk ID: 107_0, Chunk Length: 2995, Chunk Text: INFORMATION TECHNOLOGY CONSULTANT MANAGING MEMBER ...\n",
                        "Processing file 109/120: 66832845.pdf\n",
                        "Number of chunks for file 66832845.pdf: 2\n",
                        "File: 66832845.pdf, Chunk ID: 0, Unique Chunk ID: 108_0, Chunk Length: 2866, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST I Professional S...\n",
                        "File: 66832845.pdf, Chunk ID: 1, Unique Chunk ID: 108_1, Chunk Length: 2064, Chunk Text: 5000 student user accounts create home folders and...\n",
                        "Processing file 110/120: 68460556.pdf\n",
                        "Number of chunks for file 68460556.pdf: 1\n",
                        "File: 68460556.pdf, Chunk ID: 0, Unique Chunk ID: 109_0, Chunk Length: 2350, Chunk Text: INFORMATION TECHNOLOGY INTERN Professional Profile...\n",
                        "Processing file 111/120: 70089206.pdf\n",
                        "Number of chunks for file 70089206.pdf: 3\n",
                        "File: 70089206.pdf, Chunk ID: 0, Unique Chunk ID: 110_0, Chunk Length: 3247, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Summary Highly s...\n",
                        "File: 70089206.pdf, Chunk ID: 1, Unique Chunk ID: 110_1, Chunk Length: 2980, Chunk Text: subject matter on project requirements  Conducted ...\n",
                        "File: 70089206.pdf, Chunk ID: 2, Unique Chunk ID: 110_2, Chunk Length: 369, Chunk Text: References  Objects and Modules Skills Configurati...\n",
                        "Processing file 112/120: 79541391.pdf\n",
                        "Number of chunks for file 79541391.pdf: 5\n",
                        "File: 79541391.pdf, Chunk ID: 0, Unique Chunk ID: 111_0, Chunk Length: 2868, Chunk Text: SUBJECT MATTER EXPERT INFORMATION TECHNOLOGY ASSIS...\n",
                        "File: 79541391.pdf, Chunk ID: 1, Unique Chunk ID: 111_1, Chunk Length: 2806, Chunk Text: va for Fixed Assets Equipment and Real Estate Item...\n",
                        "File: 79541391.pdf, Chunk ID: 2, Unique Chunk ID: 111_2, Chunk Length: 2597, Chunk Text: Supervisor post on P3 level was filled  Managed co...\n",
                        "File: 79541391.pdf, Chunk ID: 3, Unique Chunk ID: 111_3, Chunk Length: 2710, Chunk Text: Organized Wood furniture into appropriated stockro...\n",
                        "File: 79541391.pdf, Chunk ID: 4, Unique Chunk ID: 111_4, Chunk Length: 1451, Chunk Text: accurate and faster tracking of supplies  Increase...\n",
                        "Processing file 113/120: 81761658.pdf\n",
                        "Number of chunks for file 81761658.pdf: 3\n",
                        "File: 81761658.pdf, Chunk ID: 0, Unique Chunk ID: 112_0, Chunk Length: 3052, Chunk Text: IT MANAGER Highlights  Customer and Client Relatio...\n",
                        "File: 81761658.pdf, Chunk ID: 1, Unique Chunk ID: 112_1, Chunk Length: 3122, Chunk Text: personnel  Communicate with PresidentCEO on all te...\n",
                        "File: 81761658.pdf, Chunk ID: 2, Unique Chunk ID: 112_2, Chunk Length: 350, Chunk Text: 9 2000 NT migration Network Networking PACS person...\n",
                        "Processing file 114/120: 83816738.pdf\n",
                        "Number of chunks for file 83816738.pdf: 4\n",
                        "File: 83816738.pdf, Chunk ID: 0, Unique Chunk ID: 113_0, Chunk Length: 2989, Chunk Text: INFORMATION TECHNOLOGY INTERN TEST AUTOMATION ENGI...\n",
                        "File: 83816738.pdf, Chunk ID: 1, Unique Chunk ID: 113_1, Chunk Length: 2869, Chunk Text: aviordriven testing framework to allow Quality Ass...\n",
                        "File: 83816738.pdf, Chunk ID: 2, Unique Chunk ID: 113_2, Chunk Length: 2757, Chunk Text: and versioning sophisticated IDEs such as IntelliJ...\n",
                        "File: 83816738.pdf, Chunk ID: 3, Unique Chunk ID: 113_3, Chunk Length: 263, Chunk Text: in safety and security during Tech Exhibition 2013...\n",
                        "Processing file 115/120: 89413122.pdf\n",
                        "Number of chunks for file 89413122.pdf: 2\n",
                        "File: 89413122.pdf, Chunk ID: 0, Unique Chunk ID: 114_0, Chunk Length: 3326, Chunk Text: OPERATIONS RESEARCH ANALYST Summary Personable pro...\n",
                        "File: 89413122.pdf, Chunk ID: 1, Unique Chunk ID: 114_1, Chunk Length: 1321, Chunk Text: radiation detection data in an operational setting...\n",
                        "Processing file 116/120: 90867631.pdf\n",
                        "Number of chunks for file 90867631.pdf: 4\n",
                        "File: 90867631.pdf, Chunk ID: 0, Unique Chunk ID: 115_0, Chunk Length: 2439, Chunk Text: INFORMATION TECHNOLOGY SPECIALIST Career Overview ...\n",
                        "File: 90867631.pdf, Chunk ID: 1, Unique Chunk ID: 115_1, Chunk Length: 2853, Chunk Text: troubleshooting software and hardware issues of mi...\n",
                        "File: 90867631.pdf, Chunk ID: 2, Unique Chunk ID: 115_2, Chunk Length: 3003, Chunk Text: Developed and modifies databases  Performed databa...\n",
                        "File: 90867631.pdf, Chunk ID: 3, Unique Chunk ID: 115_3, Chunk Length: 2188, Chunk Text: Credits Earned 36 Semester hours Masters of Scienc...\n",
                        "Processing file 117/120: 91121135.pdf\n",
                        "Number of chunks for file 91121135.pdf: 3\n",
                        "File: 91121135.pdf, Chunk ID: 0, Unique Chunk ID: 116_0, Chunk Length: 3001, Chunk Text: ADMINISTRATIVE ASSISTANT DIRECTOR HUMAN RESOURCES ...\n",
                        "File: 91121135.pdf, Chunk ID: 1, Unique Chunk ID: 116_1, Chunk Length: 3023, Chunk Text: site organization chart  Processed invoices  Gener...\n",
                        "File: 91121135.pdf, Chunk ID: 2, Unique Chunk ID: 116_2, Chunk Length: 487, Chunk Text: Senior Management filing Forms Human Resources ins...\n",
                        "Processing file 118/120: 91635250.pdf\n",
                        "Number of chunks for file 91635250.pdf: 3\n",
                        "File: 91635250.pdf, Chunk ID: 0, Unique Chunk ID: 117_0, Chunk Length: 3355, Chunk Text: Christopher Townes Summary Knowledgeable Informati...\n",
                        "File: 91635250.pdf, Chunk ID: 1, Unique Chunk ID: 117_1, Chunk Length: 3095, Chunk Text: functional features with minimal defects  Authored...\n",
                        "File: 91635250.pdf, Chunk ID: 2, Unique Chunk ID: 117_2, Chunk Length: 871, Chunk Text: inmates provided receipts and inspected items for ...\n",
                        "Processing file 119/120: 91697974.pdf\n",
                        "Number of chunks for file 91697974.pdf: 2\n",
                        "File: 91697974.pdf, Chunk ID: 0, Unique Chunk ID: 118_0, Chunk Length: 3198, Chunk Text: INFORMATION TECHNOLOGY COORDINATOR Professional Su...\n",
                        "File: 91697974.pdf, Chunk ID: 1, Unique Chunk ID: 118_1, Chunk Length: 2360, Chunk Text: administrative personnel related activities monito...\n",
                        "Processing file 120/120: 92069209.pdf\n",
                        "Number of chunks for file 92069209.pdf: 2\n",
                        "File: 92069209.pdf, Chunk ID: 0, Unique Chunk ID: 119_0, Chunk Length: 3232, Chunk Text: DIRECTOR OF INFORMATION TECHNOLOGY Executive Profi...\n",
                        "File: 92069209.pdf, Chunk ID: 1, Unique Chunk ID: 119_1, Chunk Length: 2101, Chunk Text: needs for the City of Greensboro  Installed and co...\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>file_name</th>\n",
                            "      <th>chunk_id</th>\n",
                            "      <th>chunk_text</th>\n",
                            "      <th>unique_chunk_id</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>0</td>\n",
                            "      <td>INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...</td>\n",
                            "      <td>0_0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>1</td>\n",
                            "      <td>Disaster Recovery plan and procedures  Researc...</td>\n",
                            "      <td>0_1</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>2</td>\n",
                            "      <td>Installing configuring and supporting McAfee a...</td>\n",
                            "      <td>0_2</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "      file_name  chunk_id                                         chunk_text  \\\n",
                            "0  10089434.pdf         0  INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...   \n",
                            "1  10089434.pdf         1  Disaster Recovery plan and procedures  Researc...   \n",
                            "2  10089434.pdf         2  Installing configuring and supporting McAfee a...   \n",
                            "\n",
                            "  unique_chunk_id  \n",
                            "0             0_0  \n",
                            "1             0_1  \n",
                            "2             0_2  "
                        ]
                    },
                    "execution_count": 23,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "import os\n",
                "import re\n",
                "import pandas as pd\n",
                "import tiktoken\n",
                "\n",
                "# Path to the directory containing PDF files\n",
                "file_path = os.getenv(\"FILE_PATH\")\n",
                "folder_path = os.path.join(os.getcwd(), file_path)\n",
                "\n",
                "def get_pdf_files(folder_path):\n",
                "    for path, subdirs, files in os.walk(folder_path):\n",
                "        for name in files:\n",
                "            if name.endswith(\".pdf\"):\n",
                "                yield os.path.join(path, name)\n",
                "\n",
                "# Function to read PDF files and extract text using Azure AI Document Intelligence\n",
                "def extract_text_from_pdf(pdf_path):\n",
                "    with open(pdf_path, \"rb\") as f:\n",
                "        poller = document_analysis_client.begin_analyze_document(\"prebuilt-layout\", document=f)\n",
                "    result = poller.result()\n",
                "    text = \"\"\n",
                "    for page in result.pages:\n",
                "        for line in page.lines:\n",
                "            text += line.content + \" \"\n",
                "    return text\n",
                "\n",
                "# Function to clean text and remove special characters\n",
                "def clean_text(text):\n",
                "    text = re.sub(r'\\s+', ' ', text)  # Remove extra whitespace\n",
                "    text = re.sub(r'[^a-zA-Z0-9\\s]', '', text)  # Remove special characters\n",
                "    return text\n",
                "\n",
                "# Function to split text into chunks of 500 tokens\n",
                "def split_text_into_token_chunks(text, max_tokens=500):\n",
                "    tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
                "    tokens = tokenizer.encode(text)\n",
                "    chunks = []\n",
                "    \n",
                "    for i in range(0, len(tokens), max_tokens):\n",
                "        chunk_tokens = tokens[i:i + max_tokens]\n",
                "        chunk_text = tokenizer.decode(chunk_tokens)\n",
                "        chunks.append(chunk_text)\n",
                "    \n",
                "    return chunks\n",
                "\n",
                "# Count the number of PDF files in the directory\n",
                "pdf_files = [f for f in get_pdf_files(folder_path)]\n",
                "num_files = len(pdf_files)\n",
                "print(f\"Number of PDF files in the directory: {num_files}\")\n",
                "\n",
                "# Create a DataFrame to store the chunks\n",
                "data = []\n",
                "\n",
                "for file_id, pdf_file in enumerate(pdf_files):\n",
                "    file_name = os.path.basename(pdf_file)  # Ensure the correct file name is used\n",
                "    print(f\"Processing file {file_id + 1}/{num_files}: {file_name}\")\n",
                "    pdf_path = os.path.join(folder_path, pdf_file)\n",
                "    text = extract_text_from_pdf(pdf_path)\n",
                "    cleaned_text = clean_text(text)\n",
                "    chunks = split_text_into_token_chunks(cleaned_text)\n",
                "    \n",
                "    print(f\"Number of chunks for file {file_name}: {len(chunks)}\")\n",
                "    \n",
                "    for chunk_id, chunk in enumerate(chunks):\n",
                "        chunk_text = chunk.strip() if chunk.strip() else \"NULL\"\n",
                "        unique_chunk_id = f\"{file_id}_{chunk_id}\"\n",
                "        print(f\"File: {file_name}, Chunk ID: {chunk_id}, Unique Chunk ID: {unique_chunk_id}, Chunk Length: {len(chunk_text)}, Chunk Text: {chunk_text[:50]}...\")\n",
                "        data.append({\n",
                "            \"file_name\": file_name,  # Store the correct file name\n",
                "            \"chunk_id\": chunk_id,\n",
                "            \"chunk_text\": chunk_text,\n",
                "            \"unique_chunk_id\": unique_chunk_id\n",
                "        })\n",
                "\n",
                "df = pd.DataFrame(data)\n",
                "df.head(3)\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 24,
            "metadata": {
                "azdata_cell_guid": "3c0b7dba-c798-40d1-ac92-80288633497a",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>file_name</th>\n",
                            "      <th>chunk_id</th>\n",
                            "      <th>chunk_text</th>\n",
                            "      <th>unique_chunk_id</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>0</td>\n",
                            "      <td>INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...</td>\n",
                            "      <td>0_0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>1</td>\n",
                            "      <td>Disaster Recovery plan and procedures  Researc...</td>\n",
                            "      <td>0_1</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>2</td>\n",
                            "      <td>Installing configuring and supporting McAfee a...</td>\n",
                            "      <td>0_2</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>10247517.pdf</td>\n",
                            "      <td>0</td>\n",
                            "      <td>INFORMATION TECHNOLOGY MANAGER Professional Su...</td>\n",
                            "      <td>1_0</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>10247517.pdf</td>\n",
                            "      <td>1</td>\n",
                            "      <td>network which entailed changing software and L...</td>\n",
                            "      <td>1_1</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "      file_name  chunk_id                                         chunk_text  \\\n",
                            "0  10089434.pdf         0  INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...   \n",
                            "1  10089434.pdf         1  Disaster Recovery plan and procedures  Researc...   \n",
                            "2  10089434.pdf         2  Installing configuring and supporting McAfee a...   \n",
                            "3  10247517.pdf         0  INFORMATION TECHNOLOGY MANAGER Professional Su...   \n",
                            "4  10247517.pdf         1  network which entailed changing software and L...   \n",
                            "\n",
                            "  unique_chunk_id  \n",
                            "0             0_0  \n",
                            "1             0_1  \n",
                            "2             0_2  \n",
                            "3             1_0  \n",
                            "4             1_1  "
                        ]
                    },
                    "execution_count": 24,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "#read the top5 rows of the dataframe\n",
                "df.head(5)\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "655a8389-1018-4e80-a39d-85187cf3c46f",
                "language": "python"
            },
            "source": [
                "### **Tokenization vs. Character Length (OPTIONAL)**\n",
                "\n",
                "In this section, we will explore the difference between the character length of a text chunk and its tokenized representation. Character length simply counts the number of characters in a text, while tokenization breaks the text into meaningful units called tokens.\n",
                "\n",
                "Character Length First, let’s add a new column to our DataFrame to view the length of each chunk in terms of characters: Here, chunk\\_length represents the number of characters in each chunk."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 25,
            "metadata": {
                "azdata_cell_guid": "5e171928-764a-4d61-899a-83a8e0c3ac79",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "      file_name  chunk_id  chunk_length\n",
                        "0  10089434.pdf         0          3035\n",
                        "1  10089434.pdf         1          2959\n",
                        "2  10089434.pdf         2          2048\n",
                        "3  10247517.pdf         0          3191\n",
                        "4  10247517.pdf         1          2744\n"
                    ]
                }
            ],
            "source": [
                "# Add a new column 'chunk_length' to the DataFrame to view the length of each chunk\n",
                "df['chunk_length'] = df['chunk_text'].apply(len)\n",
                "\n",
                "# Display the first few rows of the DataFrame with the new column\n",
                "print(df[['file_name', 'chunk_id', 'chunk_length']].head(5))\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "9b87f84b-149c-4eb5-b9f0-c1b55f6d606c",
                "language": "python"
            },
            "source": [
                "### Tokenization\n",
                "To understand how text ultimately is tokenized, it can be helpful to run the below code: \n",
                "\n",
                "- We use the tiktoken library to tokenize the text. Tokenization breaks the text into smaller units, which can be words, subwords, or characters, depending on the tokenizer used. You can see that in some cases an entire word is represented with a single token whereas in others parts of words are split across multiple tokens. \n",
                "\n",
                "- If you then check the length of the decode variable, you'll find it matches 500 our specified token number. It is simply a way of making sure none of the data we pass to the model for tokenization and embedding exceeds the input token limit of 8,192\n",
                "\n",
                "- When we pass the documents to the embeddings model, it will break the documents into tokens similar (though not necessarily identical) to the examples below and then convert the tokens to a series of floating point numbers that will be accessible via vector search"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 26,
            "metadata": {
                "azdata_cell_guid": "28188280-2bef-414a-a663-a017c944bc19",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "[b'IN',\n",
                            " b'FORMATION',\n",
                            " b' TECHNO',\n",
                            " b'LOGY',\n",
                            " b' TECH',\n",
                            " b'NIC',\n",
                            " b'IAN',\n",
                            " b' I',\n",
                            " b' Summary',\n",
                            " b' Vers',\n",
                            " b'atile',\n",
                            " b' Systems',\n",
                            " b' Administrator',\n",
                            " b' possessing',\n",
                            " b' superior',\n",
                            " b' troubleshooting',\n",
                            " b' skills',\n",
                            " b' for',\n",
                            " b' networking',\n",
                            " b' issues',\n",
                            " b' end',\n",
                            " b' user',\n",
                            " b' problems',\n",
                            " b' and',\n",
                            " b' network',\n",
                            " b' security',\n",
                            " b' Experienced',\n",
                            " b' in',\n",
                            " b' server',\n",
                            " b' management',\n",
                            " b' systems',\n",
                            " b' analysis',\n",
                            " b' and',\n",
                            " b' offering',\n",
                            " b' inde',\n",
                            " b'pth',\n",
                            " b' understanding',\n",
                            " b' of',\n",
                            " b' IT',\n",
                            " b' infrastructure',\n",
                            " b' areas',\n",
                            " b' Detail',\n",
                            " b'oriented',\n",
                            " b' independent',\n",
                            " b' and',\n",
                            " b' focused',\n",
                            " b' on',\n",
                            " b' taking',\n",
                            " b' a',\n",
                            " b' systematic',\n",
                            " b' approach',\n",
                            " b' to',\n",
                            " b' solving',\n",
                            " b' complex',\n",
                            " b' problems',\n",
                            " b' Demonstr',\n",
                            " b'ated',\n",
                            " b' exceptional',\n",
                            " b' technical',\n",
                            " b' knowledge',\n",
                            " b' and',\n",
                            " b' skills',\n",
                            " b' while',\n",
                            " b' working',\n",
                            " b' with',\n",
                            " b' various',\n",
                            " b' teams',\n",
                            " b' to',\n",
                            " b' achieve',\n",
                            " b' shared',\n",
                            " b' goals',\n",
                            " b' and',\n",
                            " b' objectives',\n",
                            " b' Highlights',\n",
                            " b' ',\n",
                            " b' Active',\n",
                            " b' Directory',\n",
                            " b' ',\n",
                            " b' New',\n",
                            " b' technology',\n",
                            " b' and',\n",
                            " b' product',\n",
                            " b' research',\n",
                            " b' ',\n",
                            " b' Group',\n",
                            " b' Policy',\n",
                            " b' Objects',\n",
                            " b' ',\n",
                            " b' Office',\n",
                            " b' ',\n",
                            " b'365',\n",
                            " b' and',\n",
                            " b' Azure',\n",
                            " b' ',\n",
                            " b' PowerShell',\n",
                            " b' and',\n",
                            " b' VB',\n",
                            " b'Script',\n",
                            " b' ',\n",
                            " b' Storage',\n",
                            " b' management',\n",
                            " b' ',\n",
                            " b' Microsoft',\n",
                            " b' Exchange',\n",
                            " b' ',\n",
                            " b' Enterprise',\n",
                            " b' backup',\n",
                            " b' management',\n",
                            " b' ',\n",
                            " b' VM',\n",
                            " b'Ware',\n",
                            " b' experience',\n",
                            " b' ',\n",
                            " b' Disaster',\n",
                            " b' recovery',\n",
                            " b' Experience',\n",
                            " b' Information',\n",
                            " b' Technology',\n",
                            " b' Technician',\n",
                            " b' I',\n",
                            " b' Aug',\n",
                            " b' ',\n",
                            " b'200',\n",
                            " b'7',\n",
                            " b' to',\n",
                            " b' Current',\n",
                            " b' Company',\n",
                            " b' Name',\n",
                            " b' i',\n",
                            " b'14',\n",
                            " b' City',\n",
                            " b' State',\n",
                            " b' ',\n",
                            " b' M',\n",
                            " b'igr',\n",
                            " b'ating',\n",
                            " b' and',\n",
                            " b' managing',\n",
                            " b' user',\n",
                            " b' accounts',\n",
                            " b' in',\n",
                            " b' Microsoft',\n",
                            " b' Office',\n",
                            " b' ',\n",
                            " b'365',\n",
                            " b' and',\n",
                            " b' Exchange',\n",
                            " b' Online',\n",
                            " b' ',\n",
                            " b' Creating',\n",
                            " b' and',\n",
                            " b' managing',\n",
                            " b' virtual',\n",
                            " b' machines',\n",
                            " b' for',\n",
                            " b' systems',\n",
                            " b' such',\n",
                            " b' as',\n",
                            " b' domain',\n",
                            " b' controllers',\n",
                            " b' and',\n",
                            " b' Active',\n",
                            " b' Directory',\n",
                            " b' Federation',\n",
                            " b' Services',\n",
                            " b' A',\n",
                            " b'DFS',\n",
                            " b' in',\n",
                            " b' Microsoft',\n",
                            " b' Windows',\n",
                            " b' Azure',\n",
                            " b' I',\n",
                            " b'aaS',\n",
                            " b' ',\n",
                            " b' Creating',\n",
                            " b' and',\n",
                            " b' managing',\n",
                            " b' storage',\n",
                            " b' in',\n",
                            " b' Microsoft',\n",
                            " b' Windows',\n",
                            " b' Azure',\n",
                            " b' I',\n",
                            " b'aaS',\n",
                            " b' ',\n",
                            " b' Installing',\n",
                            " b' and',\n",
                            " b' configuring',\n",
                            " b' St',\n",
                            " b'or',\n",
                            " b'Simple',\n",
                            " b' i',\n",
                            " b'SC',\n",
                            " b'SI',\n",
                            " b' cloud',\n",
                            " b' array',\n",
                            " b' ST',\n",
                            " b'aa',\n",
                            " b'SB',\n",
                            " b'aaS',\n",
                            " b' ',\n",
                            " b' Installing',\n",
                            " b' configuring',\n",
                            " b' and',\n",
                            " b' testing',\n",
                            " b' Twin',\n",
                            " b'str',\n",
                            " b'ata',\n",
                            " b' i',\n",
                            " b'SC',\n",
                            " b'SI',\n",
                            " b' cloud',\n",
                            " b' array',\n",
                            " b' ST',\n",
                            " b'aa',\n",
                            " b'SB',\n",
                            " b'aaS',\n",
                            " b' ',\n",
                            " b' Collabor',\n",
                            " b'ating',\n",
                            " b' on',\n",
                            " b' project',\n",
                            " b' plan',\n",
                            " b' for',\n",
                            " b' Office',\n",
                            " b' ',\n",
                            " b'365',\n",
                            " b' migration',\n",
                            " b' ',\n",
                            " b' Developing',\n",
                            " b' detailed',\n",
                            " b' specifications',\n",
                            " b' for',\n",
                            " b' the',\n",
                            " b' Office',\n",
                            " b' ',\n",
                            " b'365',\n",
                            " b' migration',\n",
                            " b' including',\n",
                            " b' business',\n",
                            " b'case',\n",
                            " b' documentation',\n",
                            " b' cost',\n",
                            " b' benefit',\n",
                            " b' analyses',\n",
                            " b' technical',\n",
                            " b' diagrams',\n",
                            " b' and',\n",
                            " b' work',\n",
                            " b' flow',\n",
                            " b' documentation',\n",
                            " b' ',\n",
                            " b' Received',\n",
                            " b' training',\n",
                            " b' in',\n",
                            " b' MVC',\n",
                            " b' ',\n",
                            " b'4',\n",
                            " b' for',\n",
                            " b' Visual',\n",
                            " b' Studio',\n",
                            " b' using',\n",
                            " b' ',\n",
                            " b' Net',\n",
                            " b' Framework',\n",
                            " b' ',\n",
                            " b'445',\n",
                            " b' to',\n",
                            " b' develop',\n",
                            " b' application',\n",
                            " b' using',\n",
                            " b' HTML',\n",
                            " b'5',\n",
                            " b' and',\n",
                            " b' CSS',\n",
                            " b'3',\n",
                            " b' ',\n",
                            " b' Installing',\n",
                            " b' configuring',\n",
                            " b' and',\n",
                            " b' supporting',\n",
                            " b' Linux',\n",
                            " b' machines',\n",
                            " b' for',\n",
                            " b' the',\n",
                            " b' open',\n",
                            " b' WiFi',\n",
                            " b' network',\n",
                            " b' project',\n",
                            " b' ',\n",
                            " b' Comp',\n",
                            " b'iling',\n",
                            " b' and',\n",
                            " b' generating',\n",
                            " b' statistical',\n",
                            " b' information',\n",
                            " b' concerning',\n",
                            " b' wireless',\n",
                            " b' network',\n",
                            " b' traffic',\n",
                            " b' using',\n",
                            " b' C',\n",
                            " b'act',\n",
                            " b'i',\n",
                            " b' ',\n",
                            " b' Config',\n",
                            " b'uring',\n",
                            " b' wireless',\n",
                            " b' LAN',\n",
                            " b' router',\n",
                            " b' networking',\n",
                            " b' and',\n",
                            " b' security',\n",
                            " b' access',\n",
                            " b' ',\n",
                            " b' Installing',\n",
                            " b' and',\n",
                            " b' configuring',\n",
                            " b' wireless',\n",
                            " b' certificates',\n",
                            " b' ',\n",
                            " b' Developing',\n",
                            " b' detailed',\n",
                            " b' specifications',\n",
                            " b' for',\n",
                            " b' the',\n",
                            " b' acquisition',\n",
                            " b' of',\n",
                            " b' an',\n",
                            " b' Enterprise',\n",
                            " b' backup',\n",
                            " b' system',\n",
                            " b' including',\n",
                            " b' systems',\n",
                            " b' design',\n",
                            " b' business',\n",
                            " b'case',\n",
                            " b' documentation',\n",
                            " b' cost',\n",
                            " b' benefit',\n",
                            " b' analysis',\n",
                            " b' technical',\n",
                            " b' diagrams',\n",
                            " b' and',\n",
                            " b' work',\n",
                            " b' flow',\n",
                            " b' documentation',\n",
                            " b' ',\n",
                            " b' Review',\n",
                            " b'ing',\n",
                            " b' evaluating',\n",
                            " b' and',\n",
                            " b' analyzing',\n",
                            " b' department',\n",
                            " b'al',\n",
                            " b' policies',\n",
                            " b' guidelines',\n",
                            " b' procedures',\n",
                            " b' and',\n",
                            " b' standards',\n",
                            " b' with',\n",
                            " b' management',\n",
                            " b' and',\n",
                            " b' staff',\n",
                            " b' ',\n",
                            " b' Developing',\n",
                            " b' test',\n",
                            " b' scripts',\n",
                            " b' for',\n",
                            " b' acceptance',\n",
                            " b' unit',\n",
                            " b' and',\n",
                            " b' system',\n",
                            " b' testing',\n",
                            " b' of',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' Phase',\n",
                            " b' ',\n",
                            " b'1',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' Phase',\n",
                            " b' ',\n",
                            " b'2',\n",
                            " b' ',\n",
                            " b' Developing',\n",
                            " b' Quality',\n",
                            " b' Assurance',\n",
                            " b' and',\n",
                            " b' testing',\n",
                            " b' plan',\n",
                            " b' for',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' Phase',\n",
                            " b' ',\n",
                            " b'1',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' Phase',\n",
                            " b' ',\n",
                            " b'2',\n",
                            " b' ',\n",
                            " b' Debug',\n",
                            " b'ging',\n",
                            " b' and',\n",
                            " b' logging',\n",
                            " b' of',\n",
                            " b' errors',\n",
                            " b' in',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' using',\n",
                            " b' Team',\n",
                            " b' Foundation',\n",
                            " b' Server',\n",
                            " b' T',\n",
                            " b'FS',\n",
                            " b' ',\n",
                            " b' Particip',\n",
                            " b'ated',\n",
                            " b' in',\n",
                            " b' various',\n",
                            " b' phases',\n",
                            " b' of',\n",
                            " b' the',\n",
                            " b' project',\n",
                            " b' life',\n",
                            " b' cycle',\n",
                            " b' such',\n",
                            " b' as',\n",
                            " b' determining',\n",
                            " b' requirements',\n",
                            " b' design',\n",
                            " b' conceptual',\n",
                            " b'ization',\n",
                            " b' testing',\n",
                            " b' implementation',\n",
                            " b' deployment',\n",
                            " b' and',\n",
                            " b' release',\n",
                            " b' for',\n",
                            " b' the',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' projects',\n",
                            " b' ',\n",
                            " b' Collabor',\n",
                            " b'ating',\n",
                            " b' on',\n",
                            " b' project',\n",
                            " b' plans',\n",
                            " b' for',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' ',\n",
                            " b' Pre',\n",
                            " b'paring',\n",
                            " b' presentations',\n",
                            " b' and',\n",
                            " b' documentation',\n",
                            " b' to',\n",
                            " b' demonstrate',\n",
                            " b' Hyper',\n",
                            " b'ion',\n",
                            " b' and',\n",
                            " b' Miami',\n",
                            " b'Biz',\n",
                            " b' functionality',\n",
                            " b' or',\n",
                            " b' design',\n",
                            " b' ',\n",
                            " b' Monitoring',\n",
                            " b' network',\n",
                            " b' traffic',\n",
                            " b' and',\n",
                            " b' compiling',\n",
                            " b' and',\n",
                            " b' generating',\n",
                            " b' statistical',\n",
                            " b' information',\n",
                            " b' using',\n",
                            " b' Solar',\n",
                            " b' Winds',\n",
                            " b' ',\n",
                            " b' Collabor',\n",
                            " b'ating',\n",
                            " b' on']"
                        ]
                    },
                    "execution_count": 26,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "import tiktoken\n",
                "tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
                "sample_encode = tokenizer.encode(df.chunk_text[0]) \n",
                "decode = tokenizer.decode_tokens_bytes(sample_encode)\n",
                "decode\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {
                "azdata_cell_guid": "31d80228-e140-413a-b1e3-c320a42b1f6c",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "500"
                        ]
                    },
                    "execution_count": 11,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "len(decode)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "ac19ee49-763a-4e2e-8c31-9ce1ba77bbe4"
            },
            "source": [
                "# **PART 2 : Generating Embeddings for Text Chunks using Azure Open AI**\n",
                "\n",
                "- After extracting and chunking the text from PDF resumes, we will generate embeddings for each chunk. These embeddings are numerical representations of the text that capture its semantic meaning. By creating embeddings for the text chunks, we can perform advanced similarity searches and enhance language model generation.\n",
                "\n",
                "- We will use the NVIDIA AI Models to generate these embeddings. The `get_embedding` function defined below takes a piece of text as input and returns its embedding using the `nvidia/embed-qa-4` model\n",
                "\n",
                "- Ensure the Environment Variables are set correctly in the .env file"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 27,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>file_name</th>\n",
                            "      <th>chunk_id</th>\n",
                            "      <th>chunk_text</th>\n",
                            "      <th>unique_chunk_id</th>\n",
                            "      <th>chunk_length</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>0</td>\n",
                            "      <td>INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...</td>\n",
                            "      <td>0_0</td>\n",
                            "      <td>3035</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>1</td>\n",
                            "      <td>Disaster Recovery plan and procedures  Researc...</td>\n",
                            "      <td>0_1</td>\n",
                            "      <td>2959</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>10089434.pdf</td>\n",
                            "      <td>2</td>\n",
                            "      <td>Installing configuring and supporting McAfee a...</td>\n",
                            "      <td>0_2</td>\n",
                            "      <td>2048</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "      file_name  chunk_id                                         chunk_text  \\\n",
                            "0  10089434.pdf         0  INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...   \n",
                            "1  10089434.pdf         1  Disaster Recovery plan and procedures  Researc...   \n",
                            "2  10089434.pdf         2  Installing configuring and supporting McAfee a...   \n",
                            "\n",
                            "  unique_chunk_id  chunk_length  \n",
                            "0             0_0          3035  \n",
                            "1             0_1          2959  \n",
                            "2             0_2          2048  "
                        ]
                    },
                    "execution_count": 27,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "df.head(3)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 28,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "        file_name  chunk_id  \\\n",
                        "0    10089434.pdf         0   \n",
                        "1    10089434.pdf         1   \n",
                        "2    10089434.pdf         2   \n",
                        "3    10247517.pdf         0   \n",
                        "4    10247517.pdf         1   \n",
                        "..            ...       ...   \n",
                        "309  91635250.pdf         2   \n",
                        "310  91697974.pdf         0   \n",
                        "311  91697974.pdf         1   \n",
                        "312  92069209.pdf         0   \n",
                        "313  92069209.pdf         1   \n",
                        "\n",
                        "                                            chunk_text unique_chunk_id  \\\n",
                        "0    INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...             0_0   \n",
                        "1    Disaster Recovery plan and procedures  Researc...             0_1   \n",
                        "2    Installing configuring and supporting McAfee a...             0_2   \n",
                        "3    INFORMATION TECHNOLOGY MANAGER Professional Su...             1_0   \n",
                        "4    network which entailed changing software and L...             1_1   \n",
                        "..                                                 ...             ...   \n",
                        "309  inmates provided receipts and inspected items ...           117_2   \n",
                        "310  INFORMATION TECHNOLOGY COORDINATOR Professiona...           118_0   \n",
                        "311  administrative personnel related activities mo...           118_1   \n",
                        "312  DIRECTOR OF INFORMATION TECHNOLOGY Executive P...           119_0   \n",
                        "313  needs for the City of Greensboro  Installed an...           119_1   \n",
                        "\n",
                        "     chunk_length  \n",
                        "0            3035  \n",
                        "1            2959  \n",
                        "2            2048  \n",
                        "3            3191  \n",
                        "4            2744  \n",
                        "..            ...  \n",
                        "309           871  \n",
                        "310          3198  \n",
                        "311          2360  \n",
                        "312          3232  \n",
                        "313          2101  \n",
                        "\n",
                        "[314 rows x 5 columns]\n"
                    ]
                }
            ],
            "source": [
                "print(df)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 29,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Completed 50 rows\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 519 exceeds maximum allowed token size 512'}\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 523 exceeds maximum allowed token size 512'}\n",
                        "Completed 100 rows\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 518 exceeds maximum allowed token size 512'}\n",
                        "Completed 150 rows\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 520 exceeds maximum allowed token size 512'}\n",
                        "Completed 200 rows\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 515 exceeds maximum allowed token size 512'}\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 513 exceeds maximum allowed token size 512'}\n",
                        "Completed 250 rows\n",
                        "Error generating embedding: Error code: 400 - {'type': 'urn:inference-service:problem-details:bad-request', 'title': 'Bad Request', 'status': 400, 'detail': 'Input length 540 exceeds maximum allowed token size 512'}\n",
                        "Completed 300 rows\n",
                        "       filename chunkid                                              chunk  \\\n",
                        "0  10089434.pdf     0_0  INFORMATION TECHNOLOGY TECHNICIAN I Summary Ve...   \n",
                        "1  10089434.pdf     0_1  Disaster Recovery plan and procedures  Researc...   \n",
                        "2  10089434.pdf     0_2  Installing configuring and supporting McAfee a...   \n",
                        "3  10247517.pdf     1_0  INFORMATION TECHNOLOGY MANAGER Professional Su...   \n",
                        "4  10247517.pdf     1_1  network which entailed changing software and L...   \n",
                        "\n",
                        "                                           embedding  \n",
                        "0  [0.0008854866027832031, -0.0499267578125, 0.02...  \n",
                        "1  [0.00724029541015625, -0.04937744140625, -0.00...  \n",
                        "2  [0.0014591217041015625, -0.06402587890625, 0.0...  \n",
                        "3  [0.0125579833984375, -0.045562744140625, 0.002...  \n",
                        "4  [0.0175628662109375, -0.04925537109375, -0.012...  \n"
                    ]
                }
            ],
            "source": [
                "import os\n",
                "import pandas as pd\n",
                "import json\n",
                "from openai import OpenAI\n",
                "\n",
                "# Set NVIDIA API credentials\n",
                "nvidia_api_key = os.getenv(\"NVIDIA_API_KEY\")  # Ensure this is set\n",
                "nvidia_base_url = \"https://integrate.api.nvidia.com/v1\"\n",
                "\n",
                "# Initialize OpenAI client for NVIDIA API\n",
                "client = OpenAI(\n",
                "    api_key=nvidia_api_key,\n",
                "    base_url=nvidia_base_url\n",
                ")\n",
                "\n",
                "def get_embedding(text):\n",
                "    \"\"\"\n",
                "    Get sentence embedding using NVIDIA's embed-qa-4 model.\n",
                "\n",
                "    Args:\n",
                "        text (str): Text to embed.\n",
                "\n",
                "    Returns:\n",
                "        list: A list containing the embedding.\n",
                "    \"\"\"\n",
                "    try:\n",
                "        response = client.embeddings.create(\n",
                "            input=[text],\n",
                "            model=\"nvidia/embed-qa-4\",\n",
                "            encoding_format=\"float\",\n",
                "            extra_body={\"input_type\": \"query\", \"truncate\": \"NONE\"}\n",
                "        )\n",
                "        return response.data[0].embedding\n",
                "    except Exception as e:\n",
                "        print(f\"Error generating embedding: {e}\")\n",
                "        return None\n",
                "\n",
                "# Example usage\n",
                "all_filenames = []\n",
                "all_chunkids = []\n",
                "all_chunks = []\n",
                "all_embeddings = []\n",
                "\n",
                "# Assuming df is already defined with the required columns\n",
                "for index, row in df.iterrows():\n",
                "    filename = row['file_name']\n",
                "    chunkid = row['unique_chunk_id']\n",
                "    chunk = row['chunk_text']\n",
                "    \n",
                "    embedding = get_embedding(chunk)\n",
                "    \n",
                "    if embedding is not None:\n",
                "        all_filenames.append(filename)\n",
                "        all_chunkids.append(chunkid)\n",
                "        all_chunks.append(chunk)\n",
                "        all_embeddings.append(embedding)\n",
                "    \n",
                "    if (index + 1) % 50 == 0:  # Print progress every 50 rows\n",
                "        print(f\"Completed {index + 1} rows\")\n",
                "\n",
                "# Create a new DataFrame with the results\n",
                "result_df = pd.DataFrame({\n",
                "    'filename': all_filenames,\n",
                "    'chunkid': all_chunkids,\n",
                "    'chunk': all_chunks,\n",
                "    'embedding': all_embeddings\n",
                "})\n",
                "\n",
                "print(result_df.head())  # Display the first few rows\n",
                "\n",
                "# Save to CSV if needed\n",
                "result_df.to_csv(\"embedded_results_nvidia.csv\", index=False)\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "04d42351-41ec-4ce9-9778-fe4ea2ecb8a2"
            },
            "source": [
                "# **PART 3 : Using Azure SQL DB as a Vector Database to store and query embeddings**\n",
                "\n",
                "### **Load the embeddings into the Vector Database : Azure SQL DB**\n",
                "\n",
                "First let us define a function to connect to Azure SQLDB"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 30,
            "metadata": {
                "azdata_cell_guid": "930a63bc-4c08-4205-b152-b1ad5c82057a",
                "language": "python"
            },
            "outputs": [],
            "source": [
                "#lets define a function to connect to SQLDB\n",
                "import os\n",
                "from dotenv import load_dotenv\n",
                "import pyodbc\n",
                "import struct\n",
                "from azure.identity import DefaultAzureCredential\n",
                "\n",
                "# Load environment variables from .env file\n",
                "load_dotenv()\n",
                "\n",
                "def get_mssql_connection():\n",
                "    # Retrieve the connection string from the environment variables\n",
                "    entra_connection_string = os.getenv('ENTRA_CONNECTION_STRING')\n",
                "\n",
                "    sql_connection_string = os.getenv('SQL_CONNECTION_STRING')\n",
                "\n",
                "    # Determine the authentication method and connect to the database\n",
                "    if entra_connection_string:\n",
                "        # Entra ID Service Principal Authentication\n",
                "        credential = DefaultAzureCredential(exclude_interactive_browser_credential=False)    \n",
                "        token = credential.get_token('https://database.windows.net/.default')\n",
                "        token_bytes = token.token.encode('UTF-16LE')\n",
                "        token_struct = struct.pack(f'<I{len(token_bytes)}s', len(token_bytes), token_bytes)\n",
                "        SQL_COPT_SS_ACCESS_TOKEN = 1256  # This connection option is defined by Microsoft in msodbcsql.h\n",
                "        conn = pyodbc.connect(entra_connection_string, attrs_before={SQL_COPT_SS_ACCESS_TOKEN: token_struct})\n",
                "    \n",
                "    elif sql_connection_string:\n",
                "        # SQL Authentication\n",
                "        conn = pyodbc.connect(sql_connection_string)\n",
                "        \n",
                "    else:\n",
                "        raise ValueError(\"No valid connection string found in the environment variables.\")\n",
                "\n",
                "    return conn\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "e929c112-65d4-46f1-a9a2-7c9434b2d7cb",
                "language": "python"
            },
            "source": [
                "### **Insert embeddings into the native 'Vector' Data Type**\n",
                "\n",
                "We will insert our vectors into the SQL Table now. Azure SQL DB now has a dedicated, native, data type for storing vectors: the `vector` data type. Read about the preview [here](https://devblogs.microsoft.com/azure-sql/eap-for-vector-support-refresh-introducing-vector-type)\n",
                "\n",
                "The table embeddings has a column called vector which is vector(1536) type. Ensure you have created the table using the script `CreateTable.sql` before running the below code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 31,
            "metadata": {
                "azdata_cell_guid": "680259d9-77ce-4b63-b412-9bea33bb0f43",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Data inserted successfully into the 'resumedocs' table.\n"
                    ]
                }
            ],
            "source": [
                "import pyodbc\n",
                "import pandas as pd\n",
                "\n",
                "# Retrieve the connection string from the function get_mssql_connection()\n",
                "conn = get_mssql_connection()\n",
                "\n",
                "# Create a cursor object\n",
                "cursor = conn.cursor()\n",
                "\n",
                "# Enable fast_executemany\n",
                "cursor.fast_executemany = True\n",
                "\n",
                "# Loop through the DataFrame rows and insert them into the table\n",
                "for index, row in result_df.iterrows():\n",
                "    chunkid = row['chunkid']\n",
                "    filename = row['filename']\n",
                "    chunk = row['chunk']\n",
                "    embedding = row['embedding']\n",
                "    \n",
                "    # Use placeholders for the parameters in the SQL query\n",
                "    query = f\"\"\"\n",
                "    INSERT INTO resumedocs (chunkid, filename, chunk, embedding)\n",
                "    VALUES (?, ?, ?, CAST(? AS VECTOR(1024)))\n",
                "    \"\"\"\n",
                "    # Execute the query with the parameters\n",
                "    cursor.execute(query, chunkid, filename, chunk, json.dumps(embedding))\n",
                "\n",
                "# Commit the changes\n",
                "conn.commit()\n",
                "\n",
                "# Print a success message\n",
                "print(\"Data inserted successfully into the 'resumedocs' table.\")\n",
                "\n",
                "# Close the connection\n",
                "conn.close()\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "ede638b9-d681-4cb9-9ab8-9651e3099a36",
                "language": "python"
            },
            "source": [
                "Let's take a look at the data in the Resume Docs table:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 32,
            "metadata": {
                "azdata_cell_guid": "88d1a90e-e93c-426e-934d-fb1a47dfd900",
                "language": "python"
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "+--------------+---------+----------------------+----------------------+\n",
                        "|   filename   | chunkid |        chunk         |      embedding       |\n",
                        "+--------------+---------+----------------------+----------------------+\n",
                        "| 10089434.pdf |   0_0   | INFORMATION TECHNOLO | [8.8548660e-004,-4.9 |\n",
                        "| 10089434.pdf |   0_1   | Disaster Recovery pl | [7.2402954e-003,-4.9 |\n",
                        "| 10089434.pdf |   0_2   | Installing configuri | [1.4591217e-003,-6.4 |\n",
                        "| 10247517.pdf |   1_0   | INFORMATION TECHNOLO | [1.2557983e-002,-4.5 |\n",
                        "| 10247517.pdf |   1_1   | network which entail | [1.7562866e-002,-4.9 |\n",
                        "| 10247517.pdf |   1_2   | i4 City State 2015 M | [2.1152496e-003,-4.9 |\n",
                        "| 10265057.pdf |   2_0   | WORKING RF SYSTEMS E | [-6.2789917e-003,-4. |\n",
                        "| 10265057.pdf |   2_1   | surveys ElectricalVa | [6.9570541e-004,-2.8 |\n",
                        "| 10553553.pdf |   3_0   | INFORMATION TECHNOLO | [-1.4390945e-003,-4. |\n",
                        "| 10553553.pdf |   3_1   | XP Vista and Mac ope | [2.0690918e-002,-5.7 |\n",
                        "+--------------+---------+----------------------+----------------------+\n"
                    ]
                }
            ],
            "source": [
                "from prettytable import PrettyTable\n",
                "\n",
                "import pyodbc\n",
                "import pandas as pd\n",
                "\n",
                "# Load environment variables from .env file\n",
                "load_dotenv()\n",
                "\n",
                "# Retrieve the connection string from the environment variables\n",
                "conn = get_mssql_connection()\n",
                "\n",
                "# Create a cursor object\n",
                "cursor = conn.cursor()\n",
                "\n",
                "# Use placeholders for the parameters in the SQL query\n",
                "query = \"SELECT TOP(10) filename, chunkid, chunk, CAST(embedding AS NVARCHAR(MAX)) as embedding FROM dbo.resumedocs ORDER BY Id\"\n",
                "\n",
                "# Execute the query with the parameters\n",
                "cursor.execute(query)\n",
                "queryresults = cursor.fetchall()\n",
                "\n",
                "# Get column names from cursor.description\n",
                "column_names = [column[0] for column in cursor.description]\n",
                "\n",
                "# Create a PrettyTable object\n",
                "table = PrettyTable()\n",
                "\n",
                "# Add column names to the table\n",
                "table.field_names = column_names\n",
                "\n",
                "# Set max width for each column to truncate data\n",
                "table.max_width = 20\n",
                "\n",
                "# Add rows to the table\n",
                "for row in queryresults:\n",
                "    # Truncate each value to 20 characters\n",
                "    truncated_row = [str(value)[:20] for value in row]\n",
                "    table.add_row(truncated_row)\n",
                "\n",
                "# Print the table\n",
                "print(table)\n",
                "\n",
                "# Commit the changes\n",
                "conn.commit()\n",
                "# Close the connection\n",
                "conn.close()\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "b7b0fda6-3322-4bbc-8d08-d0b567da79ec",
                "language": "python"
            },
            "source": [
                "### **Performing Vector Similarity Search in Azure SQL DB using VECTOR\\_DISTANCE built in function**\n",
                "\n",
                "Let's now query our ResumeDocs table to get the top similar candidates given the User search query.\n",
                "\n",
                "What we are doing: Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user’s query. This helps in finding the most suitable candidates based on their resumes.\n",
                "\n",
                "The most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the new distance functions.\n",
                "\n",
                "```\n",
                "VECTOR_DISTANCE('distance metric', V1, V2)\n",
                "\n",
                "```\n",
                "\n",
                "We can use **cosine**, **euclidean**, and **dot** as the distance metric today.\n",
                "\n",
                "We will define the function `vector_search_sql`."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 33,
            "metadata": {
                "azdata_cell_guid": "1b4f0ca2-2401-4f90-a44d-75dce03500cc",
                "language": "python"
            },
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "[('38753827.pdf', '95_3', 'Certified Administrator CCA Technical Skills', 0.6364656039761232, 0.3635343960238768),\n",
                            " ('20237244.pdf', '43_1', 'Digital Media Collector', 0.6110346951661374, 0.3889653048338626),\n",
                            " ('29051656.pdf', '81_1', 'and ACFS file systems  Migrated multiple standalone databases to RAC databases using Rconfig RMAN and OEM  Migrated single instance databases from NONASM to ASM file system to improve performance  Installed Oracle 10g 11g and 12c software and created multiple databases including Plugable and container databases in oracle 12c  Implemented TDE Data Redaction and database Auditing to improved data integrity and security  Applied PSU on standalone TEST environment using OPatch and same PSU Grid and RDBMS Patching on production cluster using OPatch Auto  Replicated data in real time using Oracle Golden Gate and Oracle streams  Analyzed interpreted and troubleshot Golden Gate related issues  Designed and implemented different backup strategies like Cold Hot backup using RMAN with Flash Recovery Area and Logical Backups with EXPDPIMPDP  Scheduled RMAN backups purge jobs Maintenance Jobs using DBMSJOBS DBMSSCHEDULER Crontab and OEM  Implemented and managed logical backuprecovery with Datapump and ExportImport utilities  Cloned databases using RMAN and Manually using scripts  Implemented Point In Time Recovery on one or more tablespaces to recover lost of a tables while eliminating down time for the database  Installed SQL Server EE 2012 on windows and created databases triggers tables procedures functions and database diagrams  Upgraded databases from 10g to 11g and to 12c using DBUA Datapump Manually Transportable Tablespace and Oracle Dataguard Oracle Database Administrator 072012 to 102014 Company Name City  State  Prepared technical architecture proposals for enhancements and migrated an existing standalone database to RAC database  Provided senior technical support to Developers and troubleshot performance issues and maintained high availability and security of databases  Analyzed and tuned the Database to identify potential database bottle necks such as response delay locking contention wait event using tools like STATSPACK EXPLAIN PLAN and TKPROF  Collaborated and worked together with development and operations staff and resolved problems quickly and efficiently  Maintained Physical Logical Active standby databases supported by Data guard on Oracle 11g and 10g RAC servers for the purpose of disaster recovery procedures  Installed configured deployed and monitored different databases and application servers using Oracle OEM 12c Grid Control  Worked efficiently in Database administration activities such as User management Space management Monitoring Creating Database Managing Oracle Instance and Database security and Materialized views  Maintained and administered high availability solutions such as Real Application Cluster RAC in Oracle 11gR2 Grid and', 0.5997577835851646, 0.4002422164148354)]"
                        ]
                    },
                    "execution_count": 33,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "import os\n",
                "import pyodbc\n",
                "import json\n",
                "from dotenv import load_dotenv\n",
                "\n",
                "def vector_search_sql(query, num_results=5):\n",
                "    # Load environment variables from .env file\n",
                "    load_dotenv()\n",
                "\n",
                "    # Use the get_mssql_connection function to get the connection string details\n",
                "    conn = get_mssql_connection()\n",
                "\n",
                "    # Create a cursor object\n",
                "    cursor = conn.cursor()\n",
                "\n",
                "    # Generate the query embedding for the user's search query\n",
                "    user_query_embedding = get_embedding(query)\n",
                "    \n",
                "    # SQL query for similarity search using the function vector_distance to calculate cosine similarity\n",
                "    sql_similarity_search = f\"\"\"\n",
                "    SELECT TOP(?) filename, chunkid, chunk,\n",
                "           1-vector_distance('cosine', CAST(? AS VECTOR(1024)), embedding) AS similarity_score,\n",
                "           vector_distance('cosine', CAST(? AS VECTOR(1024)), embedding) AS distance_score\n",
                "    FROM dbo.resumedocs\n",
                "    ORDER BY distance_score \n",
                "    \"\"\"\n",
                "\n",
                "    cursor.execute(sql_similarity_search, num_results, json.dumps(user_query_embedding), json.dumps(user_query_embedding))\n",
                "    results = cursor.fetchall()\n",
                "\n",
                "    # Close the database connection\n",
                "    conn.close()\n",
                "\n",
                "    return results\n",
                "    \n",
                "#example usage\n",
                "vector_search_sql(\"database administrator\", num_results=3)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {
                "azdata_cell_guid": "1e194f3c-6a7a-4f16-95ec-05f60a3770a4",
                "language": "python"
            },
            "source": [
                "# **Part 4 : Use embeddings retrieved from a Azure SQL vector database to augment LLM generation**\n",
                "\n",
                "Lets create a helper function to feed prompts into the [Completions model](https://build.nvidia.com/meta/llama-3_3-70b-instruct/) & create interactive loop where you can pose questions to the model and receive information grounded in your data.\n",
                "\n",
                "The function `generate_completion` is defined to help ground the gpt-4o model with prompts and system instructions.   \n",
                "Note that we are passing the results of the `vector_search_sql` we defined earlier to the model and we define the system prompt .  \n",
                "We are using [NVIDIA llama-3_3-70b-instruct](https://build.nvidia.com/meta/llama-3_3-70b-instruct/) model here. \n",
                "\n",
                "You can get more information on using NVIDIA AI models [here](https://build.nvidia.com/)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 34,
            "metadata": {},
            "outputs": [],
            "source": [
                "import os\n",
                "from dotenv import load_dotenv\n",
                "from openai import OpenAI\n",
                "\n",
                "# Load environment variables from a .env file\n",
                "load_dotenv()\n",
                "\n",
                "# Use environment variables for the API key\n",
                "chat_api_key = os.getenv(\"NVIDIA_CHAT_API_KEY\") \n",
                "nvidia_endpoint = \"https://integrate.api.nvidia.com/v1\"\n",
                "chat_model = \"meta/llama-3.3-70b-instruct\"  # NVIDIA-supported LLaMA model\n",
                "\n",
                "# Initialize OpenAI client for NVIDIA API\n",
                "client = OpenAI(\n",
                "    base_url=nvidia_endpoint,\n",
                "    api_key=chat_api_key\n",
                ")\n",
                "\n",
                "def generate_completion(search_results, user_input):\n",
                "    system_prompt = '''\n",
                "You are an intelligent & funny assistant who will exclusively answer based on the data provided in the `search_results`:\n",
                "- Use the information from `search_results` to generate your top 3 responses. If the data is not a perfect match for the user's query, use your best judgment to provide helpful suggestions and include the following format:\n",
                "  File: {filename}\n",
                "  Chunk ID: {chunkid}\n",
                "  Similarity Score: {similarity_score}\n",
                "  Add a small snippet from the Relevant Text: {chunktext}\n",
                "  Do not use the entire chunk.\n",
                "- Avoid any other external data sources.\n",
                "- Add a summary about why the candidate may be a good fit even if exact skills and the role being hired for are not matching. Ensure you call out which skills match the description and which ones are missing. If the candidate doesn't have prior experience for the hiring role, highlight areas to focus on during the interview.\n",
                "- Add a Microsoft-related interesting fact about the technology that was searched.\n",
                "'''\n",
                "\n",
                "    messages = [{\"role\": \"system\", \"content\": system_prompt}]\n",
                "\n",
                "    # Create an empty list to store the results\n",
                "    result_list = []\n",
                "\n",
                "    # Iterate through the search results and append relevant information to the list\n",
                "    for result in search_results:\n",
                "        filename = result  # Assuming filename is in the result\n",
                "        chunkid = result\n",
                "        chunktext = result\n",
                "        similarity_score = result  # Assuming similarity_score is part of the result\n",
                "        \n",
                "        # Append the relevant information as a dictionary to the result_list\n",
                "        result_list.append({\n",
                "            \"filename\": filename,\n",
                "            \"chunkid\": chunkid,\n",
                "            \"chunktext\": chunktext,\n",
                "            \"similarity_score\": similarity_score\n",
                "        })\n",
                "\n",
                "    # Convert the search results into a structured message\n",
                "    messages.append({\"role\": \"system\", \"content\": f\"{result_list}\"})\n",
                "    messages.append({\"role\": \"user\", \"content\": user_input})\n",
                "\n",
                "    # Make a request to NVIDIA's LLaMA model\n",
                "    completion = client.chat.completions.create(\n",
                "        model=chat_model,\n",
                "        messages=messages,\n",
                "        temperature=0.2,\n",
                "        top_p=0.7,\n",
                "        max_tokens=1024,\n",
                "        stream=True\n",
                "    )\n",
                "\n",
                "    # Print the response in a streaming manner\n",
                "    response_text = \"\"\n",
                "    for chunk in completion:\n",
                "        if chunk.choices[0].delta.content is not None:\n",
                "            print(chunk.choices[0].delta.content, end=\"\")\n",
                "            response_text += chunk.choices[0].delta.content\n",
                "\n",
                "    return response_text\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 35,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "🤖 Ask me anything about hiring roles and required skills! Type 'end' to exit.\n",
                        "\n",
                        "Based on the provided search results, here are my top 3 responses:\n",
                        "\n",
                        "1. **File: 70089206.pdf, Chunk ID: 110_0, Similarity Score: 0.6360205426579945**\n",
                        "The candidate has experience in software development, configuration management, and testing, with skills in languages like JavaScript, C, and Oracle PL/SQL. They also have experience with Serena Version Manager and Serena Business Manager.\n",
                        "\n",
                        "2. **File: 10089434.pdf, Chunk ID: 0_2, Similarity Score: 0.6129052569918564**\n",
                        "This candidate has experience in installing, configuring, and supporting Microsoft Windows Server, Microsoft Office, and Microsoft Exchange. They also have skills in programming languages like C, Java, and Python, as well as experience with database administration and network administration.\n",
                        "\n",
                        "3. **File: 10265057.pdf, Chunk ID: 2_0, Similarity Score: 0.6114908210512771**\n",
                        "The candidate has experience as a Working RF Systems Engineer, with skills in RF/microwave software, circuit design software, and programming languages like C, Python, and Java. They also have experience with data analysis and SQL commands.\n",
                        "\n",
                        "**Summary:** While none of the candidates have a perfect match for the software engineer role, they all have relevant experience and skills in software development, testing, and configuration management. The first candidate has experience in software development and testing, while the second candidate has experience in installing and configuring Microsoft products. The third candidate has experience in RF systems engineering and data analysis. \n",
                        "\n",
                        "**Missing Skills:** Depending on the specific requirements of the software engineer role, some candidates may be missing skills in areas like cloud computing, artificial intelligence, or cybersecurity.\n",
                        "\n",
                        "**Microsoft-related interesting fact:** Did you know that Microsoft has a program called \"Microsoft Garage\" that allows employees to work on side projects and develop new ideas? This program has led to the development of several innovative products and services, including the Microsoft Bot Framework and the Microsoft Cognitive Toolkit.\n",
                        "⚠️ Error: Invalid JSON response from AI model.\n",
                        "Based on the provided search results, here are my top 3 responses:\n",
                        "\n",
                        "1. **File: 28672970.pdf, Chunk ID: 78_2, Similarity Score: 0.554, Relevant Text:** \"...artificial intelligence...\" \n",
                        "The candidate may be a good fit for an AI-related role, as they have experience with machine learning and data analysis. However, their skills in natural language processing are limited. \n",
                        "2. **File: 18752129.pdf, Chunk ID: 37_3, Similarity Score: 0.553, Relevant Text:** \"...deep learning...\" \n",
                        "The candidate has a strong background in computer vision and deep learning, making them a suitable fit for a role involving image recognition or object detection. Nevertheless, they lack experience in reinforcement learning. \n",
                        "3. **File: 28035460.pdf, Chunk ID: 76_2, Similarity Score: 0.553, Relevant Text:** \"...neural networks...\" \n",
                        "The candidate has a solid understanding of neural networks and their applications, but they need to improve their skills in explainable AI and model interpretability.\n",
                        "\n",
                        "Microsoft-related interesting fact: Did you know that Microsoft has developed an AI-powered platform called Azure Machine Learning, which enables developers to build, train, and deploy machine learning models at scale?\n",
                        "⚠️ Error: Invalid JSON response from AI model.\n",
                        "Goodbye! 👋\n"
                    ]
                }
            ],
            "source": [
                "import json\n",
                "\n",
                "print(\"🤖 Ask me anything about hiring roles and required skills! Type 'end' to exit.\\n\")\n",
                "\n",
                "while True:\n",
                "    user_input = input(\"You: \")\n",
                "    if user_input.lower() == \"end\":\n",
                "        print(\"Goodbye! 👋\")\n",
                "        break\n",
                "\n",
                "    # Perform vector search to get relevant stored data\n",
                "    search_results = vector_search_sql(user_input)\n",
                "\n",
                "    # Generate AI response based on search results\n",
                "    completion_results = generate_completion(search_results, user_input)\n",
                "\n",
                "    # Ensure response is a dictionary\n",
                "    if isinstance(completion_results, str):  # If response is a string, convert it to JSON\n",
                "        try:\n",
                "            completion_results = json.loads(completion_results)\n",
                "        except json.JSONDecodeError:\n",
                "            print(\"\\n⚠️ Error: Invalid JSON response from AI model.\")\n",
                "            continue\n",
                "\n",
                "    # Check if response has the expected structure\n",
                "    if \"choices\" in completion_results and len(completion_results[\"choices\"]) > 0:\n",
                "        ai_response = completion_results[\"choices\"][0][\"message\"][\"content\"]\n",
                "        print(f\"\\nAI: {ai_response}\\n\")\n",
                "    else:\n",
                "        print(\"\\n⚠️ Error: Unexpected response format from AI model.\\n\")\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "!pip install gradio"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 45,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "* Running on local URL:  http://127.0.0.1:7867\n",
                        "\n",
                        "To create a public link, set `share=True` in `launch()`.\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "<div><iframe src=\"http://127.0.0.1:7867/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
                        ],
                        "text/plain": [
                            "<IPython.core.display.HTML object>"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "import os\n",
                "import pyodbc\n",
                "import json\n",
                "import gradio as gr\n",
                "from dotenv import load_dotenv\n",
                "from openai import OpenAI\n",
                "\n",
                "# Load environment variables\n",
                "load_dotenv()\n",
                "\n",
                "\n",
                "# NVIDIA Chat API setup\n",
                "chat_api_key = os.getenv(\"NVIDIA_CHAT_API_KEY\") \n",
                "nvidia_endpoint = \"https://integrate.api.nvidia.com/v1\"\n",
                "chat_model = \"meta/llama-3.3-70b-instruct\"\n",
                "client = OpenAI(base_url=nvidia_endpoint, api_key=chat_api_key)\n",
                "\n",
                "\n",
                "def get_embedding(text):\n",
                "    \"\"\"\n",
                "    Get sentence embedding using NVIDIA's embed-qa-4 model.\n",
                "\n",
                "    Args:\n",
                "        text (str): Text to embed.\n",
                "\n",
                "    Returns:\n",
                "        list: A list containing the embedding.\n",
                "    \"\"\"\n",
                "    try:\n",
                "        response = client.embeddings.create(\n",
                "            input=[text],\n",
                "            model=\"nvidia/embed-qa-4\",\n",
                "            encoding_format=\"float\",\n",
                "            extra_body={\"input_type\": \"query\", \"truncate\": \"NONE\"}\n",
                "        )\n",
                "        return response.data[0].embedding\n",
                "    except Exception as e:\n",
                "        print(f\"Error generating embedding: {e}\")\n",
                "        return None\n",
                "\n",
                "\n",
                "# Function to perform vector search\n",
                "def vector_search_sql(query, num_results=5):\n",
                "    conn = get_mssql_connection()\n",
                "    cursor = conn.cursor()\n",
                "    user_query_embedding = get_embedding(query)\n",
                "\n",
                "    sql_similarity_search = \"\"\"\n",
                "    SELECT TOP(?) filename, chunkid, chunk,\n",
                "           1-vector_distance('cosine', CAST(? AS VECTOR(1024)), embedding) AS similarity_score,\n",
                "           vector_distance('cosine', CAST(? AS VECTOR(1024)), embedding) AS distance_score\n",
                "    FROM dbo.resumedocs\n",
                "    ORDER BY distance_score \n",
                "    \"\"\"\n",
                "    \n",
                "    cursor.execute(sql_similarity_search, num_results, json.dumps(user_query_embedding), json.dumps(user_query_embedding))\n",
                "    results = cursor.fetchall()\n",
                "    conn.close()\n",
                "    \n",
                "    formatted_results = [\n",
                "        {\n",
                "            \"Filename\": row[0],\n",
                "            \"Chunk ID\": row[1],\n",
                "            \"Chunk\": row[2],\n",
                "            \"Similarity Score\": round(row[3], 4),\n",
                "            \"Distance Score\": round(row[4], 4)\n",
                "        }\n",
                "        for row in results\n",
                "    ]\n",
                "    return formatted_results\n",
                "\n",
                "# Function to generate AI insights\n",
                "def generate_completion(search_results, user_input):\n",
                "    system_prompt = '''\n",
                "You are an intelligent & funny assistant who will exclusively answer based on the data provided in the `search_results`:\n",
                "- Use the information from `search_results` to generate your top 3 responses. If the data is not a perfect match for the user's query, use your best judgment to provide helpful suggestions and include the following format:\n",
                "  File: {filename}\n",
                "  Chunk ID: {chunkid}\n",
                "  Similarity Score: {similarity_score}\n",
                "  Add a small snippet from the Relevant Text: {chunktext}\n",
                "  Do not use the entire chunk.\n",
                "- Avoid any other external data sources.\n",
                "- Add a summary about why the candidate may be a good fit even if exact skills and the role being hired for are not matching.\n",
                "- Add a Microsoft-related interesting fact about the technology that was searched.\n",
                "'''\n",
                "\n",
                "    messages = [{\"role\": \"system\", \"content\": system_prompt}]\n",
                "    messages.append({\"role\": \"system\", \"content\": f\"{json.dumps(search_results, indent=4)}\"})\n",
                "    messages.append({\"role\": \"user\", \"content\": user_input})\n",
                "\n",
                "    completion = client.chat.completions.create(\n",
                "        model=chat_model,\n",
                "        messages=messages,\n",
                "        temperature=0.2,\n",
                "        top_p=0.7,\n",
                "        max_tokens=1024,\n",
                "        stream=True\n",
                "    )\n",
                "\n",
                "    response_text = \"\"\n",
                "    for chunk in completion:\n",
                "        if chunk.choices[0].delta.content is not None:\n",
                "            response_text += chunk.choices[0].delta.content\n",
                "\n",
                "    return response_text\n",
                "\n",
                "# Gradio UI\n",
                "def search_and_generate(query, num_results):\n",
                "    results = vector_search_sql(query, num_results)\n",
                "    ai_response = generate_completion(results, query)\n",
                "    return json.dumps(results, indent=4), ai_response\n",
                "\n",
                "iface = gr.Interface(\n",
                "    fn=search_and_generate,\n",
                "    inputs=[gr.Textbox(label=\"Search Query\"), gr.Slider(minimum=1, maximum=10, step=1, label=\"Number of Results\")],\n",
                "    outputs=[gr.Textbox(label=\"Search Results\"), gr.Textbox(label=\"AI Insights\")],\n",
                "    title=\"Resume Search with AI Insights\",\n",
                "    description=\"Enter a search query to find the most relevant resume chunks and receive AI-powered insights.\"\n",
                ")\n",
                "\n",
                "if __name__ == \"__main__\":\n",
                "    iface.launch()\n"
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.12.9"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}