examples/getting_started/populate_index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "\n",
    "# How to create and populate a vector search index\n",
    "\n",
    "This notebook covers the basic of populating a vector index in Redis. If you are brand new to the concept of vector search and/or RAG with Redis and are looking for more details check out [Redis AI Resources](https://github.com/redis-developer/redis-ai-resources) for more recipes on how to get going.\n",
    "\n",
    "## Creating chunks from a PDF\n",
    "\n",
    "If starting from a pdf document that you want to make searchable you can use langchain pointing to your pdf file to break into chunks\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/redis-applied-ai/retrieval-optimizer/blob/main/examples/getting_started/populate_index.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "## Install requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install langchain langchain-community pypdf redisvl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grab data (if colab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/redis-applied-ai/retrieval-optimizer.git temp_repo\n",
    "!mv temp_repo/examples/getting_started/data .\n",
    "!mv temp_repo/examples/getting_started/schema .\n",
    "!rm -rf temp_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use langchain tools to process pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done preprocessing. Created 354 chunks of the original pdf data/volvo_c30.pdf\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "path = \"data/volvo_c30.pdf\" # path to pdf or other type of file to load\n",
    "\n",
    "# set up the file loader/extractor and text splitter to create chunks\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=2500, chunk_overlap=0\n",
    ")\n",
    "\n",
    "# load and split\n",
    "loader = PyPDFLoader(path)\n",
    "pages = loader.load()\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=2500, chunk_overlap=0)\n",
    "chunks = text_splitter.split_documents(pages)\n",
    "\n",
    "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': 'data/volvo_c30.pdf', 'page': 0, 'page_label': '1'}, page_content='VOLVO C30\\nOwners Manual Web Edition\\nDownloaded from www.Manualslib.com  manuals search engine')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Save the chunks to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "output_file = \"data/volvo_chunks.json\"\n",
    "\n",
    "with open(output_file, \"w\") as f:\n",
    "    json_chunks = [\n",
    "        {\n",
    "            \"text\": chunk.page_content,\n",
    "            \"make\": \"volvo\",\n",
    "            \"model\": \"c30\",\n",
    "            \"item_id\": f\"volvo_c30:{i}\"\n",
    "        } for i, chunk in enumerate(chunks)\n",
    "    ]\n",
    "\n",
    "    json.dump(json_chunks, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining and populating and index from existing chunks\n",
    "\n",
    "Read existing chunks from data folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open(\"data/combined_chunks.json\", \"r\") as f:\n",
    "    chunks = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': \"Mazda3_8Y64-EA-08A_Edition1 Page1 Tuesday, November 27 2007 9:0 AM\\n\\nForm No.8Y64-EA-08A\\n\\nBlack plate (1,1)\\n\\nMazda3_8Y64-EA-08A_Edition1 Page2 Tuesday, November 27 2007 9:0 AM\\n\\nForm No.8Y64-EA-08A\\n\\nBlack plate (2,1)\\n\\nMazda3_8Y64-EA-08A_Edition1 Page3 Tuesday, November 27 2007 9:0 AM\\n\\nBlack plate (3,1)\\n\\nA Word to Mazda Owners\\n\\nThank you for choosing a Mazda. We at Mazda design and build vehicles with complete customer satisfaction in mind.\\n\\nTo help ensure enjoyable and trouble-free operation of your Mazda, read this manual carefully and follow its recommendations.\\n\\nAn Authorized Mazda Dealer knows your vehicle best. So when maintenance or service is necessary, that's the place to go.\\n\\nOur nationwide network of Mazda professionals is dedicated to providing you with the best possible service.\\n\\nWe assure you that all of us at Mazda have an ongoing interest in your motoring pleasure and in your full satisfaction with your Mazda product.\\n\\nMazda Motor Corporation HIROSHIMA, JAPAN\\n\\nImportant Notes About This Manual Keep this manual in the glove box as a handy reference for the safe and enjoyable use of your Mazda. Should you resell the vehicle, leave this manual with it for the next owner.\\n\\nAll specifications and descriptions are accurate at the time of printing. Because improvement is a constant goal at Mazda, we reserve the right to make changes in specifications at any time without notice and without obligation.\\n\\nEvent Data Recorder This vehicle is equipped with an event data recorder. In the event of a crash, this device records data related to vehicle dynamics and safety systems for a short period of time. These data can help provide a better understanding of the circumstances in which crashes and injuries occur and lead to the designing of safer vehicles.\\n\\nAir Conditioning and the Environment Your Mazda's genuine air conditioner is filled with HFC134a (R134a), a refrigerant that has been found not to damage the earth's ozone layer. If the air conditioner does not operate properly, consult an Authorized Mazda Dealer.\\n\\nPerchlorate Certain components of this vehicle such as [air bag modules, seat belt pretensioners, lithium batteries, ...] may contain Perchlorate Material– Special handling may apply for service or vehicle end of life disposal. See www.dtsc.ca.gov/hazardouswaste/perchlorate.\\n\\nPlease be aware that this manual applies to all models, equipment and options. As a result, you may find some explanations for equipment not installed on your vehicle.\",\n",
       " 'query_metadata': {'make': 'mazda', 'model': '3'},\n",
       " 'item_id': 'mazda_3:0'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create vector embeddings of chunk content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from redisvl.utils.vectorize import HFTextVectorizer\n",
    "\n",
    "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
    "\n",
    "# Embed each chunk content\n",
    "embeddings = hf.embed_many([chunk[\"text\"] for chunk in chunks])\n",
    "\n",
    "# Check to make sure we've created enough embeddings, 1 per document chunk\n",
    "len(embeddings) == len(chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run redis instance\n",
    "Later in this tutorial, Redis will be used to store, index, and query vector\n",
    "embeddings created from PDF document chunks. **We need to make sure we have a Redis\n",
    "instance available.**\n",
    "\n",
    "#### For Colab\n",
    "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly\n",
    "from the Redis package archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update  > /dev/null 2>&1\n",
    "sudo apt-get install redis-stack-server  > /dev/null 2>&1\n",
    "redis-stack-server --daemonize yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For Alternative Environments\n",
    "There are many ways to get the necessary redis-stack instance running\n",
    "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
    "own version of Redis Enterprise running, that works too!\n",
    "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
    "3. With docker: `docker run -d --name redis -p 6379:6379 -p 8001:8001 redis/redis-stack:latest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redis import Redis\n",
    "from redisvl.index import SearchIndex\n",
    "\n",
    "REDIS_URL = \"redis://localhost:6379/0\"\n",
    "\n",
    "# connect to redis\n",
    "client = Redis.from_url(REDIS_URL)\n",
    "\n",
    "# path to the schema file\n",
    "path_to_yaml = \"schema/mazda_schema.yaml\"\n",
    "\n",
    "# create an index from schema and the client\n",
    "index = SearchIndex.from_yaml(path_to_yaml)\n",
    "index.set_client(client)\n",
    "index.create(overwrite=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the index is created we can load documents into it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from redisvl.redis.utils import array_to_buffer\n",
    "\n",
    "data = [\n",
    "    {\n",
    "        \"item_id\": chunk[\"item_id\"],\n",
    "        \"text\": chunk[\"text\"],\n",
    "        **chunk[\"query_metadata\"],\n",
    "        # For HASH -- must convert embeddings to bytes\n",
    "        'text_embedding': array_to_buffer(embeddings[i], dtype=\"float32\")\n",
    "    } for i, chunk in enumerate(chunks)\n",
    "]\n",
    "\n",
    "# RedisVL handles batching automatically\n",
    "keys = index.load(data, id_field=\"item_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the index.info to see if the documents were loaded successfully"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "464"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.info()[\"num_docs\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You know have a vector index setup!\n",
    "\n",
    "You can check out your data with [RedisInsight](https://redis.io/insight/) (already running on localhost:8001 if running redis-stack with docker command above) to see the populated fields\n",
    "\n",
    "![r_insight](../../images/r_insight.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}