examples/5_evaluate_haystack_rag_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating a Haystack RAG pipeline with FlowJudge\n",
    "\n",
    "## Overview\n",
    "\n",
    "This tutorial demonstrates how to evaluate a Retrieval-Augmented Generation (RAG) pipeline built with Haystack using `FlowJudge`. We'll showcase how to:\n",
    "\n",
    "1. Set up a basic RAG pipeline using Haystack\n",
    "2. Integrate `FlowJudge` evaluators into a Haystack evaluation pipeline\n",
    "3. Assess the RAG system's performance using multiple metrics:\n",
    "   - Semantic Answer Similarity (SAS)\n",
    "   - Context Relevancy\n",
    "   - Faithfulness\n",
    "\n",
    "Key highlights:\n",
    "\n",
    "- Use of the `HaystackFlowJudge` class to seamlessly incorporate `FlowJudge` evaluators\n",
    "- Demonstration of custom metric creation for tailored evaluations\n",
    "- Utilization of both pre-built Haystack evaluators and custom `FlowJudge` evaluators\n",
    "\n",
    "By the end of this tutorial, you'll have a clear understanding of how to comprehensively evaluate your Haystack RAG pipelines using `FlowJudge`, enabling you to iteratively improve your system's performance without relying on proprietary large language models.\n",
    "\n",
    "### Additional requirements\n",
    "\n",
    "- Haystack: Make sure you have Haystack installed. You can install it via pip:\n",
    "  ```bash\n",
    "  pip install haystack-ai\n",
    "  ```\n",
    "\n",
    "- Sentence Transformers: Make sure you have Sentence Transformers installed. You can install it via pip:\n",
    "  ```bash\n",
    "  pip install \"sentence-transformers>=3.0.0\"\n",
    "  ```\n",
    "\n",
    "- Set your free HuggingFace API token as an environment variable:\n",
    "    ```python\n",
    "    import os\n",
    "    os.environ[\"HF_TOKEN\"] = \"your_token_here\"\n",
    "    ```\n",
    "\n",
    "    You can get your HuggingFace API token [here](https://huggingface.co/settings/tokens)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this notebook primarily demonstrates the integration of `FlowJudge` with Haystack for evaluating RAG pipelines. While we do set up a basic RAG pipeline using Haystack, the main emphasis is on the evaluation process using FlowJudge evaluators.\n",
    "\n",
    "For detailed explanations on building RAG pipelines with Haystack, please refer to the official [Haystack documentation](https://docs.haystack.deepset.ai/docs/intro)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "For this tutorial, we are going to use a subset of the `LegalBench` dataset, which contains contracts and questions from the contracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from datasets import Dataset\n",
    "except ImportError as e:\n",
    "    print(\"datasets is not installed. \")\n",
    "    print(\"Please run `pip install datasets` to install it.\")\n",
    "    print(\"\\nAfter installation, restart the kernel and run this cell again.\")\n",
    "    raise SystemExit(f\"Stopping execution due to missing datasets dependency: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"flowaicom/legalbench_contracts_qa_subset\", \"default\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'question', 'context', 'original_answer', 'answer', 'answer_with_reasoning'],\n",
       "        num_rows: 100\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset contains:\n",
    "- Questions: A question about the contract.\n",
    "- Context: The contract itself.\n",
    "- Original answer: The original answer to the question which can be considered as the ground truth.\n",
    "- Answer: The answer used for generating the answer with reasoning, which can include noise with respect to the original answer.\n",
    "- Answer with reasoning: An answer to the question including the reasoning for the answer based on the contract.\n",
    "\n",
    "For this tutorial:\n",
    "- We use instances without perturbations (where original_answer == answer)\n",
    "- The contract text (context) is used to create documents\n",
    "- We use `answer_with_reasoning` as the ground truth for evaluators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/haystack/core/errors.py:34: DeprecationWarning: PipelineMaxLoops is deprecated and will be remove in version '2.7.0'; use PipelineMaxComponentRuns instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents: 59\n",
      "Number of questions: 59\n",
      "Number of ground truths: 59\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from haystack import Document\n",
    "except ImportError:\n",
    "    print(\"Haystack is not installed. \")\n",
    "    print(\"Please install it according to the 'Additional Requirements' section above.\")\n",
    "    print(\"\\nAfter installation, restart the kernel and run this cell again.\")\n",
    "    raise SystemExit(\"Stopping execution due to missing Haystack dependency.\")\n",
    "\n",
    "filtered_ds = ds.filter(lambda x: x['original_answer'] == x['answer'])\n",
    "\n",
    "all_documents = [Document(content=context) for context in filtered_ds['train']['context']]\n",
    "all_questions = [q for q in filtered_ds['train']['question']]\n",
    "all_ground_truths = [a for a in filtered_ds['train']['answer_with_reasoning']]\n",
    "\n",
    "print(f\"Number of documents: {len(all_documents)}\")\n",
    "print(f\"Number of questions: {len(all_questions)}\")\n",
    "print(f\"Number of ground truths: {len(all_ground_truths)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Question:** Does CNN permit using bots to artificially increase page visits for certain content?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Context:** The following terms apply to content submitted by users, and user conduct, on the Site's Interactive Areas, including CNN iReport:\n",
       "\n",
       "(A) Interactive Areas. The Site may contain comments sections, discussion forums, or other interactive features, including CNN iReport, CNN's user-generated news community (\"Interactive Areas\") in which you may post or upload user-generated content, including but not limited to iReports, comments, video, photos, messages, other materials or items (collectively, \"User Content\"). You are solely responsible for your use of any Interactive Areas and you use them at your own risk. Interactive Areas are available for individuals aged 13 years or older. By submitting User Content to an Interactive Area, you represent that you are 13 years of age or older and, if you are under the age of 18, you either are an emancipated minor, or have obtained the legal consent of your parent or legal guardian to enter into these Terms of Use, submit content, participate on the Site, and fulfill the obligations set forth in these Terms of Use, which forms a binding contract between you and CNN. Employees of Cable News Network, Inc. may not submit User Content without permission from their supervisors.\n",
       "\n",
       "(B) Community Guidelines. By submitting any User Content or participating in an Interactive Area within or in connection with the Site, you agree to abide by the following rules of conduct:\n",
       "You agree not to upload, post or otherwise transmit any User Content that:\n",
       "violates or infringes in any way upon the rights of others, including any statements which may defame, harass, stalk or threaten others.\n",
       "you know to be false, misleading or inaccurate.\n",
       "contains blatant expressions of bigotry, racism, racially or ethnically offensive content, hate speech, abusiveness, vulgarity or profanity.\n",
       "contains or advocates pornography or sexually explicit content, pedophilia, incest, bestiality, or that is otherwise obscene or lewd.\n",
       "violates any law or advocates or provides instruction on dangerous, illegal, or predatory acts, or discusses illegal activities with the intent to commit them.\n",
       "advocates violent behavior.\n",
       "poses a reasonable threat to personal or public safety.\n",
       "contains violent images of killing or physical abuse that appear to have been captured solely, or principally, for exploitive, prurient, or gratuitous purposes.\n",
       "is protected by copyright, trademark, trade secret, right of publicity or other proprietary right without the express permission of the owner of such copyright, trademark, trade secret, right of publicity or other proprietary right. The burden of determining that any User Content is not protected by copyright, trademark, trade secret, right of publicity or other proprietary right rests with you. You shall be solely liable for any damage resulting from any infringement of copyrights, trademarks, trade secrets, rights of publicity or other proprietary rights or any other harm resulting from such a submission. Any person determined by CNN, in its sole discretion, to have violated the intellectual property or other rights of others shall be barred from submitting or posting any further material on the Site.\n",
       "does not generally pertain to the designated topic or theme of any Interactive Area.\n",
       "contains any unsolicited or unauthorized advertising or promotional materials with respect to products or services, \"junk mail,\" \"spam,\" \"chain letters,\" \"pyramid schemes,\" or any other form of solicitation.\n",
       "You agree not to engage in activity that would constitute a criminal offense or give rise to a civil liability.\n",
       "You agree that if necessary, you have the consent of each and every identifiable natural person in any submission to use such persons name or likeness in the manner contemplated by the Site.\n",
       "You agree that any person who appears in your submission who is a current member of the Screen Actors Guild (SAG), the American Federation of Television and Radio Actors (AFTRA) or any other rights society is not entitled to compensation by CNN.\n",
       "You agree not to impersonate any person or entity, including, but not limited to, CNN or any CNN employee, or falsely state or otherwise misrepresent your affiliation with any person or entity.\n",
       "You agree not to represent or suggest, directly or indirectly, CNN's endorsement of User Content.\n",
       "You agree not to interfere with any other user's right to privacy, including by harvesting or collecting personally-identifiable information about the Site users or posting private information about a third party.\n",
       "You agree not to upload, post or otherwise transmit any User Content, software or other materials which contain a virus or other harmful or disruptive component.\n",
       "You agree not to interfere with or disrupt the Site or the servers or networks connected to the Site, or disobey any requirements, procedures, policies or regulations of networks connected to the Site.\n",
       "You agree not to reproduce, duplicate, copy, sell, resell or exploit for any commercial purpose, any portion of the Site, use the Site, or access to the Site.\n",
       "You agree not to use any service, technology or automated system to artificially inflate the page views that your User Content receives. This includes pay-per-click services, web \"robots\" and any other current or future technologies. You also agree not to direct any third party to use these services, technologies or automated systems on your behalf.\n",
       "You agree not to use any technology, service or automated system to post more User Content than an individual could upload in a given period of time. You also agree not to direct any third party to use these services, technologies or automated systems on your behalf.\n",
       "Any conduct that in CNN's sole discretion restricts or inhibits anyone else from using or enjoying the Site will not be permitted. CNN reserves the right in its sole discretion to remove or edit User Content by you and to terminate Your Account for any reason.\n",
       "CNN does not vouch for the accuracy or credibility of any User Content, and does not take any responsibility or assume any liability for any actions you may take as a result of reading User Content posted on the Site. Through your use of Interactive Areas, you may be exposed to content that you may find offensive, objectionable, harmful, inaccurate or deceptive. There may also be risks of dealing with underage persons, people acting under false pretense, international trade issues and foreign nationals. By using Interactive Areas, you assume all associated risks."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Ground truth answer:** No, CNN does not permit using bots to artificially increase page visits for certain content. \n",
       "\n",
       "This is explicitly stated in the Community Guidelines, which prohibits users from using \"any service, technology or automated system to artificially inflate the page views that your User Content receives.\" This rule specifically includes \"web 'robots' and any other current or future technologies\" and extends to directing third parties to use these services on the user's behalf. This means that any attempt to manipulate page views through automated means is a direct violation of CNN's terms of use and can result in the removal of the content or even termination of the user's account."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "display(Markdown(f\"**Question:** {all_questions[0]}\"))\n",
    "display(Markdown(f\"**Context:** {all_documents[0].content}\"))\n",
    "display(Markdown(f\"**Ground truth answer:** {all_ground_truths[0]}\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a RAG pipeline with Haystack\n",
    "\n",
    "We will be creating a very simple RAG pipeline with Haystack.\n",
    "\n",
    "For more detail explanations about building the RAG pipeline, please refer to this tutorial in the Haystack documentation - [Tutorial: Evaluating RAG pipelines](https://haystack.deepset.ai/tutorials/35_evaluating_rag_pipelines)\n",
    "\n",
    ">Note that we have made minor modifications to the pipeline for this tutorial. In particular, we are using `HuggingFaceAPIChatGenerator` and `ChatPromptBuilder`.\n",
    "\n",
    "### Indexing the documents\n",
    "\n",
    "We need to index the documents so we can later use a retriever to find the most similar document to the question.\n",
    "\n",
    "We are using the `InMemoryDocumentStore`, which is a simple in-memory document store that doesn't require setting up a database.\n",
    "\n",
    "We are also using an small open-source embedding model from Sentence Transformers to convert the documents into embeddings.\n",
    "\n",
    "Finally, we are using the `DocumentWriter` to write the documents into the document store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py:181: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v4 of SentenceTransformers.\n",
      "  warnings.warn(\n",
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "309c03a388ed48adba64537dc277c552",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ID 'c0836ba2f9d26978182748cd7560eb3f4e88a82850b29a2243b1cc63da11b6e5' already exists\n",
      "ID '88987c8c6fbb5fac2bbe6689fce472c5df6f7d650fbe42d218dcf7e1b77aa975' already exists\n",
      "ID '676bedf216ff078645168c0b1567e4c78dbc22ec710fc74216a74f7a0b5167fc' already exists\n",
      "ID 'f4fcb43ca4803f5ac8a34374d7cbf9cf248347e5d9ca63f32aaac6296794de6d' already exists\n",
      "ID 'c59aba33746d10420a8e8f09a70d2fa5961a81fc9c41cdc47850a49f6568808b' already exists\n",
      "ID '8805408d3f6c8ae955cf721219c0b4b3891438faa059189e1fd4c1111712c205' already exists\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'document_writer': {'documents_written': 53}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from haystack import Pipeline\n",
    "from haystack.components.embedders import SentenceTransformersDocumentEmbedder\n",
    "from haystack.components.writers import DocumentWriter\n",
    "from haystack.document_stores.in_memory import InMemoryDocumentStore\n",
    "from haystack.document_stores.types import DuplicatePolicy\n",
    "\n",
    "document_store = InMemoryDocumentStore()\n",
    "\n",
    "document_embedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
    "document_writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.SKIP)\n",
    "\n",
    "indexing = Pipeline()\n",
    "indexing.add_component(instance=document_embedder, name=\"document_embedder\")\n",
    "indexing.add_component(instance=document_writer, name=\"document_writer\")\n",
    "\n",
    "indexing.connect(\"document_embedder.documents\", \"document_writer.documents\")\n",
    "\n",
    "indexing.run({\"document_embedder\": {\"documents\": all_documents}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the RAG pipeline\n",
    "\n",
    "Haystack lets us easily create a RAG pipeline using:\n",
    "\n",
    "- `InMemoryEmbeddingRetriever` which will get the relevant documents to the query.\n",
    "- `HuggingFaceAPIChatGenerator` to generate the answer to the question. We are going to use a small open model for this example.\n",
    "\n",
    ">Note you can use the free serverless inference API from HuggingFace to quickly experiment with different models. However, it's rate-limited and not suitable for production. To make use of the API, you just need to provide [your free HuggingFace API token](https://huggingface.co/settings/tokens).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<haystack.core.pipeline.pipeline.Pipeline object at 0x7f648a251f90>\n",
       "🚅 Components\n",
       "  - query_embedder: SentenceTransformersTextEmbedder\n",
       "  - retriever: InMemoryEmbeddingRetriever\n",
       "  - prompt_builder: ChatPromptBuilder\n",
       "  - llm: HuggingFaceAPIChatGenerator\n",
       "  - answer_builder: AnswerBuilder\n",
       "🛤️ Connections\n",
       "  - query_embedder.embedding -> retriever.query_embedding (List[float])\n",
       "  - retriever.documents -> prompt_builder.documents (List[Document])\n",
       "  - retriever.documents -> answer_builder.documents (List[Document])\n",
       "  - prompt_builder.prompt -> llm.messages (List[ChatMessage])\n",
       "  - llm.replies -> answer_builder.replies (List[ChatMessage])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from haystack.components.builders import AnswerBuilder, ChatPromptBuilder\n",
    "from haystack.components.embedders import SentenceTransformersTextEmbedder\n",
    "from haystack.components.generators.chat import HuggingFaceAPIChatGenerator\n",
    "from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever\n",
    "from haystack.utils.hf import HFGenerationAPIType, Secret\n",
    "from haystack.dataclasses import ChatMessage\n",
    "\n",
    "api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API\n",
    "llm = HuggingFaceAPIChatGenerator(api_type=api_type,\n",
    "                                        api_params={\"model\": \"microsoft/Phi-3.5-mini-instruct\"},\n",
    "                                        token=Secret.from_env_var(\"HF_TOKEN\")\n",
    "                                        )\n",
    "\n",
    "\n",
    "template_str = \"\"\"\n",
    "You have to answer the following question based on the given context information only.\n",
    "\n",
    "Context:\n",
    "{% for document in documents %}\n",
    "    {{ document.content }}\n",
    "{% endfor %}\n",
    "\n",
    "Question: {{question}}\n",
    "Answer:\n",
    "\"\"\"\n",
    "template = [ChatMessage.from_user(template_str)]\n",
    "prompt_builder = ChatPromptBuilder(template=template)\n",
    "\n",
    "rag_pipeline = Pipeline()\n",
    "rag_pipeline.add_component(\n",
    "    \"query_embedder\", SentenceTransformersTextEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
    ")\n",
    "rag_pipeline.add_component(\"retriever\", InMemoryEmbeddingRetriever(document_store, top_k=3))\n",
    "rag_pipeline.add_component(\"prompt_builder\", prompt_builder)\n",
    "rag_pipeline.add_component(\"llm\", llm)\n",
    "rag_pipeline.add_component(\"answer_builder\", AnswerBuilder())\n",
    "\n",
    "rag_pipeline.connect(\"query_embedder\", \"retriever.query_embedding\")\n",
    "rag_pipeline.connect(\"retriever\", \"prompt_builder.documents\")\n",
    "rag_pipeline.connect(\"prompt_builder\", \"llm\")\n",
    "rag_pipeline.connect(\"llm.replies\", \"answer_builder.replies\")\n",
    "rag_pipeline.connect(\"retriever\", \"answer_builder.documents\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test the pipeline with a single question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9870a48c82404b1583400d0dafa666cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No, CNN does not permit the use of any technology, service, or automated system, including bots, to artificially inflate page views that User Content receives. The use of pay-per-click services, web \"robots,\" or any other current or future technologies for this purpose is strictly prohibited, as is directing any third party to use these services.\n"
     ]
    }
   ],
   "source": [
    "# Quick test of the pipeline\n",
    "question = \"Does CNN permit using bots to artificially increase page visits for certain content?\"\n",
    "\n",
    "response = rag_pipeline.run(\n",
    "    {\n",
    "        \"query_embedder\": {\"text\": question},\n",
    "        \"prompt_builder\": {\"question\": question},\n",
    "        \"answer_builder\": {\"query\": question},\n",
    "    }\n",
    ")\n",
    "print(response[\"answer_builder\"][\"answers\"][0].data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Document 1 (Score: 0.3840):**\n",
       "\n",
       "The Site contains copyrighted material, trademarks and other proprietary information, including, but not limited to, text, software, photos, video, graphics, music and sound, and the entire contents of the Site are copyrighted as a collective work under the United States copyright laws. CNN owns copyright in the selection, coordination, arrangement and enhancement of such content, as well as in the content original to it. You may not modify, publish, transmit, participate in the transfer or sale..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Document 2 (Score: 0.3641):**\n",
       "\n",
       "The following terms apply to content submitted by users, and user conduct, on the Site's Interactive Areas, including CNN iReport:\n",
       "\n",
       "(A) Interactive Areas. The Site may contain comments sections, discussion forums, or other interactive features, including CNN iReport, CNN's user-generated news community (\"Interactive Areas\") in which you may post or upload user-generated content, including but not limited to iReports, comments, video, photos, messages, other materials or items (collectively, \"Use..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Document 3 (Score: 0.3209):**\n",
       "\n",
       "7. Indemnification.\n",
       "You agree to defend, indemnify and hold harmless CNN, its affiliates and their respective directors, officers, employees and agents from and against all claims and expenses, including attorneys' fees, arising out of the use of the Site by you or your Account. CNN reserves the right to take over the exclusive defense of any claim for which we are entitled to indemnification under this Section. In such event, you shall provide CNN with such cooperation as is reasonably requeste..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# display the retrieved documents and similarity scores\n",
    "for i, doc in enumerate(response['answer_builder']['answers'][0].documents, 1):\n",
    "    display(Markdown(f\"\"\"**Document {i} (Score: {doc.score:.4f}):**\\n\\n{doc.content[:500]}...\"\"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the pipeline\n",
    "\n",
    "With our initial RAG pipeline prototype in place, we can now focus on evaluation.\n",
    "\n",
    "To showcase the integration of `FlowJudge` within the Haystack framework, we'll evaluate the pipeline using both statistical and model-based evaluators.\n",
    "\n",
    "Haystack employs the concept of an __Evaluation pipeline__, which computes scoring metrics to assess the RAG pipeline's performance.\n",
    "\n",
    "Our evaluation pipeline will incorporate three key metrics:\n",
    "- __Semantic Answer Similarity (SAS)__: Measures the semantic similarity between generated and ground truth answers, going beyond simple lexical matching.\n",
    "- __Context Relevancy__: Determines how well the retrieved documents align with the given query.\n",
    "- __Faithfulness__: Assesses the extent to which the generated answer is grounded in the retrieved documents.\n",
    "\n",
    "For context relevancy and faithfulness, we'll leverage `FlowJudge` evaluators, eliminating the need for proprietary large models like GPT-4 or Claude 3.5 Sonnet.\n",
    "\n",
    "### Obtaining generated answers\n",
    "\n",
    "Our first step is to generate answers using the RAG pipeline.\n",
    "\n",
    ">Note: We're using HuggingFace's free serverless inference API, which may take several minutes. To avoid rate limits, we're processing only 20 questions. If execution fails, you can resume from the last successful point by rerunning the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f381282ca6c64c59826e0644540b425d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does CNN permit using bots to artificially increase page visits for certain content?\n",
      "Answer from pipeline:\n",
      "No, CNN does not permit the use of any technology, service, or automated system, including bots, to artificially inflate page views that User Content receives. The use of pay-per-click services, web \"robots,\" or any other current or future technologies for this purpose is strictly prohibited, as is directing any third party to use these services.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbf3baaf21c74ee89b0b35ebd786cca6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does eBay promise that the product data it provides will be somewhat accurate?\n",
      "Answer from pipeline:\n",
      "eBay represents and warrants that the product data it provides is accurate. However, it does not guarantee that the content provided through its Services will always be available, accurate, complete, and up-to-date. Therefore, while eBay provides accurate product data, it does not promise its perpetual accuracy for all time.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54ed602ea3dd4a88b3ddcb9142bd2cca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Will certain terms remain in force notwithstanding a users termination of the service?\n",
      "Answer from pipeline:\n",
      "Yes, certain terms will remain in force even if a user terminates the service. Specifically, sections 4 (Your Content), 6 (Things You Cannot Do), 10 (Indemnity), 11 (Disclaimers), 12 (Limitation of Liability), 13 (Governing Law and Venue), 16 (Termination), and 17 (Miscellaneous) will continue to be valid and survive any termination of the terms or the user's account. This means that even after a user stops using the service, these sections remain enforceable. Additionally, all provisions of these AUP Guidelines that should survive termination will continue to do so.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "133b2d2635ed4ddcbd17723a95c156af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Where a Netflix subscriber requires customer support, is there an alternative means to obtaining customer support apart from enabling Netflix to remotely access the device or devices in question?\n",
      "Answer from pipeline:\n",
      "Yes, if a Netflix subscriber does not want to consent to support through remote access, they can seek assistance through other means as mentioned in the context. The document outlines that in certain instances, Customer Service may be able to assist using remote access support, but if the user does not want this, they should not consent to support through the remote access tool, and help will be provided through other channels. However, the specific alternative means are not described in the given context. To find more information about obtaining support, subscribers should visit the Netflix Help Center on their website, which can offer guidance on how to receive assistance without remote access.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "897559871bae487eb7cc8634fce164de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can Apple disclose personal information about me if asked to by the government?\n",
      "Answer from pipeline:\n",
      "Yes, according to the provided context, Apple may disclose any information about you, including your identity, if they determine that such disclosure is necessary in connection with any investigation or complaint regarding your use of the Site, or to identify, contact, or take legal action against someone who may be causing injury to or interference with Apples rights, property, or the rights or property of visitors to or users of the Site. This includes complying with applicable laws, regulations, legal processes, governmental requests, or for fraud protection purposes.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa77708b13da4205886b464c86a31596",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Am I allowed to scrape data on the services for a research project Im doing?\n",
      "Answer from pipeline:\n",
      "No, you are not allowed to scrape data from the Services in violation of the Terms and Conditions. Specifically, you are prohibited from:\n",
      "\n",
      "(ii) using robots, spiders, scripts, service, software or any manual or automatic device, tool, or process designed to data mine or scrape the Content, data or information from the Services, or otherwise access or collect the Content, data or information from the Services using automated means.\n",
      "\n",
      "Engaging in such activity may result in civil, criminal, and/or administrative penalties, fines, or sanctions against you. To use the Services for a research project, you may need to seek prior written consent from The New York Times Rights and Permissions Department or the copyright holder identified in the copyright notice contained in the Content.\n",
      "\n",
      "It's essential to ensure that the use of content from the Internet, especially when it involves copyrighted materials, respects the intellectual property rights of the content owners and complies with applicable laws and Terms of Service.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d01e4a3270e4f9babd1bf406f363993",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does Amazon take responsibility for items sold by third parties through Amazon?\n",
      "Answer from pipeline:\n",
      "No, Amazon does not take responsibility for items sold by third parties through its platform. Users are purchasing directly from those third parties. Amazon operates stores, provides services, software, or sells product lines through Amazon Services but is not accountable for the actions, products, or content of third-party businesses or individuals. Users should review the privacy statements and other conditions of use of these third parties independently.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41ddc1f4271f4ffe8df5dac9c3ce7c91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does NYT check all user content before its uploaded and posted?\n",
      "Answer from pipeline:\n",
      "Based on the provided context, it does not explicitly state that The New York Times (NYT) checks all user content before it is uploaded and posted. However, the terms outline limitations and prohibitions on content use, protection of copyrighted materials, and guidelines for appropriate use to prevent harm, legal issues, or disruptions to services. Users are required to respect copyright laws, intellectual property, and adhere to content policies, but the specific process for content review prior to posting is not detailed in this context.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68ec93feebb546818680ba7efc9350be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Are there some changes to the terms that Instructure might not let me know about?\n",
      "Answer from pipeline:\n",
      "Yes, Instructure may modify these AUP Guidelines from time to time, and they may not always provide notice to users if a revision significantly reduces your rights. However, if such revisions are meaningful and occur, Instructure will send a notice to the associated email address. By continuing to use the Service after the revisions come into effect, you implicitly agree to be bound by the revised AUP Guidelines.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c99c0b311c046f796207bef70e0fe6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: If someone else is using content I posted on Instagram, could this prevent the content from being deleted?\n",
      "Answer from pipeline:\n",
      "Yes, if someone else has used your content in accordance with the license you granted to Verizon Media and they have not deleted it, this could potentially prevent your content from being deleted immediately. In such cases, the license will continue to apply until that content is eventually deleted. Additionally, if the deletion of your content within 90 days is not possible due to technical limitations or if deleting the content would restrict Verizon Media's ability to comply with legal obligations or respond to judicial or administrative authority requests, the content may be retained for as long as necessary for the purposes for which it has been retained. However, once these situations are no longer in effect, or once your account is deleted, the content may proceed to be deleted as per your request and our standard procedures.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e917384e9bd44ff2881e202a5aba0ba8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Is eBay entitled to decide which party pays the costs of arbitration?\n",
      "Answer from pipeline:\n",
      "Yes, according to the context provided, eBay reserves the right to determine whether costs should be reimbursed to eBay if they prevail in arbitration, as stated under the \"D. Arbitration Fees\" section. Additionally, for disputes involving more than $75,000, the AAA rules will govern payment of filing fees and arbitrator's fees and expenses. This implies that eBay has some discretion in deciding the financial aspects related to arbitration fees depending on the amount involved in the dispute. However, the decision on who pays the costs is subject to the arbitrator's determination in cases where the parties are unable to afford the arbitration costs.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10356f2baaf94973bdf06cbbf73c1506",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Is it sufficient for me to disclose in the edit summary the compensation I received for the edits I made?\n",
      "Answer from pipeline:\n",
      "No, based on the context provided, it is not sufficient to disclose the compensation received for the edits in the edit summary. The terms of use and conditions outlined by Apple focus on privacy, data disclosure, user behavior, and legal compliance, among other things. There is no specific mention about disclosing compensation details in the context provided.\n",
      "\n",
      "Furthermore, Apple's privacy and data disclosure policies do not seem to extend to sharing information about compensation received. However, specific policies may vary by region or change over time. Always ensure you adhere to all related privacy laws and regulations when interacting with such platforms.\n",
      "\n",
      "It's important to remember that any disclosure provision should align with the user agreement terms and be made in a transparent and appropriate manner to maintain trust and avoid misuse. Consult the Site's specific policies if there are uncertainties.\n",
      "\n",
      "Please check the most updated and detailed terms and conditions provided by Apple, or consult with a relevant legal professional if necessary.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dbdab98795bb46578b123c69004b1fde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does Verizon need to tell me if it makes changes to the services?\n",
      "Answer from pipeline:\n",
      "Yes, Verizon is required to notify you when they make changes to the Services. According to the provided context, they may change these Terms at any time and will inform you when such changes become effective. Usage of the Services after these changes take effect implies your agreement to the new terms. If you do not agree to the new terms, you must cease using the Services. This implies that it is Verizon's responsibility to communicate any such updates.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa2ddb43c04842e3b96404e9192ea3c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does Shopify agree to ensure the encryption of all data provided to it?\n",
      "Answer from pipeline:\n",
      "No, Shopify does not guarantee the encryption of all data provided to it. The context states that \"Credit card information is always encrypted during transfer over networks,\" but it does not mention a blanket guarantee for all types of data. Additionally, it is mentioned that \"your Materials (not including credit card information), may be transferred unencrypted and involve (a) transmissions over various networks; and (b) changes to conform and adapt to technical requirements of connecting networks or devices.\" This implies that while credit card information is always encrypted, other data might not be.\n",
      "\n",
      "Moreover, in the 'Limits on Liability' section, Shopify has limited its liability for any lost profits, revenues, or data, further emphasizing the lack of an explicit guarantee in the context provided. This does not mean that Shopify does not take data protection seriously, but instead, these statements express the limitations and general procedures of data handling as per the provided terms. For substantial reassurances related to data security, it would be best to directly refer to Shopify's data protection and security policies and not just base judgement on selected terms.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3a5c81950f345ee907a94a44f89adf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Are there some Disney products that I cant access anonymously?\n",
      "Answer from pipeline:\n",
      "Yes, some Disney products may not allow anonymous access. When you use Disney products, particularly those that integrate with third-party services, applications, platforms, or require an account for participation in certain activities, you will likely be required to provide certain information, such as your email, username, password, or contact details. This is to ensure that you are a legitimate user and to enable Disney to send you communications, provide support, and offer additional benefits.\n",
      "\n",
      "For products that require an account or specific identifying information, you will not be able to access them completely anonymously. However, some items or services may be available without creating an account. It's important to review the terms and conditions for each specific product to understand how your personal information is used and what identifying information, if any, is required to access the product.\n",
      "\n",
      "Always remember to protect your personal information and to be cautious when sharing it, as your privacy and security are of utmost importance. If someone ever requests your account information or tries to use your credentials without your authorization, immediately report such activities to Disney and terminate your access to the affected product(s).\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "254850819f8d43eabc204b1dec25c5b1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can Twitch continue using my content for advertising and marketing after Ive deleted that content?\n",
      "Answer from pipeline:\n",
      "No, Twitch cannot continue using your content for advertising and marketing after you have deleted it. According to the Terms of Service, the rights granted to Twitch terminate once you delete your User Content from the Twitch Services. Although Twitch may use the content for promotional purposes while it's available on the platform, they do not retain the rights to use it once it's removed or deleted from your account. However, Twitch retains the right to use it for the reasonable time it takes to remove from backup and other systems. If you believe your account has been wrongfully terminated, you can appeal using the provided form.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc7292a80dff4c649555cff256e9d0e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does Facebook take responsibility for racist content posted on Facebook?\n",
      "Answer from pipeline:\n",
      "Facebook, as the company behind the Service mentioned in the context, does not take responsibility for individual posts or content shared on its platform. According to the provided context, Facebook operates under a set of Terms of Use and Community Guidelines. If a user posts racist, or any other type of inappropriate content, it is not directly the responsibility of Facebook to intervene or remove the content.\n",
      "\n",
      "However, Facebook has content policies and may take action if inappropriate material violates these policies. According to the \"3. USER GENERATED CONTENT: SUBMISSIONS INCLUDING COMMENTS, READER REVIEWS AND MORE\" section, the rules state that users must not use language that abuses or discriminates on the basis of race, religion, nationality, gender, sexual preference, age, region, disability, etc. Hate speech is grounds for immediate and permanent suspension of access to all or part of the Service.\n",
      "\n",
      "So while Facebook may not be directly liable for the content, it does have guidelines that prohibit racist or discriminatory posts and will take action - such as suspension - if these rules are violated. User content remains under their limited control and does not represent Facebook's responsibility for each individual's actions.\n",
      "\n",
      "The enforcement of these policies might depend on infringement algorithms, user reports, and various moderation efforts. However, issuing accountability and responsibility, as a tech platform with billions of users, is a noteworthy issue in society today. Some argue that companies like Facebook should do more, while others believe that content curation is a complex task that involves balancing free speech with community standards and safety.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b663017dbc9142a0be0b153b34b4dd24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Are there circumstances in which a security breach of a users Microsoft account could give rise to a claim by Microsoft against such user?\n",
      "Answer from pipeline:\n",
      "Yes, according to the context provided, a security breach of a user's Microsoft account could give rise to a claim by Microsoft against such user. The terms of use state that users are entirely responsible for maintaining the confidentiality of their password and account. If there is unauthorized use of the account or a security breach, Microsoft may disclose any information necessary to comply with legal processes, enforce the terms of use, or protect their rights and personal safety as well as the rights and safety of others. Additionally, any violation by the user of these terms of use will constitute an unlawful and unfair business practice, potentially causing irreparable harm to Microsoft, for which monetary damages would be inadequate. In such cases, Microsoft is entitled to seek any injunctive or equitable relief deemed necessary or appropriate. Users also acknowledge that they agree to pay all reasonable attorney's fees and costs of such action, in addition to any other relief granted to Microsoft. Thus, a security breach leading to unauthorized access or misuse of the account can indeed lead to Microsoft initiating legal action against the user for the damages and upholding the terms of the agreement.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27def983534c445eb2f78f6224affcae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does NYT require parental consent in respect of persons aged 17 who wish to access NYT services?\n",
      "Answer from pipeline:\n",
      "No, NYT requires parental consent for persons aged 13 or older to use their services in the USA and the UK. For any age outside of these regions, if a person is under 18, they must ask their parent or legal guardian to review and agree to these Terms of Service before using any part of the services. The specific age at which parental consent is required for a 17-year-old individual is not directly stated; however, since 17 is less than 18, parental consent would be needed.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "324311c25a13486dbcc861ead4f05aab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Does Instagram become the owner of content that I post on Instagram?\n",
      "Answer from pipeline:\n",
      "No, Instagram does not claim ownership of the content you post on their platform. As part of the agreement, you grant Instagram a non-exclusive, royalty-free, transferable, sub-licensable, worldwide license to host, use, distribute, modify, run, copy, publicly perform or display, translate, and create derivative works of your content, as long as it adheres to your privacy and application settings. However, you retain your rights to the content, and you are free to share it with others and decide when to delete it. Instagram's license, which enables them to provide the Service, ends when your content is deleted from their systems. For more details on how you can control or delete your content, review the Instagram Data Policy and the Instagram Help Center.\n",
      "\n",
      "-----------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "questions = all_questions[:20]\n",
    "ground_truths = all_ground_truths[:20]\n",
    "\n",
    "rag_answers = []\n",
    "retrieved_docs = []\n",
    "\n",
    "for question in questions:\n",
    "    response = rag_pipeline.run(\n",
    "        {\n",
    "            \"query_embedder\": {\"text\": question},\n",
    "            \"prompt_builder\": {\"question\": question},\n",
    "            \"answer_builder\": {\"query\": question},\n",
    "        }\n",
    "    )\n",
    "    print(f\"Question: {question}\")\n",
    "    print(\"Answer from pipeline:\")\n",
    "    print(response[\"answer_builder\"][\"answers\"][0].data)\n",
    "    print(\"\\n-----------------------------------\\n\")\n",
    "\n",
    "    rag_answers.append(response[\"answer_builder\"][\"answers\"][0].data)\n",
    "    retrieved_docs.append(response[\"answer_builder\"][\"answers\"][0].documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now convert the retrieved documents into a single string so `FlowJudge` can format the prompt properly under the hood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate the retrieved documents into a single string\n",
    "str_retrieved_docs = []\n",
    "for docs in retrieved_docs:\n",
    "    str_retrieved_doc = \"\"\n",
    "    for i, doc in enumerate(docs, 1):\n",
    "        str_retrieved_doc += doc.content\n",
    "        str_retrieved_doc += \"\\n\"\n",
    "    str_retrieved_docs.append(str_retrieved_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Retrieved documents:** The Site contains copyrighted material, trademarks and other proprietary information, including, but not limited to, text, software, photos, video, graphics, music and sound, and the entire contents of the Site are copyrighted as a collective work under the United States copyright laws. CNN owns copyright in the selection, coordination, arrangement and enhancement of such content, as well as in the content original to it. You may not modify, publish, transmit, participate in the transfer or sale, create derivative works, or in any way exploit, any of the content, in whole or in part. You may download copyrighted material for your personal use only. Except as otherwise expressly permitted under copyright law, no copying, redistribution, retransmission, publication or commercial exploitation of downloaded material will be permitted without the express permission of CNN and the copyright owner. In the event of any permitted copying, redistribution or publication of copyrighted material, no changes in or deletion of author attribution, trademark legend or copyright notice shall be made. You acknowledge that you do not acquire any ownership rights by downloading copyrighted material.\n",
       "The following terms apply to content submitted by users, and user conduct, on the Site's Interactive Areas, including CNN iReport:\n",
       "\n",
       "(A) Interactive Areas. The Site may contain comments sections, discussion forums, or other interactive features, including CNN iReport, CNN's user-generated news community (\"Interactive Areas\") in which you may post or upload user-generated content, including but not limited to iReports, comments, video, photos, messages, other materials or items (collectively, \"User Content\"). You are solely responsible for your use of any Interactive Areas and you use them at your own risk. Interactive Areas are available for individuals aged 13 years or older. By submitting User Content to an Interactive Area, you represent that you are 13 years of age or older and, if you are under the age of 18, you either are an emancipated minor, or have obtained the legal consent of your parent or legal guardian to enter into these Terms of Use, submit content, participate on the Site, and fulfill the obligations set forth in these Terms of Use, which forms a binding contract between you and CNN. Employees of Cable News Network, Inc. may not submit User Content without permission from their supervisors.\n",
       "\n",
       "(B) Community Guidelines. By submitting any User Content or participating in an Interactive Area within or in connection with the Site, you agree to abide by the following rules of conduct:\n",
       "You agree not to upload, post or otherwise transmit any User Content that:\n",
       "violates or infringes in any way upon the rights of others, including any statements which may defame, harass, stalk or threaten others.\n",
       "you know to be false, misleading or inaccurate.\n",
       "contains blatant expressions of bigotry, racism, racially or ethnically offensive content, hate speech, abusiveness, vulgarity or profanity.\n",
       "contains or advocates pornography or sexually explicit content, pedophilia, incest, bestiality, or that is otherwise obscene or lewd.\n",
       "violates any law or advocates or provides instruction on dangerous, illegal, or predatory acts, or discusses illegal activities with the intent to commit them.\n",
       "advocates violent behavior.\n",
       "poses a reasonable threat to personal or public safety.\n",
       "contains violent images of killing or physical abuse that appear to have been captured solely, or principally, for exploitive, prurient, or gratuitous purposes.\n",
       "is protected by copyright, trademark, trade secret, right of publicity or other proprietary right without the express permission of the owner of such copyright, trademark, trade secret, right of publicity or other proprietary right. The burden of determining that any User Content is not protected by copyright, trademark, trade secret, right of publicity or other proprietary right rests with you. You shall be solely liable for any damage resulting from any infringement of copyrights, trademarks, trade secrets, rights of publicity or other proprietary rights or any other harm resulting from such a submission. Any person determined by CNN, in its sole discretion, to have violated the intellectual property or other rights of others shall be barred from submitting or posting any further material on the Site.\n",
       "does not generally pertain to the designated topic or theme of any Interactive Area.\n",
       "contains any unsolicited or unauthorized advertising or promotional materials with respect to products or services, \"junk mail,\" \"spam,\" \"chain letters,\" \"pyramid schemes,\" or any other form of solicitation.\n",
       "You agree not to engage in activity that would constitute a criminal offense or give rise to a civil liability.\n",
       "You agree that if necessary, you have the consent of each and every identifiable natural person in any submission to use such persons name or likeness in the manner contemplated by the Site.\n",
       "You agree that any person who appears in your submission who is a current member of the Screen Actors Guild (SAG), the American Federation of Television and Radio Actors (AFTRA) or any other rights society is not entitled to compensation by CNN.\n",
       "You agree not to impersonate any person or entity, including, but not limited to, CNN or any CNN employee, or falsely state or otherwise misrepresent your affiliation with any person or entity.\n",
       "You agree not to represent or suggest, directly or indirectly, CNN's endorsement of User Content.\n",
       "You agree not to interfere with any other user's right to privacy, including by harvesting or collecting personally-identifiable information about the Site users or posting private information about a third party.\n",
       "You agree not to upload, post or otherwise transmit any User Content, software or other materials which contain a virus or other harmful or disruptive component.\n",
       "You agree not to interfere with or disrupt the Site or the servers or networks connected to the Site, or disobey any requirements, procedures, policies or regulations of networks connected to the Site.\n",
       "You agree not to reproduce, duplicate, copy, sell, resell or exploit for any commercial purpose, any portion of the Site, use the Site, or access to the Site.\n",
       "You agree not to use any service, technology or automated system to artificially inflate the page views that your User Content receives. This includes pay-per-click services, web \"robots\" and any other current or future technologies. You also agree not to direct any third party to use these services, technologies or automated systems on your behalf.\n",
       "You agree not to use any technology, service or automated system to post more User Content than an individual could upload in a given period of time. You also agree not to direct any third party to use these services, technologies or automated systems on your behalf.\n",
       "Any conduct that in CNN's sole discretion restricts or inhibits anyone else from using or enjoying the Site will not be permitted. CNN reserves the right in its sole discretion to remove or edit User Content by you and to terminate Your Account for any reason.\n",
       "CNN does not vouch for the accuracy or credibility of any User Content, and does not take any responsibility or assume any liability for any actions you may take as a result of reading User Content posted on the Site. Through your use of Interactive Areas, you may be exposed to content that you may find offensive, objectionable, harmful, inaccurate or deceptive. There may also be risks of dealing with underage persons, people acting under false pretense, international trade issues and foreign nationals. By using Interactive Areas, you assume all associated risks.\n",
       "7. Indemnification.\n",
       "You agree to defend, indemnify and hold harmless CNN, its affiliates and their respective directors, officers, employees and agents from and against all claims and expenses, including attorneys' fees, arising out of the use of the Site by you or your Account. CNN reserves the right to take over the exclusive defense of any claim for which we are entitled to indemnification under this Section. In such event, you shall provide CNN with such cooperation as is reasonably requested by CNN.\n",
       "8. Termination.\n",
       "CNN may terminate or suspend these Terms of Use at any time without notice to you. Without limiting the foregoing, CNN shall have the right to immediately terminate Your Account in the event of any conduct by you which CNN, in its sole discretion, considers to be unacceptable, or in the event of any breach by you of these Terms of Use. The provisions of Sections 1 - 13 shall survive termination of these Terms of Use.\n",
       "9. Trademarks.\n",
       "CNN, its parent, subsidiaries and affiliates, own all rights to their logos and trademarks used in connection with the Site. All other logos and trademarks appearing on the Site are the property of their respective owners.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(f\"**Retrieved documents:** {str_retrieved_docs[0]}\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluators in Haystack\n",
    "\n",
    "__Evaluators__ in Haystack are versatile components that can operate independently or as integral parts of a pipeline.\n",
    "\n",
    "We'll construct an evaluation pipeline to efficiently obtain scores from all evaluators in a single pass. Additionally, Haystack provides functionality to generate a comprehensive evaluation report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating FlowJudge evaluators using the HaystackFlowJudge class\n",
    "\n",
    "We can use our integration with Haystack to create FlowJudge evaluators in a flexible way. The process is as follows:\n",
    "1. Create a `CustomMetric` that will be used to compute the score for the evaluator.\n",
    "2. Initialize the model - We are using the transformers configuration for Flow-Judge-v0.1.\n",
    "3. Instantiate the `HaystackFlowJudge` evaluator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Important Note on Model Selection:**\n",
    "> \n",
    "> There's a known issue with Phi-3 models producing gibberish outputs for contexts exceeding 4096 tokens (including input and output). While this has been addressed in recent transformers library updates, still remains an issue in the vLLM engine. We recommend the following:\n",
    "> \n",
    "> - For longer contexts: Use the `Flow-Judge-v0.1_HF` model configuration.\n",
    "> - **Caveat:** Inference with transformers is significantly slower than with optimized runtimes.\n",
    "> \n",
    "> This approach ensures reliable outputs for extensive contexts, albeit with a trade-off in processing speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:flow_judge.models.huggingface:Downloading the model from Hugging Face Hub using hf-transfer for faster downloads...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9e3c09daeec4f92837adfb3a1e66e6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 12 files:   0%|          | 0/12 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eac77b45b135445e8b32bd64b2265ee9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from flow_judge.integrations.haystack import HaystackFlowJudge\n",
    "from flow_judge.metrics.metric import CustomMetric, RubricItem\n",
    "from flow_judge.metrics.presets import RESPONSE_FAITHFULNESS_5POINT\n",
    "from flow_judge import Vllm, Hf, Llamafile, Baseten\n",
    "\n",
    "# Create a model using Hugging Face Transformers with Flash Attention\n",
    "model = Hf()\n",
    "\n",
    "# Or if not running on Ampere GPU or newer, create a model using no flash attn and Hugging Face Transformers\n",
    "# model = Hf(flash_attn=False)\n",
    "\n",
    "# Creating a model using Vllm\n",
    "# model = Vllm()\n",
    "\n",
    "# If you have other applications open taking up VRAM, you can use less VRAM by setting gpu_memory_utilization to a lower value.\n",
    "# model = Vllm(gpu_memory_utilization=0.70)\n",
    "\n",
    "# If you are running on a Silicon Mac, you can create a model using Llamafile\n",
    "# model = Llamafile()\n",
    "\n",
    "# Or create a model using Baseten if you don't want to run locally.\n",
    "# As a pre-requisite step:\n",
    "#  - # to Baseten\n",
    "#  - Generate an api key https://app.baseten.co/settings/api_keys\n",
    "#  - Set the api key as an environment variable & initialize:\n",
    "# import os\n",
    "# os.environ[\"BASETEN_API_KEY\"] = \"your_api_key\"\n",
    "# model = Baseten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the context relevancy metric from scratch. For learning more about how to create custom metrics, refer to the [custom metrics tutorial](https://github.com/flowaicom/flow-judge/blob/main/examples/2_custom_evaluation_criteria.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Context relevancy\n",
    "cr_criteria = \"Based on the provided query and context, how relevant and sufficient is the context for responding to the query?\"\n",
    "cr_rubric = [\n",
    "    RubricItem(\n",
    "        score=1,\n",
    "        description=\"The context provided is not relevant or insufficient to respond to the query.\"\n",
    "    ),\n",
    "    RubricItem(\n",
    "        score=2,\n",
    "        description=\"The context is mostly irrelevant to the query. It may contain some tangentially related information but is insufficient for adequately responding to the query.\"\n",
    "    ),\n",
    "    RubricItem(\n",
    "        score=3,\n",
    "        description=\"The context is somewhat relevant to the query. It contains some information that could be used to partially respond to the query, but key details are missing for a complete response.\"\n",
    "    ),\n",
    "    RubricItem(\n",
    "        score=4,\n",
    "        description=\"The context is mostly relevant to the query. It contains most of the necessary information to respond to the query, but may be missing some minor details.\"\n",
    "    ),\n",
    "    RubricItem(\n",
    "        score=5,\n",
    "        description=\"The context is highly relevant to the query. It contains all the necessary information to comprehensively respond to the query without needing any additional context.\"\n",
    "    )\n",
    "]\n",
    "cr_metric = CustomMetric(\n",
    "    name=\"Context Relevancy\",\n",
    "    criteria=cr_criteria,\n",
    "    rubric=cr_rubric,\n",
    "    required_inputs=[\"question\"],\n",
    "    required_output=\"contexts\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For creating the faithfulness evaluator, we are going to use the `RESPONSE_FAITHFULNESS_5POINT` preset in flow-judge library as a template.\n",
    "\n",
    "> Note that we need to use the expected keys so we need to update required inputs and outputs to match the expected keys in the RAG pipeline. In this case, the score descriptions are still relevant with these changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Criteria:** Based on the given context, evaluate how consistent and faithful the generated response is to the context. The response should not contain any hallucinated or fabricated information that is not supported by the context."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Rubric:**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "- **Score 1:** The response is completely inconsistent with the provided context. It contains significant amount of hallucinated or fabricated information that directly contradicts or is not supported at all by the context."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "- **Score 2:** The response is mostly inconsistent with the provided context. While it may contain some information from the context, it introduces a substantial amount of hallucinated or fabricated details that deviate from the context."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "- **Score 3:** The response is somewhat consistent with the provided context. It includes a mix of information from the context and some hallucinated or fabricated details. The fabrications are minor and do not significantly contradict the context."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "- **Score 4:** The response is mostly consistent with the provided context. The vast majority of the content is supported by the context, with only minor and inconsequential inconsistencies or fabrications, if any."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "- **Score 5:** The response is completely consistent with and faithful to the provided context. All details in the response are directly supported by the context, without any hallucinated or fabricated information."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ff_criteria = RESPONSE_FAITHFULNESS_5POINT.criteria\n",
    "ff_rubric = RESPONSE_FAITHFULNESS_5POINT.rubric\n",
    "\n",
    "display(Markdown(f\"**Criteria:** {ff_criteria}\"))\n",
    "display(Markdown(\"**Rubric:**\"))\n",
    "\n",
    "for item in ff_rubric:\n",
    "    display(Markdown(f\"- **Score {item.score}:** {item.description}\"))\n",
    "\n",
    "ff_metric = CustomMetric(\n",
    "    name=\"Faithfulness\",\n",
    "    criteria=ff_criteria,\n",
    "    rubric=ff_rubric,\n",
    "    required_inputs=[\"question\", \"contexts\"],\n",
    "    required_output=\"predicted_answers\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create the Flow Judge evaluators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "cr_evaluator = HaystackFlowJudge(\n",
    "    metric=cr_metric,\n",
    "    model=model, # the vLLM instance of Flow-Judge-v0.1\n",
    "    progress_bar=True,\n",
    "    raise_on_failure=True, # to raise an error when pipeline run fails\n",
    "    save_results=True, # to save evaluation results to disk\n",
    "    fail_on_parse_error=False # to fail if there is a parsing error, otherwise return \"Error\" and score -1\n",
    ")\n",
    "\n",
    "ff_evaluator = HaystackFlowJudge(\n",
    "    metric=ff_metric,\n",
    "    model=model,\n",
    "    progress_bar=True,\n",
    "    raise_on_failure=True,\n",
    "    save_results=True,\n",
    "    fail_on_parse_error=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Haystack evaluators\n",
    "\n",
    "Now let's crete the semantic answer similarity evaluator using the Haystack implementation. This evaluator will use the same embedding model as the retriever in the RAG pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.components.evaluators.sas_evaluator import SASEvaluator\n",
    "\n",
    "sas_evaluator = SASEvaluator(model=\"sentence-transformers/all-MiniLM-L6-v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation pipeline\n",
    "\n",
    "We can now create a Haystack evaluation pipeline that will evaluate the RAG pipeline and obtains the evaluation results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_pipeline = Pipeline()\n",
    "\n",
    "# add components to the pipeline\n",
    "eval_pipeline.add_component(\"sas_evaluator\", sas_evaluator)\n",
    "eval_pipeline.add_component(\"cr_evaluator\", cr_evaluator)\n",
    "eval_pipeline.add_component(\"ff_evaluator\", ff_evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note that executing the following cell might take a while to complete due to the size of the inputs, specially if running on a machine with low resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:haystack.core.pipeline.base:Warming up component sas_evaluator...\n",
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py:991: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.\n",
      "  warnings.warn(\n",
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py:181: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v4 of SentenceTransformers.\n",
      "  warnings.warn(\n",
      "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n",
      "/home/ks/repos/flow-judge/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n",
      "INFO:haystack.core.pipeline.pipeline:Running component sas_evaluator\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f0909a19421487aa320e951a71e7b3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9dd03aeef9145ab96d517e8c2b89877",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:haystack.core.pipeline.pipeline:Running component cr_evaluator\n",
      "INFO:root:Starting llamafile server...\n",
      "INFO:root:Llamafile path: /home/ks/.cache/flow-judge/flow-judge.llamafile\n",
      "INFO:root:Starting llamafile server with command: sh -c '/home/ks/.cache/flow-judge/flow-judge.llamafile --server --host 127.0.0.1 --port 8085 -c 8192 -ngl 34 --temp 0.1 -n 2000 --threads 32 --nobrowser -b 32 --parallel 1 --cont-batching'\n",
      "INFO:root:Subprocess started with PID: 192232\n",
      "INFO:openai._base_client:Retrying request to /models in 0.408262 seconds\n",
      "INFO:root:import_cuda_impl: initializing gpu module...\n",
      "INFO:root:get_rocm_bin_path: note: hipcc not found on $PATH\n",
      "INFO:root:get_rocm_bin_path: note: $HIP_PATH/bin/hipcc does not exist\n",
      "INFO:root:get_rocm_bin_path: note: /opt/rocm/bin/hipcc does not exist\n",
      "INFO:root:link_cuda_dso: note: dynamically linking /home/ks/.llamafile/v/0.8.13/ggml-rocm.so\n",
      "INFO:root:link_cuda_dso: warning: libamdhip64.so.6: cannot open shared object file: No such file or directory: failed to load library\n",
      "INFO:root:link_cuda_dso: note: dynamically linking /home/ks/.llamafile/v/0.8.13/ggml-rocm.so\n",
      "INFO:root:link_cuda_dso: warning: libamdhip64.so.6: cannot open shared object file: No such file or directory: failed to load library\n",
      "INFO:root:link_cuda_dso: note: dynamically linking /home/ks/.llamafile/v/0.8.13/ggml-cuda.so\n",
      "INFO:root:ggml_cuda_link: welcome to CUDA SDK with tinyBLAS\n",
      "INFO:root:link_cuda_dso: GPU support loaded\n",
      "INFO:root:{\"build\":1500,\"commit\":\"a30b324\",\"function\":\"server_cli\",\"level\":\"INFO\",\"line\":2841,\"msg\":\"build info\",\"tid\":\"11681088\",\"timestamp\":1728313424}\n",
      "INFO:root:{\"function\":\"server_cli\",\"level\":\"INFO\",\"line\":2844,\"msg\":\"system info\",\"n_threads\":32,\"n_threads_batch\":-1,\"system_info\":\"AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \",\"tid\":\"11681088\",\"timestamp\":1728313424,\"total_threads\":32}\n",
      "INFO:root:llama_model_loader: loaded meta data with 33 key-value pairs and 197 tensors from flow-judge-v0.1-Q4_K_M.gguf (version GGUF V3 (latest))\n",
      "INFO:root:llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "INFO:root:llama_model_loader: - kv   0:                       general.architecture str              = phi3\n",
      "INFO:root:llama_model_loader: - kv   1:                               general.type str              = model\n",
      "INFO:root:llama_model_loader: - kv   2:                               general.name str              = Phi 3.5 Mini Instruct\n",
      "INFO:root:llama_model_loader: - kv   3:                       general.organization str              = Microsoft\n",
      "INFO:root:llama_model_loader: - kv   4:                           general.finetune str              = instruct\n",
      "INFO:root:llama_model_loader: - kv   5:                           general.basename str              = Phi-3.5\n",
      "INFO:root:llama_model_loader: - kv   6:                         general.size_label str              = mini\n",
      "INFO:root:llama_model_loader: - kv   7:                        phi3.context_length u32              = 131072\n",
      "INFO:root:llama_model_loader: - kv   8:  phi3.rope.scaling.original_context_length u32              = 4096\n",
      "INFO:root:llama_model_loader: - kv   9:                      phi3.embedding_length u32              = 3072\n",
      "INFO:root:llama_model_loader: - kv  10:                   phi3.feed_forward_length u32              = 8192\n",
      "INFO:root:llama_model_loader: - kv  11:                           phi3.block_count u32              = 32\n",
      "INFO:root:llama_model_loader: - kv  12:                  phi3.attention.head_count u32              = 32\n",
      "INFO:root:llama_model_loader: - kv  13:               phi3.attention.head_count_kv u32              = 32\n",
      "INFO:root:llama_model_loader: - kv  14:      phi3.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "INFO:root:llama_model_loader: - kv  15:                  phi3.rope.dimension_count u32              = 96\n",
      "INFO:root:llama_model_loader: - kv  16:                        phi3.rope.freq_base f32              = 10000.000000\n",
      "INFO:root:llama_model_loader: - kv  17:                          general.file_type u32              = 15\n",
      "INFO:root:llama_model_loader: - kv  18:              phi3.attention.sliding_window u32              = 262144\n",
      "INFO:root:llama_model_loader: - kv  19:              phi3.rope.scaling.attn_factor f32              = 1.190238\n",
      "INFO:root:llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = llama\n",
      "INFO:root:llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = default\n",
      "INFO:root:llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,32064]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "INFO:root:llama_model_loader: - kv  23:                      tokenizer.ggml.scores arr[f32,32064]   = [-1000.000000, -1000.000000, -1000.00...\n",
      "INFO:root:llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,32064]   = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "INFO:root:llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "INFO:root:llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 32000\n",
      "INFO:root:llama_model_loader: - kv  27:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "INFO:root:llama_model_loader: - kv  28:            tokenizer.ggml.padding_token_id u32              = 32000\n",
      "INFO:root:llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = false\n",
      "INFO:root:llama_model_loader: - kv  30:               tokenizer.ggml.add_eos_token bool             = false\n",
      "INFO:root:llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
      "INFO:root:llama_model_loader: - kv  32:               general.quantization_version u32              = 2\n",
      "INFO:root:llama_model_loader: - type  f32:   67 tensors\n",
      "INFO:root:llama_model_loader: - type q4_K:   81 tensors\n",
      "INFO:root:llama_model_loader: - type q5_K:   32 tensors\n",
      "INFO:root:llama_model_loader: - type q6_K:   17 tensors\n",
      "INFO:root:llm_load_vocab: special tokens cache size = 14\n",
      "INFO:root:llm_load_vocab: token to piece cache size = 0.1685 MB\n",
      "INFO:root:llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "INFO:root:llm_load_print_meta: arch             = phi3\n",
      "INFO:root:llm_load_print_meta: vocab type       = SPM\n",
      "INFO:root:llm_load_print_meta: n_vocab          = 32064\n",
      "INFO:root:llm_load_print_meta: n_merges         = 0\n",
      "INFO:root:llm_load_print_meta: vocab_only       = 0\n",
      "INFO:root:llm_load_print_meta: n_ctx_train      = 131072\n",
      "INFO:root:llm_load_print_meta: n_embd           = 3072\n",
      "INFO:root:llm_load_print_meta: n_layer          = 32\n",
      "INFO:root:llm_load_print_meta: n_head           = 32\n",
      "INFO:root:llm_load_print_meta: n_head_kv        = 32\n",
      "INFO:root:llm_load_print_meta: n_rot            = 96\n",
      "INFO:root:llm_load_print_meta: n_swa            = 262144\n",
      "INFO:root:llm_load_print_meta: n_embd_head_k    = 96\n",
      "INFO:root:llm_load_print_meta: n_embd_head_v    = 96\n",
      "INFO:root:llm_load_print_meta: n_gqa            = 1\n",
      "INFO:root:llm_load_print_meta: n_embd_k_gqa     = 3072\n",
      "INFO:root:llm_load_print_meta: n_embd_v_gqa     = 3072\n",
      "INFO:root:llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "INFO:root:llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "INFO:root:llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "INFO:root:llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "INFO:root:llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
      "INFO:root:llm_load_print_meta: n_ff             = 8192\n",
      "INFO:root:llm_load_print_meta: n_expert         = 0\n",
      "INFO:root:llm_load_print_meta: n_expert_used    = 0\n",
      "INFO:root:llm_load_print_meta: causal attn      = 1\n",
      "INFO:root:llm_load_print_meta: pooling type     = 0\n",
      "INFO:root:llm_load_print_meta: rope type        = 2\n",
      "INFO:root:llm_load_print_meta: rope scaling     = linear\n",
      "INFO:root:llm_load_print_meta: freq_base_train  = 10000.0\n",
      "INFO:root:llm_load_print_meta: freq_scale_train = 1\n",
      "INFO:root:llm_load_print_meta: n_ctx_orig_yarn  = 4096\n",
      "INFO:root:llm_load_print_meta: rope_finetuned   = unknown\n",
      "INFO:root:llm_load_print_meta: ssm_d_conv       = 0\n",
      "INFO:root:llm_load_print_meta: ssm_d_inner      = 0\n",
      "INFO:root:llm_load_print_meta: ssm_d_state      = 0\n",
      "INFO:root:llm_load_print_meta: ssm_dt_rank      = 0\n",
      "INFO:root:llm_load_print_meta: model type       = 3B\n",
      "INFO:root:llm_load_print_meta: model ftype      = Q4_K - Medium\n",
      "INFO:root:llm_load_print_meta: model params     = 3.82 B\n",
      "INFO:root:llm_load_print_meta: model size       = 2.23 GiB (5.01 BPW)\n",
      "INFO:root:llm_load_print_meta: general.name     = Phi 3.5 Mini Instruct\n",
      "INFO:root:llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "INFO:root:llm_load_print_meta: EOS token        = 32000 '<|endoftext|>'\n",
      "INFO:root:llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "INFO:root:llm_load_print_meta: PAD token        = 32000 '<|endoftext|>'\n",
      "INFO:root:llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "INFO:root:llm_load_print_meta: EOT token        = 32007 '<|end|>'\n",
      "INFO:root:llm_load_print_meta: max token length = 48\n",
      "INFO:root:ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no\n",
      "INFO:root:ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\n",
      "INFO:root:ggml_cuda_init: found 3 CUDA devices:\n",
      "INFO:root:Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes\n",
      "INFO:root:Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes\n",
      "INFO:root:Device 2: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes\n",
      "INFO:root:llm_load_tensors: ggml ctx size =    0.48 MiB\n",
      "INFO:root:llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "INFO:root:llm_load_tensors: offloaded 32/33 layers to GPU\n",
      "INFO:root:llm_load_tensors:        CPU buffer size =  2281.66 MiB\n",
      "INFO:root:llm_load_tensors:      CUDA0 buffer size =   608.28 MiB\n",
      "INFO:root:llm_load_tensors:      CUDA1 buffer size =   794.54 MiB\n",
      "INFO:root:llm_load_tensors:      CUDA2 buffer size =   748.95 MiB\n",
      "INFO:openai._base_client:Retrying request to /models in 0.963148 seconds\n",
      "INFO:root:............................................................................................\n",
      "INFO:root:llama_new_context_with_model: n_ctx      = 8192\n",
      "INFO:root:llama_new_context_with_model: n_batch    = 32\n",
      "INFO:root:llama_new_context_with_model: n_ubatch   = 32\n",
      "INFO:root:llama_new_context_with_model: flash_attn = 0\n",
      "INFO:root:llama_new_context_with_model: freq_base  = 10000.0\n",
      "INFO:root:llama_new_context_with_model: freq_scale = 1\n",
      "INFO:root:llama_kv_cache_init:      CUDA0 KV buffer size =   864.00 MiB\n",
      "INFO:root:llama_kv_cache_init:      CUDA1 KV buffer size =  1152.00 MiB\n",
      "INFO:root:llama_kv_cache_init:      CUDA2 KV buffer size =  1056.00 MiB\n",
      "INFO:root:llama_new_context_with_model: KV self size  = 3072.00 MiB, K (f16): 1536.00 MiB, V (f16): 1536.00 MiB\n",
      "INFO:root:llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB\n",
      "INFO:root:llama_new_context_with_model:      CUDA0 compute buffer size =    81.35 MiB\n",
      "INFO:root:llama_new_context_with_model:      CUDA1 compute buffer size =    35.25 MiB\n",
      "INFO:root:llama_new_context_with_model:      CUDA2 compute buffer size =    35.25 MiB\n",
      "INFO:root:llama_new_context_with_model:  CUDA_Host compute buffer size =     1.38 MiB\n",
      "INFO:root:llama_new_context_with_model: graph nodes  = 1286\n",
      "INFO:root:llama_new_context_with_model: graph splits = 6\n",
      "INFO:root:{\"function\":\"initialize\",\"level\":\"INFO\",\"line\":491,\"msg\":\"initializing slots\",\"n_slots\":1,\"tid\":\"11681088\",\"timestamp\":1728313425}\n",
      "INFO:root:\n",
      "INFO:root:llama server listening at http://127.0.0.1:8085\n",
      "INFO:root:{\"function\":\"initialize\",\"level\":\"INFO\",\"line\":500,\"msg\":\"new slot\",\"n_ctx_slot\":8192,\"slot_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313425}\n",
      "INFO:root:\n",
      "INFO:root:{\"function\":\"server_cli\",\"level\":\"INFO\",\"line\":3062,\"msg\":\"model loaded\",\"tid\":\"11681088\",\"timestamp\":1728313425}\n",
      "INFO:root:{\"function\":\"server_cli\",\"hostname\":\"127.0.0.1\",\"level\":\"INFO\",\"line\":3185,\"msg\":\"HTTP server listening\",\"port\":\"8085\",\"tid\":\"11681088\",\"timestamp\":1728313425}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1661,\"msg\":\"all slots are idle and system prompt is empty, clear the KV cache\",\"tid\":\"11681088\",\"timestamp\":1728313425}\n",
      "INFO:httpx:HTTP Request: GET http://127.0.0.1:8085/v1/models \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"GET\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/models\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":49990,\"status\":200,\"tid\":\"139974063720448\",\"timestamp\":1728313425}\n",
      "INFO:root:Llamafile server started successfully\n",
      "INFO:httpx:HTTP Request: GET http://127.0.0.1:8085/v1/models \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"GET\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/models\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":49990,\"status\":200,\"tid\":\"139974063720448\",\"timestamp\":1728313425}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313426}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313426}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1038.82 ms /  2764 tokens (    0.38 ms per token,  2660.72 tokens per second)\",\"n_tokens_second\":2660.71887541309,\"num_prompt_tokens_processed\":2764,\"slot_id\":0,\"t_prompt_processing\":1038.817,\"t_token\":0.3758382778581766,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313429}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2839.36 ms /   196 runs   (   14.49 ms per token,    69.03 tokens per second)\",\"n_decoded\":196,\"n_tokens_second\":69.02968910577673,\"slot_id\":0,\"t_token\":14.486520408163265,\"t_token_generation\":2839.358,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313429}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3878.18 ms\",\"slot_id\":0,\"t_prompt_processing\":1038.817,\"t_token_generation\":2839.358,\"t_total\":3878.175,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313429}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2960,\"n_ctx\":8192,\"n_past\":2959,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":0,\"tid\":\"11681088\",\"timestamp\":1728313429,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":49990,\"status\":200,\"tid\":\"139974063720448\",\"timestamp\":1728313429}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313429}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313429}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1390.33 ms /  3331 tokens (    0.42 ms per token,  2395.83 tokens per second)\",\"n_tokens_second\":2395.8289129294926,\"num_prompt_tokens_processed\":3331,\"slot_id\":0,\"t_prompt_processing\":1390.333,\"t_token\":0.4173920744521165,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313434}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2829.24 ms /   173 runs   (   16.35 ms per token,    61.15 tokens per second)\",\"n_decoded\":173,\"n_tokens_second\":61.14714158320199,\"slot_id\":0,\"t_token\":16.353994219653178,\"t_token_generation\":2829.241,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313434}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4219.57 ms\",\"slot_id\":0,\"t_prompt_processing\":1390.333,\"t_token_generation\":2829.241,\"t_total\":4219.5740000000005,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313434}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3504,\"n_ctx\":8192,\"n_past\":3503,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":198,\"tid\":\"11681088\",\"timestamp\":1728313434,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":49990,\"status\":200,\"tid\":\"139974063720448\",\"timestamp\":1728313434}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313434}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313434}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     491.71 ms /  1501 tokens (    0.33 ms per token,  3052.59 tokens per second)\",\"n_tokens_second\":3052.5874797138176,\"num_prompt_tokens_processed\":1501,\"slot_id\":0,\"t_prompt_processing\":491.714,\"t_token\":0.32759093937375083,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313438}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3605.20 ms /   321 runs   (   11.23 ms per token,    89.04 tokens per second)\",\"n_decoded\":321,\"n_tokens_second\":89.03808083825608,\"slot_id\":0,\"t_token\":11.23114953271028,\"t_token_generation\":3605.199,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313438}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4096.91 ms\",\"slot_id\":0,\"t_prompt_processing\":491.714,\"t_token_generation\":3605.199,\"t_total\":4096.9130000000005,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313438}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":1822,\"n_ctx\":8192,\"n_past\":1821,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":373,\"tid\":\"11681088\",\"timestamp\":1728313438,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":49990,\"status\":200,\"tid\":\"139974063720448\",\"timestamp\":1728313438}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313438}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313438}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1418.58 ms /  3472 tokens (    0.41 ms per token,  2447.51 tokens per second)\",\"n_tokens_second\":2447.512764498094,\"num_prompt_tokens_processed\":3472,\"slot_id\":0,\"t_prompt_processing\":1418.583,\"t_token\":0.40857805299539174,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313443}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4098.04 ms /   258 runs   (   15.88 ms per token,    62.96 tokens per second)\",\"n_decoded\":258,\"n_tokens_second\":62.95692574987067,\"slot_id\":0,\"t_token\":15.883875968992248,\"t_token_generation\":4098.04,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313443}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5516.62 ms\",\"slot_id\":0,\"t_prompt_processing\":1418.583,\"t_token_generation\":4098.04,\"t_total\":5516.623,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313443}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3730,\"n_ctx\":8192,\"n_past\":3729,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":696,\"tid\":\"11681088\",\"timestamp\":1728313443,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44176,\"status\":200,\"tid\":\"139974063725296\",\"timestamp\":1728313443}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313443}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313443}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     932.43 ms /  2545 tokens (    0.37 ms per token,  2729.43 tokens per second)\",\"n_tokens_second\":2729.430337323271,\"num_prompt_tokens_processed\":2545,\"slot_id\":0,\"t_prompt_processing\":932.429,\"t_token\":0.36637681728880156,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313449}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4469.02 ms /   334 runs   (   13.38 ms per token,    74.74 tokens per second)\",\"n_decoded\":334,\"n_tokens_second\":74.73672645530196,\"slot_id\":0,\"t_token\":13.38030239520958,\"t_token_generation\":4469.021,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313449}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5401.45 ms\",\"slot_id\":0,\"t_prompt_processing\":932.429,\"t_token_generation\":4469.021,\"t_total\":5401.45,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313449}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2879,\"n_ctx\":8192,\"n_past\":2878,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":956,\"tid\":\"11681088\",\"timestamp\":1728313449,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44176,\"status\":200,\"tid\":\"139974063725296\",\"timestamp\":1728313449}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313449}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313449}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     827.72 ms /  2328 tokens (    0.36 ms per token,  2812.56 tokens per second)\",\"n_tokens_second\":2812.5622949928415,\"num_prompt_tokens_processed\":2328,\"slot_id\":0,\"t_prompt_processing\":827.715,\"t_token\":0.35554768041237117,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313455}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    5082.31 ms /   393 runs   (   12.93 ms per token,    77.33 tokens per second)\",\"n_decoded\":393,\"n_tokens_second\":77.32696615617097,\"slot_id\":0,\"t_token\":12.93209923664122,\"t_token_generation\":5082.315,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313455}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5910.03 ms\",\"slot_id\":0,\"t_prompt_processing\":827.715,\"t_token_generation\":5082.315,\"t_total\":5910.03,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313455}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2721,\"n_ctx\":8192,\"n_past\":2720,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":1292,\"tid\":\"11681088\",\"timestamp\":1728313455,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44176,\"status\":200,\"tid\":\"139974063725296\",\"timestamp\":1728313455}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313455}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313455}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     836.55 ms /  2335 tokens (    0.36 ms per token,  2791.24 tokens per second)\",\"n_tokens_second\":2791.239214579951,\"num_prompt_tokens_processed\":2335,\"slot_id\":0,\"t_prompt_processing\":836.546,\"t_token\":0.35826381156316917,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313460}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4042.20 ms /   311 runs   (   13.00 ms per token,    76.94 tokens per second)\",\"n_decoded\":311,\"n_tokens_second\":76.9382057565116,\"slot_id\":0,\"t_token\":12.997443729903537,\"t_token_generation\":4042.205,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313460}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4878.75 ms\",\"slot_id\":0,\"t_prompt_processing\":836.546,\"t_token_generation\":4042.205,\"t_total\":4878.751,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313460}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2646,\"n_ctx\":8192,\"n_past\":2645,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":1687,\"tid\":\"11681088\",\"timestamp\":1728313460,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44176,\"status\":200,\"tid\":\"139974063725296\",\"timestamp\":1728313460}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313460}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313460}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     795.98 ms /  2248 tokens (    0.35 ms per token,  2824.19 tokens per second)\",\"n_tokens_second\":2824.1880145380355,\"num_prompt_tokens_processed\":2248,\"slot_id\":0,\"t_prompt_processing\":795.981,\"t_token\":0.3540840747330961,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313465}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4495.38 ms /   354 runs   (   12.70 ms per token,    78.75 tokens per second)\",\"n_decoded\":354,\"n_tokens_second\":78.74760170174902,\"slot_id\":0,\"t_token\":12.69879943502825,\"t_token_generation\":4495.375,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313465}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5291.36 ms\",\"slot_id\":0,\"t_prompt_processing\":795.981,\"t_token_generation\":4495.375,\"t_total\":5291.356,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313465}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2602,\"n_ctx\":8192,\"n_past\":2601,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":2000,\"tid\":\"11681088\",\"timestamp\":1728313465,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44176,\"status\":200,\"tid\":\"139974063725296\",\"timestamp\":1728313465}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313465}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313465}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     767.71 ms /  2181 tokens (    0.35 ms per token,  2840.92 tokens per second)\",\"n_tokens_second\":2840.916491904495,\"num_prompt_tokens_processed\":2181,\"slot_id\":0,\"t_prompt_processing\":767.71,\"t_token\":0.35199908298945437,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313469}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3661.02 ms /   294 runs   (   12.45 ms per token,    80.31 tokens per second)\",\"n_decoded\":294,\"n_tokens_second\":80.30542282853726,\"slot_id\":0,\"t_token\":12.45245918367347,\"t_token_generation\":3661.023,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313469}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4428.73 ms\",\"slot_id\":0,\"t_prompt_processing\":767.71,\"t_token_generation\":3661.023,\"t_total\":4428.733,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313469}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2475,\"n_ctx\":8192,\"n_past\":2474,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":2356,\"tid\":\"11681088\",\"timestamp\":1728313469,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":46474,\"status\":200,\"tid\":\"139974063715552\",\"timestamp\":1728313469}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313469}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313469}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     946.33 ms /  2565 tokens (    0.37 ms per token,  2710.48 tokens per second)\",\"n_tokens_second\":2710.4767057510717,\"num_prompt_tokens_processed\":2565,\"slot_id\":0,\"t_prompt_processing\":946.328,\"t_token\":0.36893879142300195,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313473}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3026.20 ms /   226 runs   (   13.39 ms per token,    74.68 tokens per second)\",\"n_decoded\":226,\"n_tokens_second\":74.68109355591383,\"slot_id\":0,\"t_token\":13.390269911504426,\"t_token_generation\":3026.201,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313473}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3972.53 ms\",\"slot_id\":0,\"t_prompt_processing\":946.328,\"t_token_generation\":3026.201,\"t_total\":3972.529,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313473}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2791,\"n_ctx\":8192,\"n_past\":2790,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":2652,\"tid\":\"11681088\",\"timestamp\":1728313473,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":46474,\"status\":200,\"tid\":\"139974063715552\",\"timestamp\":1728313473}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313473}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313473}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1902.70 ms /  4288 tokens (    0.44 ms per token,  2253.64 tokens per second)\",\"n_tokens_second\":2253.6371959455555,\"num_prompt_tokens_processed\":4288,\"slot_id\":0,\"t_prompt_processing\":1902.702,\"t_token\":0.44372714552238807,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313482}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    6309.32 ms /   397 runs   (   15.89 ms per token,    62.92 tokens per second)\",\"n_decoded\":397,\"n_tokens_second\":62.92275732277456,\"slot_id\":0,\"t_token\":15.892501259445845,\"t_token_generation\":6309.323,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313482}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    8212.02 ms\",\"slot_id\":0,\"t_prompt_processing\":1902.702,\"t_token_generation\":6309.323,\"t_total\":8212.025,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313482}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":4685,\"n_ctx\":8192,\"n_past\":4684,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":2880,\"tid\":\"11681088\",\"timestamp\":1728313482,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":46474,\"status\":200,\"tid\":\"139974063715552\",\"timestamp\":1728313482}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313482}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313482}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1022.09 ms /  2739 tokens (    0.37 ms per token,  2679.79 tokens per second)\",\"n_tokens_second\":2679.7926609489928,\"num_prompt_tokens_processed\":2739,\"slot_id\":0,\"t_prompt_processing\":1022.094,\"t_token\":0.37316319824753563,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313487}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4702.17 ms /   336 runs   (   13.99 ms per token,    71.46 tokens per second)\",\"n_decoded\":336,\"n_tokens_second\":71.45640053694382,\"slot_id\":0,\"t_token\":13.994547619047617,\"t_token_generation\":4702.168,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313487}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5724.26 ms\",\"slot_id\":0,\"t_prompt_processing\":1022.094,\"t_token_generation\":4702.168,\"t_total\":5724.262,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313487}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3075,\"n_ctx\":8192,\"n_past\":3074,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":3279,\"tid\":\"11681088\",\"timestamp\":1728313487,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":46474,\"status\":200,\"tid\":\"139974063715552\",\"timestamp\":1728313487}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313487}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313487}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1230.53 ms /  3121 tokens (    0.39 ms per token,  2536.30 tokens per second)\",\"n_tokens_second\":2536.3013720894705,\"num_prompt_tokens_processed\":3121,\"slot_id\":0,\"t_prompt_processing\":1230.532,\"t_token\":0.3942749118872156,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313494}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4996.51 ms /   329 runs   (   15.19 ms per token,    65.85 tokens per second)\",\"n_decoded\":329,\"n_tokens_second\":65.84600001561091,\"slot_id\":0,\"t_token\":15.186951367781154,\"t_token_generation\":4996.507,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313494}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    6227.04 ms\",\"slot_id\":0,\"t_prompt_processing\":1230.532,\"t_token_generation\":4996.507,\"t_total\":6227.039,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313494}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3450,\"n_ctx\":8192,\"n_past\":3449,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":3617,\"tid\":\"11681088\",\"timestamp\":1728313494,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":46474,\"status\":200,\"tid\":\"139974063715552\",\"timestamp\":1728313494}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313494}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313494}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1200.43 ms /  3084 tokens (    0.39 ms per token,  2569.07 tokens per second)\",\"n_tokens_second\":2569.0708527082706,\"num_prompt_tokens_processed\":3084,\"slot_id\":0,\"t_prompt_processing\":1200.434,\"t_token\":0.38924578469520105,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313499}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4092.05 ms /   274 runs   (   14.93 ms per token,    66.96 tokens per second)\",\"n_decoded\":274,\"n_tokens_second\":66.95908726455266,\"slot_id\":0,\"t_token\":14.934492700729926,\"t_token_generation\":4092.051,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313499}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5292.48 ms\",\"slot_id\":0,\"t_prompt_processing\":1200.434,\"t_token_generation\":4092.051,\"t_total\":5292.485,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313499}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3358,\"n_ctx\":8192,\"n_past\":3357,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":3948,\"tid\":\"11681088\",\"timestamp\":1728313499,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44220,\"status\":200,\"tid\":\"139974063730096\",\"timestamp\":1728313499}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313499}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313499}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1685.84 ms /  3906 tokens (    0.43 ms per token,  2316.94 tokens per second)\",\"n_tokens_second\":2316.9444805293024,\"num_prompt_tokens_processed\":3906,\"slot_id\":0,\"t_prompt_processing\":1685.841,\"t_token\":0.4316029185867895,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313507}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    6239.01 ms /   394 runs   (   15.84 ms per token,    63.15 tokens per second)\",\"n_decoded\":394,\"n_tokens_second\":63.151014431289056,\"slot_id\":0,\"t_token\":15.835058375634517,\"t_token_generation\":6239.013,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313507}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    7924.85 ms\",\"slot_id\":0,\"t_prompt_processing\":1685.841,\"t_token_generation\":6239.013,\"t_total\":7924.853999999999,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313507}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":4300,\"n_ctx\":8192,\"n_past\":4299,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":4224,\"tid\":\"11681088\",\"timestamp\":1728313507,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44220,\"status\":200,\"tid\":\"139974063730096\",\"timestamp\":1728313507}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313507}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313507}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     912.94 ms /  2478 tokens (    0.37 ms per token,  2714.32 tokens per second)\",\"n_tokens_second\":2714.319514182812,\"num_prompt_tokens_processed\":2478,\"slot_id\":0,\"t_prompt_processing\":912.936,\"t_token\":0.3684164648910412,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313512}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4355.85 ms /   327 runs   (   13.32 ms per token,    75.07 tokens per second)\",\"n_decoded\":327,\"n_tokens_second\":75.07140392478809,\"slot_id\":0,\"t_token\":13.32065137614679,\"t_token_generation\":4355.853,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313512}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5268.79 ms\",\"slot_id\":0,\"t_prompt_processing\":912.936,\"t_token_generation\":4355.853,\"t_total\":5268.789,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313512}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2805,\"n_ctx\":8192,\"n_past\":2804,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":4620,\"tid\":\"11681088\",\"timestamp\":1728313512,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44220,\"status\":200,\"tid\":\"139974063730096\",\"timestamp\":1728313512}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313512}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313512}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1120.52 ms /  2942 tokens (    0.38 ms per token,  2625.57 tokens per second)\",\"n_tokens_second\":2625.5690443446297,\"num_prompt_tokens_processed\":2942,\"slot_id\":0,\"t_prompt_processing\":1120.519,\"t_token\":0.3808698164513936,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313518}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4857.20 ms /   333 runs   (   14.59 ms per token,    68.56 tokens per second)\",\"n_decoded\":333,\"n_tokens_second\":68.5580451939575,\"slot_id\":0,\"t_token\":14.58618018018018,\"t_token_generation\":4857.198,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313518}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5977.72 ms\",\"slot_id\":0,\"t_prompt_processing\":1120.519,\"t_token_generation\":4857.198,\"t_total\":5977.717000000001,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313518}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3275,\"n_ctx\":8192,\"n_past\":3274,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":4949,\"tid\":\"11681088\",\"timestamp\":1728313518,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44220,\"status\":200,\"tid\":\"139974063730096\",\"timestamp\":1728313518}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313518}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313518}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1026.92 ms /  2722 tokens (    0.38 ms per token,  2650.65 tokens per second)\",\"n_tokens_second\":2650.6523896283734,\"num_prompt_tokens_processed\":2722,\"slot_id\":0,\"t_prompt_processing\":1026.917,\"t_token\":0.37726561351947097,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313524}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4669.24 ms /   330 runs   (   14.15 ms per token,    70.68 tokens per second)\",\"n_decoded\":330,\"n_tokens_second\":70.67529819086228,\"slot_id\":0,\"t_token\":14.149215151515152,\"t_token_generation\":4669.241,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313524}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5696.16 ms\",\"slot_id\":0,\"t_prompt_processing\":1026.917,\"t_token_generation\":4669.241,\"t_total\":5696.157999999999,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313524}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3052,\"n_ctx\":8192,\"n_past\":3051,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":5284,\"tid\":\"11681088\",\"timestamp\":1728313524,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":44220,\"status\":200,\"tid\":\"139974063730096\",\"timestamp\":1728313524}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313524}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313524}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1308.73 ms /  3253 tokens (    0.40 ms per token,  2485.62 tokens per second)\",\"n_tokens_second\":2485.6215238166556,\"num_prompt_tokens_processed\":3253,\"slot_id\":0,\"t_prompt_processing\":1308.727,\"t_token\":0.4023138641254227,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313529}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3417.11 ms /   222 runs   (   15.39 ms per token,    64.97 tokens per second)\",\"n_decoded\":222,\"n_tokens_second\":64.96723690537054,\"slot_id\":0,\"t_token\":15.392373873873874,\"t_token_generation\":3417.107,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313529}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4725.83 ms\",\"slot_id\":0,\"t_prompt_processing\":1308.727,\"t_token_generation\":3417.107,\"t_total\":4725.834,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313529}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3475,\"n_ctx\":8192,\"n_past\":3474,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":5616,\"tid\":\"11681088\",\"timestamp\":1728313529,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":60374,\"status\":200,\"tid\":\"139974063735008\",\"timestamp\":1728313529}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313529}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313529}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1029.62 ms /  2753 tokens (    0.37 ms per token,  2673.80 tokens per second)\",\"n_tokens_second\":2673.801985198423,\"num_prompt_tokens_processed\":2753,\"slot_id\":0,\"t_prompt_processing\":1029.62,\"t_token\":0.37399927351979656,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313534}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4625.75 ms /   326 runs   (   14.19 ms per token,    70.48 tokens per second)\",\"n_decoded\":326,\"n_tokens_second\":70.47507333406979,\"slot_id\":0,\"t_token\":14.189414110429448,\"t_token_generation\":4625.749,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313534}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5655.37 ms\",\"slot_id\":0,\"t_prompt_processing\":1029.62,\"t_token_generation\":4625.749,\"t_total\":5655.369,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313534}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3079,\"n_ctx\":8192,\"n_past\":3078,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":5840,\"tid\":\"11681088\",\"timestamp\":1728313534,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":60374,\"status\":200,\"tid\":\"139974063735008\",\"timestamp\":1728313534}\n",
      "INFO:flow_judge.flow_judge:Saving results to output/\n",
      "INFO:flow_judge.utils.result_writer:Results saved to output/context_relevancy/results_context_relevancy_sariola__flow-judge-llamafile_llamafile_2024-10-07T15-05-34.864.jsonl\n",
      "INFO:haystack.core.pipeline.pipeline:Running component ff_evaluator\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313534}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313534}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1126.18 ms /  2935 tokens (    0.38 ms per token,  2606.15 tokens per second)\",\"n_tokens_second\":2606.1460649414307,\"num_prompt_tokens_processed\":2935,\"slot_id\":0,\"t_prompt_processing\":1126.184,\"t_token\":0.3837083475298126,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313540}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4684.98 ms /   325 runs   (   14.42 ms per token,    69.37 tokens per second)\",\"n_decoded\":325,\"n_tokens_second\":69.37064178942957,\"slot_id\":0,\"t_token\":14.415320000000001,\"t_token_generation\":4684.979,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313540}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5811.16 ms\",\"slot_id\":0,\"t_prompt_processing\":1126.184,\"t_token_generation\":4684.979,\"t_total\":5811.1630000000005,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313540}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3260,\"n_ctx\":8192,\"n_past\":3259,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":6168,\"tid\":\"11681088\",\"timestamp\":1728313540,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":60374,\"status\":200,\"tid\":\"139974063735008\",\"timestamp\":1728313540}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313540}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313540}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1423.46 ms /  3493 tokens (    0.41 ms per token,  2453.87 tokens per second)\",\"n_tokens_second\":2453.8748109364274,\"num_prompt_tokens_processed\":3493,\"slot_id\":0,\"t_prompt_processing\":1423.463,\"t_token\":0.40751875178929287,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313546}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4311.76 ms /   270 runs   (   15.97 ms per token,    62.62 tokens per second)\",\"n_decoded\":270,\"n_tokens_second\":62.61942626226268,\"slot_id\":0,\"t_token\":15.969485185185187,\"t_token_generation\":4311.761,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313546}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5735.22 ms\",\"slot_id\":0,\"t_prompt_processing\":1423.463,\"t_token_generation\":4311.761,\"t_total\":5735.224,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313546}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3763,\"n_ctx\":8192,\"n_past\":3762,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":6495,\"tid\":\"11681088\",\"timestamp\":1728313546,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":60374,\"status\":200,\"tid\":\"139974063735008\",\"timestamp\":1728313546}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313546}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313546}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     584.93 ms /  1751 tokens (    0.33 ms per token,  2993.53 tokens per second)\",\"n_tokens_second\":2993.5308277258055,\"num_prompt_tokens_processed\":1751,\"slot_id\":0,\"t_prompt_processing\":584.928,\"t_token\":0.33405368360936605,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313549}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2865.38 ms /   252 runs   (   11.37 ms per token,    87.95 tokens per second)\",\"n_decoded\":252,\"n_tokens_second\":87.94638899804633,\"slot_id\":0,\"t_token\":11.370563492063493,\"t_token_generation\":2865.382,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313549}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3450.31 ms\",\"slot_id\":0,\"t_prompt_processing\":584.928,\"t_token_generation\":2865.382,\"t_total\":3450.31,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313549}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2003,\"n_ctx\":8192,\"n_past\":2002,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":6767,\"tid\":\"11681088\",\"timestamp\":1728313549,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":60374,\"status\":200,\"tid\":\"139974063735008\",\"timestamp\":1728313549}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313549}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313549}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1548.69 ms /  3699 tokens (    0.42 ms per token,  2388.47 tokens per second)\",\"n_tokens_second\":2388.4671710062426,\"num_prompt_tokens_processed\":3699,\"slot_id\":0,\"t_prompt_processing\":1548.692,\"t_token\":0.4186785617734523,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313556}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    5256.16 ms /   316 runs   (   16.63 ms per token,    60.12 tokens per second)\",\"n_decoded\":316,\"n_tokens_second\":60.119946904193725,\"slot_id\":0,\"t_token\":16.633414556962023,\"t_token_generation\":5256.159,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313556}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    6804.85 ms\",\"slot_id\":0,\"t_prompt_processing\":1548.692,\"t_token_generation\":5256.159,\"t_total\":6804.851,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313556}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":4015,\"n_ctx\":8192,\"n_past\":4014,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":7021,\"tid\":\"11681088\",\"timestamp\":1728313556,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":54516,\"status\":200,\"tid\":\"139974063744688\",\"timestamp\":1728313556}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313556}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313556}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1033.81 ms /  2750 tokens (    0.38 ms per token,  2660.07 tokens per second)\",\"n_tokens_second\":2660.0684072864597,\"num_prompt_tokens_processed\":2750,\"slot_id\":0,\"t_prompt_processing\":1033.808,\"t_token\":0.37593018181818183,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313560}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2490.20 ms /   178 runs   (   13.99 ms per token,    71.48 tokens per second)\",\"n_decoded\":178,\"n_tokens_second\":71.48008757515449,\"slot_id\":0,\"t_token\":13.989910112359551,\"t_token_generation\":2490.204,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313560}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3524.01 ms\",\"slot_id\":0,\"t_prompt_processing\":1033.808,\"t_token_generation\":2490.204,\"t_total\":3524.012,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313560}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2928,\"n_ctx\":8192,\"n_past\":2927,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":7339,\"tid\":\"11681088\",\"timestamp\":1728313560,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":54516,\"status\":200,\"tid\":\"139974063744688\",\"timestamp\":1728313560}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313560}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313560}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     986.99 ms /  2640 tokens (    0.37 ms per token,  2674.81 tokens per second)\",\"n_tokens_second\":2674.8099770412146,\"num_prompt_tokens_processed\":2640,\"slot_id\":0,\"t_prompt_processing\":986.986,\"t_token\":0.37385833333333335,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313564}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2792.64 ms /   206 runs   (   13.56 ms per token,    73.77 tokens per second)\",\"n_decoded\":206,\"n_tokens_second\":73.76532599977082,\"slot_id\":0,\"t_token\":13.556504854368931,\"t_token_generation\":2792.64,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313564}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3779.63 ms\",\"slot_id\":0,\"t_prompt_processing\":986.986,\"t_token_generation\":2792.64,\"t_total\":3779.6259999999997,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313564}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2846,\"n_ctx\":8192,\"n_past\":2845,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":7519,\"tid\":\"11681088\",\"timestamp\":1728313564,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":54516,\"status\":200,\"tid\":\"139974063744688\",\"timestamp\":1728313564}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313564}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313564}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     932.47 ms /  2513 tokens (    0.37 ms per token,  2695.00 tokens per second)\",\"n_tokens_second\":2695.0044291159143,\"num_prompt_tokens_processed\":2513,\"slot_id\":0,\"t_prompt_processing\":932.466,\"t_token\":0.37105690409868686,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313569}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3947.06 ms /   296 runs   (   13.33 ms per token,    74.99 tokens per second)\",\"n_decoded\":296,\"n_tokens_second\":74.99256408190607,\"slot_id\":0,\"t_token\":13.334655405405405,\"t_token_generation\":3947.058,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313569}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4879.52 ms\",\"slot_id\":0,\"t_prompt_processing\":932.466,\"t_token_generation\":3947.058,\"t_total\":4879.524,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313569}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2809,\"n_ctx\":8192,\"n_past\":2808,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":7727,\"tid\":\"11681088\",\"timestamp\":1728313569,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":54516,\"status\":200,\"tid\":\"139974063744688\",\"timestamp\":1728313569}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313569}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313569}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     897.03 ms /  2451 tokens (    0.37 ms per token,  2732.34 tokens per second)\",\"n_tokens_second\":2732.3379046948053,\"num_prompt_tokens_processed\":2451,\"slot_id\":0,\"t_prompt_processing\":897.034,\"t_token\":0.36598694410444715,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313572}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2745.34 ms /   210 runs   (   13.07 ms per token,    76.49 tokens per second)\",\"n_decoded\":210,\"n_tokens_second\":76.49317407697326,\"slot_id\":0,\"t_token\":13.073061904761904,\"t_token_generation\":2745.343,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313572}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3642.38 ms\",\"slot_id\":0,\"t_prompt_processing\":897.034,\"t_token_generation\":2745.343,\"t_total\":3642.377,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313572}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2661,\"n_ctx\":8192,\"n_past\":2660,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":8025,\"tid\":\"11681088\",\"timestamp\":1728313572,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":54516,\"status\":200,\"tid\":\"139974063744688\",\"timestamp\":1728313572}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313572}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313572}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =     843.27 ms /  2365 tokens (    0.36 ms per token,  2804.57 tokens per second)\",\"n_tokens_second\":2804.571748416277,\"num_prompt_tokens_processed\":2365,\"slot_id\":0,\"t_prompt_processing\":843.266,\"t_token\":0.3565606765327695,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313576}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    2674.62 ms /   208 runs   (   12.86 ms per token,    77.77 tokens per second)\",\"n_decoded\":208,\"n_tokens_second\":77.76814399968295,\"slot_id\":0,\"t_token\":12.858735576923078,\"t_token_generation\":2674.617,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313576}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    3517.88 ms\",\"slot_id\":0,\"t_prompt_processing\":843.266,\"t_token_generation\":2674.617,\"t_total\":3517.8830000000003,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313576}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2573,\"n_ctx\":8192,\"n_past\":2572,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":8237,\"tid\":\"11681088\",\"timestamp\":1728313576,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":59274,\"status\":200,\"tid\":\"139974063749520\",\"timestamp\":1728313576}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313576}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313576}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1071.25 ms /  2819 tokens (    0.38 ms per token,  2631.50 tokens per second)\",\"n_tokens_second\":2631.5003379223563,\"num_prompt_tokens_processed\":2819,\"slot_id\":0,\"t_prompt_processing\":1071.252,\"t_token\":0.3800113515431004,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313580}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3595.44 ms /   253 runs   (   14.21 ms per token,    70.37 tokens per second)\",\"n_decoded\":253,\"n_tokens_second\":70.36690919609283,\"slot_id\":0,\"t_token\":14.211225296442688,\"t_token_generation\":3595.44,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313580}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4666.69 ms\",\"slot_id\":0,\"t_prompt_processing\":1071.252,\"t_token_generation\":3595.44,\"t_total\":4666.692,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313580}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3072,\"n_ctx\":8192,\"n_past\":3071,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":8447,\"tid\":\"11681088\",\"timestamp\":1728313580,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":59274,\"status\":200,\"tid\":\"139974063749520\",\"timestamp\":1728313580}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313580}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313580}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    2112.01 ms /  4537 tokens (    0.47 ms per token,  2148.19 tokens per second)\",\"n_tokens_second\":2148.1895690884185,\"num_prompt_tokens_processed\":4537,\"slot_id\":0,\"t_prompt_processing\":2112.011,\"t_token\":0.46550826537359485,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313589}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    6718.16 ms /   409 runs   (   16.43 ms per token,    60.88 tokens per second)\",\"n_decoded\":409,\"n_tokens_second\":60.879746573541986,\"slot_id\":0,\"t_token\":16.425823960880198,\"t_token_generation\":6718.162,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313589}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    8830.17 ms\",\"slot_id\":0,\"t_prompt_processing\":2112.011,\"t_token_generation\":6718.162,\"t_total\":8830.173,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313589}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":4946,\"n_ctx\":8192,\"n_past\":4945,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":8702,\"tid\":\"11681088\",\"timestamp\":1728313589,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":59274,\"status\":200,\"tid\":\"139974063749520\",\"timestamp\":1728313589}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313589}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313589}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1193.70 ms /  3051 tokens (    0.39 ms per token,  2555.92 tokens per second)\",\"n_tokens_second\":2555.916431334145,\"num_prompt_tokens_processed\":3051,\"slot_id\":0,\"t_prompt_processing\":1193.701,\"t_token\":0.3912490986561783,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313595}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4236.97 ms /   288 runs   (   14.71 ms per token,    67.97 tokens per second)\",\"n_decoded\":288,\"n_tokens_second\":67.973183634984,\"slot_id\":0,\"t_token\":14.711684027777778,\"t_token_generation\":4236.965,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313595}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5430.67 ms\",\"slot_id\":0,\"t_prompt_processing\":1193.701,\"t_token_generation\":4236.965,\"t_total\":5430.666,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313595}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3339,\"n_ctx\":8192,\"n_past\":3338,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":9113,\"tid\":\"11681088\",\"timestamp\":1728313595,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":59274,\"status\":200,\"tid\":\"139974063749520\",\"timestamp\":1728313595}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313595}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313595}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1347.72 ms /  3308 tokens (    0.41 ms per token,  2454.52 tokens per second)\",\"n_tokens_second\":2454.519417266817,\"num_prompt_tokens_processed\":3308,\"slot_id\":0,\"t_prompt_processing\":1347.718,\"t_token\":0.4074117291414752,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313600}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3398.45 ms /   221 runs   (   15.38 ms per token,    65.03 tokens per second)\",\"n_decoded\":221,\"n_tokens_second\":65.02955019266108,\"slot_id\":0,\"t_token\":15.37762443438914,\"t_token_generation\":3398.455,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313600}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4746.17 ms\",\"slot_id\":0,\"t_prompt_processing\":1347.718,\"t_token_generation\":3398.455,\"t_total\":4746.173,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313600}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3529,\"n_ctx\":8192,\"n_past\":3528,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":9403,\"tid\":\"11681088\",\"timestamp\":1728313600,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":59274,\"status\":200,\"tid\":\"139974063749520\",\"timestamp\":1728313600}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313600}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313600}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1386.80 ms /  3421 tokens (    0.41 ms per token,  2466.82 tokens per second)\",\"n_tokens_second\":2466.8229973377634,\"num_prompt_tokens_processed\":3421,\"slot_id\":0,\"t_prompt_processing\":1386.804,\"t_token\":0.4053797135340544,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313605}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3910.40 ms /   248 runs   (   15.77 ms per token,    63.42 tokens per second)\",\"n_decoded\":248,\"n_tokens_second\":63.42062193126023,\"slot_id\":0,\"t_token\":15.76774193548387,\"t_token_generation\":3910.4,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313605}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5297.20 ms\",\"slot_id\":0,\"t_prompt_processing\":1386.804,\"t_token_generation\":3910.4,\"t_total\":5297.204,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313605}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3669,\"n_ctx\":8192,\"n_past\":3668,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":9626,\"tid\":\"11681088\",\"timestamp\":1728313605,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":57608,\"status\":200,\"tid\":\"139974063739840\",\"timestamp\":1728313605}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313605}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313605}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1911.74 ms /  4237 tokens (    0.45 ms per token,  2216.30 tokens per second)\",\"n_tokens_second\":2216.3044052515484,\"num_prompt_tokens_processed\":4237,\"slot_id\":0,\"t_prompt_processing\":1911.741,\"t_token\":0.451201557705924,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313611}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4204.52 ms /   268 runs   (   15.69 ms per token,    63.74 tokens per second)\",\"n_decoded\":268,\"n_tokens_second\":63.740865791228686,\"slot_id\":0,\"t_token\":15.688522388059702,\"t_token_generation\":4204.524,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313611}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    6116.27 ms\",\"slot_id\":0,\"t_prompt_processing\":1911.741,\"t_token_generation\":4204.524,\"t_total\":6116.265,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313611}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":4505,\"n_ctx\":8192,\"n_past\":4504,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":9876,\"tid\":\"11681088\",\"timestamp\":1728313611,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":57608,\"status\":200,\"tid\":\"139974063739840\",\"timestamp\":1728313611}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313611}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313611}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1019.84 ms /  2704 tokens (    0.38 ms per token,  2651.41 tokens per second)\",\"n_tokens_second\":2651.409296601901,\"num_prompt_tokens_processed\":2704,\"slot_id\":0,\"t_prompt_processing\":1019.835,\"t_token\":0.37715791420118344,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313616}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    3494.14 ms /   251 runs   (   13.92 ms per token,    71.83 tokens per second)\",\"n_decoded\":251,\"n_tokens_second\":71.83461896313739,\"slot_id\":0,\"t_token\":13.92086454183267,\"t_token_generation\":3494.137,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313616}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    4513.97 ms\",\"slot_id\":0,\"t_prompt_processing\":1019.835,\"t_token_generation\":3494.137,\"t_total\":4513.972,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313616}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":2955,\"n_ctx\":8192,\"n_past\":2954,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":10146,\"tid\":\"11681088\",\"timestamp\":1728313616,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":57608,\"status\":200,\"tid\":\"139974063739840\",\"timestamp\":1728313616}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313616}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313616}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1396.20 ms /  3408 tokens (    0.41 ms per token,  2440.91 tokens per second)\",\"n_tokens_second\":2440.9145407743026,\"num_prompt_tokens_processed\":3408,\"slot_id\":0,\"t_prompt_processing\":1396.198,\"t_token\":0.40968251173708925,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313621}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4492.59 ms /   284 runs   (   15.82 ms per token,    63.22 tokens per second)\",\"n_decoded\":284,\"n_tokens_second\":63.21523362480602,\"slot_id\":0,\"t_token\":15.818971830985914,\"t_token_generation\":4492.588,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313621}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5888.79 ms\",\"slot_id\":0,\"t_prompt_processing\":1396.198,\"t_token_generation\":4492.588,\"t_total\":5888.786,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313621}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3692,\"n_ctx\":8192,\"n_past\":3691,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":10399,\"tid\":\"11681088\",\"timestamp\":1728313621,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":57608,\"status\":200,\"tid\":\"139974063739840\",\"timestamp\":1728313621}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313622}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313622}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1212.24 ms /  3075 tokens (    0.39 ms per token,  2536.62 tokens per second)\",\"n_tokens_second\":2536.6201330921276,\"num_prompt_tokens_processed\":3075,\"slot_id\":0,\"t_prompt_processing\":1212.243,\"t_token\":0.3942253658536585,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313627}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4402.63 ms /   294 runs   (   14.97 ms per token,    66.78 tokens per second)\",\"n_decoded\":294,\"n_tokens_second\":66.7783121304621,\"slot_id\":0,\"t_token\":14.974921768707484,\"t_token_generation\":4402.627,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313627}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5614.87 ms\",\"slot_id\":0,\"t_prompt_processing\":1212.243,\"t_token_generation\":4402.627,\"t_total\":5614.870000000001,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313627}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3369,\"n_ctx\":8192,\"n_past\":3368,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":10685,\"tid\":\"11681088\",\"timestamp\":1728313627,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":57608,\"status\":200,\"tid\":\"139974063739840\",\"timestamp\":1728313627}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313627}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313627}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1420.57 ms /  3462 tokens (    0.41 ms per token,  2437.05 tokens per second)\",\"n_tokens_second\":2437.046485500207,\"num_prompt_tokens_processed\":3462,\"slot_id\":0,\"t_prompt_processing\":1420.572,\"t_token\":0.4103327556325823,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313634}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    5474.30 ms /   342 runs   (   16.01 ms per token,    62.47 tokens per second)\",\"n_decoded\":342,\"n_tokens_second\":62.47377517149691,\"slot_id\":0,\"t_token\":16.006716374269004,\"t_token_generation\":5474.297,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313634}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    6894.87 ms\",\"slot_id\":0,\"t_prompt_processing\":1420.572,\"t_token_generation\":5474.297,\"t_total\":6894.869,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313634}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3804,\"n_ctx\":8192,\"n_past\":3803,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":10981,\"tid\":\"11681088\",\"timestamp\":1728313634,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":45758,\"status\":200,\"tid\":\"139974063759280\",\"timestamp\":1728313634}\n",
      "INFO:root:{\"function\":\"launch_slot_with_data\",\"level\":\"INFO\",\"line\":886,\"msg\":\"slot is processing task\",\"slot_id\":0,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313634}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1912,\"msg\":\"kv cache rm [p0, end)\",\"p0\":0,\"slot_id\":0,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313634}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":313,\"msg\":\"prompt eval time     =    1181.00 ms /  3018 tokens (    0.39 ms per token,  2555.47 tokens per second)\",\"n_tokens_second\":2555.467964778911,\"num_prompt_tokens_processed\":3018,\"slot_id\":0,\"t_prompt_processing\":1180.997,\"t_token\":0.39131776010603053,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313640}\n",
      "INFO:httpx:HTTP Request: POST http://127.0.0.1:8085/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":327,\"msg\":\"generation eval time =    4384.16 ms /   296 runs   (   14.81 ms per token,    67.52 tokens per second)\",\"n_decoded\":296,\"n_tokens_second\":67.51576869553833,\"slot_id\":0,\"t_token\":14.81135472972973,\"t_token_generation\":4384.161,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313640}\n",
      "INFO:root:{\"function\":\"print_timings\",\"level\":\"INFO\",\"line\":337,\"msg\":\"          total time =    5565.16 ms\",\"slot_id\":0,\"t_prompt_processing\":1180.997,\"t_token_generation\":4384.161,\"t_total\":5565.158,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313640}\n",
      "INFO:root:{\"function\":\"update_slots\",\"level\":\"INFO\",\"line\":1723,\"msg\":\"slot released\",\"n_cache_tokens\":3314,\"n_ctx\":8192,\"n_past\":3313,\"n_system_tokens\":0,\"slot_id\":0,\"task_id\":11325,\"tid\":\"11681088\",\"timestamp\":1728313640,\"truncated\":false}\n",
      "INFO:root:{\"function\":\"log_server_request\",\"level\":\"INFO\",\"line\":2766,\"method\":\"POST\",\"msg\":\"request\",\"params\":{},\"path\":\"/v1/chat/completions\",\"remote_addr\":\"127.0.0.1\",\"remote_port\":45758,\"status\":200,\"tid\":\"139974063759280\",\"timestamp\":1728313640}\n",
      "INFO:flow_judge.flow_judge:Saving results to output/\n",
      "INFO:flow_judge.utils.result_writer:Results saved to output/faithfulness/results_faithfulness_sariola__flow-judge-llamafile_llamafile_2024-10-07T15-07-20.160.jsonl\n"
     ]
    }
   ],
   "source": [
    "results = eval_pipeline.run(\n",
    "    {\n",
    "        \"sas_evaluator\": {\n",
    "            'predicted_answers': rag_answers,\n",
    "            'ground_truth_answers': ground_truths,\n",
    "        },\n",
    "        \"cr_evaluator\": {\n",
    "            'question': questions,\n",
    "            'contexts': str_retrieved_docs,\n",
    "        },\n",
    "        \"ff_evaluator\": {\n",
    "            'question': questions,\n",
    "            'contexts': str_retrieved_docs,\n",
    "            'predicted_answers': rag_answers,\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sas_evaluator': {'score': 0.8722669303417205,\n",
       "  'individual_scores': [0.8511078953742981,\n",
       "   0.9077075719833374,\n",
       "   0.8875406980514526,\n",
       "   0.881871223449707,\n",
       "   0.7968022227287292,\n",
       "   0.870733380317688,\n",
       "   0.9579464197158813,\n",
       "   0.8721626996994019,\n",
       "   0.896784782409668,\n",
       "   0.6027394533157349,\n",
       "   0.9327013492584229,\n",
       "   0.7158268690109253,\n",
       "   0.8787927627563477,\n",
       "   0.91588294506073,\n",
       "   0.9341529607772827,\n",
       "   0.91018146276474,\n",
       "   0.8761457204818726,\n",
       "   0.9175541400909424,\n",
       "   0.9164606332778931,\n",
       "   0.9222434163093567]},\n",
       " 'cr_evaluator': {'results': [{'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 2},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 2},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5}],\n",
       "  'metadata': {'model_id': 'sariola/flow-judge-llamafile',\n",
       "   'model_type': 'llamafile'},\n",
       "  'score': 4.7,\n",
       "  'individual_scores': [5.0,\n",
       "   2.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   2.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0]},\n",
       " 'ff_evaluator': {'results': [{'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 5},\n",
       "   {'feedback': '', 'score': 4},\n",
       "   {'feedback': '', 'score': 5}],\n",
       "  'metadata': {'model_id': 'sariola/flow-judge-llamafile',\n",
       "   'model_type': 'llamafile'},\n",
       "  'score': 4.7,\n",
       "  'individual_scores': [5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   4.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   4.0,\n",
       "   4.0,\n",
       "   5.0,\n",
       "   4.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   4.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   5.0,\n",
       "   4.0,\n",
       "   5.0]}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation report\n",
    "\n",
    "Haystack provides a convenient way to generate an evaluation report using the `EvaluationRunResult` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metrics</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>sas_evaluator</td>\n",
       "      <td>0.872267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cr_evaluator</td>\n",
       "      <td>4.650000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ff_evaluator</td>\n",
       "      <td>4.700000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         metrics     score\n",
       "0  sas_evaluator  0.872267\n",
       "1   cr_evaluator  4.650000\n",
       "2   ff_evaluator  4.700000"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from haystack.evaluation.eval_run_result import EvaluationRunResult\n",
    "\n",
    "inputs = {\n",
    "    \"question\": questions,\n",
    "    \"contexts\": str_retrieved_docs,\n",
    "    \"answer\": ground_truths,\n",
    "    \"predicted_answer\": rag_answers,\n",
    "}\n",
    "\n",
    "evaluation_result = EvaluationRunResult(run_name=\"report\", inputs=inputs, results=results)\n",
    "evaluation_result.score_report()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also easily conver to a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>contexts</th>\n",
       "      <th>answer</th>\n",
       "      <th>predicted_answer</th>\n",
       "      <th>sas_evaluator</th>\n",
       "      <th>cr_evaluator</th>\n",
       "      <th>ff_evaluator</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Does CNN permit using bots to artificially inc...</td>\n",
       "      <td>The Site contains copyrighted material, tradem...</td>\n",
       "      <td>No, CNN does not permit using bots to artifici...</td>\n",
       "      <td>No, CNN does not permit the use of any technol...</td>\n",
       "      <td>0.824386</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Does eBay promise that the product data it pro...</td>\n",
       "      <td>When you provide content using our Services (d...</td>\n",
       "      <td>No, eBay does not promise that the product dat...</td>\n",
       "      <td>Yes, eBay does acknowledge and warrant that th...</td>\n",
       "      <td>0.922890</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Will certain terms remain in force notwithstan...</td>\n",
       "      <td>16. Termination\\nYou may terminate these Terms...</td>\n",
       "      <td>Yes, certain terms will remain in force notwit...</td>\n",
       "      <td>Yes, certain terms will remain in force even i...</td>\n",
       "      <td>0.879336</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Where a Netflix subscriber requires customer s...</td>\n",
       "      <td>8. Miscellaneous\\n\\n8.1. Governing Law. These ...</td>\n",
       "      <td>Yes, according to the provided context, there ...</td>\n",
       "      <td>Yes, according to the provided context, if a N...</td>\n",
       "      <td>0.903150</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Can Apple disclose personal information about ...</td>\n",
       "      <td>Violation of These Terms of Use\\nApple may dis...</td>\n",
       "      <td>Yes, Apple can disclose personal information a...</td>\n",
       "      <td>Yes, Apple can disclose personal information a...</td>\n",
       "      <td>0.896643</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  Does CNN permit using bots to artificially inc...   \n",
       "1  Does eBay promise that the product data it pro...   \n",
       "2  Will certain terms remain in force notwithstan...   \n",
       "3  Where a Netflix subscriber requires customer s...   \n",
       "4  Can Apple disclose personal information about ...   \n",
       "\n",
       "                                            contexts  \\\n",
       "0  The Site contains copyrighted material, tradem...   \n",
       "1  When you provide content using our Services (d...   \n",
       "2  16. Termination\\nYou may terminate these Terms...   \n",
       "3  8. Miscellaneous\\n\\n8.1. Governing Law. These ...   \n",
       "4  Violation of These Terms of Use\\nApple may dis...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  No, CNN does not permit using bots to artifici...   \n",
       "1  No, eBay does not promise that the product dat...   \n",
       "2  Yes, certain terms will remain in force notwit...   \n",
       "3  Yes, according to the provided context, there ...   \n",
       "4  Yes, Apple can disclose personal information a...   \n",
       "\n",
       "                                    predicted_answer  sas_evaluator  \\\n",
       "0  No, CNN does not permit the use of any technol...       0.824386   \n",
       "1  Yes, eBay does acknowledge and warrant that th...       0.922890   \n",
       "2  Yes, certain terms will remain in force even i...       0.879336   \n",
       "3  Yes, according to the provided context, if a N...       0.903150   \n",
       "4  Yes, Apple can disclose personal information a...       0.896643   \n",
       "\n",
       "   cr_evaluator  ff_evaluator  \n",
       "0           5.0           5.0  \n",
       "1           4.0           2.0  \n",
       "2           5.0           4.0  \n",
       "3           5.0           5.0  \n",
       "4           5.0           5.0  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = evaluation_result.to_pandas()\n",
    "results_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, we demonstrated how to evaluate a Retrieval-Augmented Generation (RAG) pipeline using `FlowJudge` and Haystack. Key aspects covered include:\n",
    "\n",
    "1. Setting up a basic RAG pipeline with Haystack:\n",
    "   - Using `InMemoryDocumentStore` for document storage\n",
    "   - Implementing `SentenceTransformersDocumentEmbedder` for document embedding\n",
    "   - Utilizing `HuggingFaceAPIChatGenerator` for answer generation\n",
    "\n",
    "2. Creating custom evaluators with `FlowJudge`:\n",
    "   - Developing a custom metric for context relevancy\n",
    "   - Adapting a preset metric for faithfulness\n",
    "   - Using the `HaystackFlowJudge` class to integrate FlowJudge evaluators into Haystack\n",
    "\n",
    "3. Building a comprehensive evaluation pipeline:\n",
    "   - Incorporating both FlowJudge and native Haystack evaluators\n",
    "   - Using `SASEvaluator` for semantic answer similarity\n",
    "\n",
    "4. Executing the evaluation and analyzing results:\n",
    "   - Running the evaluation pipeline on a subset of questions\n",
    "   - Utilizing `EvaluationRunResult` to generate a summary report\n",
    "   - Converting results to a pandas DataFrame for further analysis\n",
    "\n",
    "5. Demonstrating the flexibility of `FlowJudge`:\n",
    "   - Seamless integration with Haystack's evaluation framework\n",
    "   - Ability to create custom metrics and adapt existing ones\n",
    "   - Using open-source models to avoid reliance on proprietary large language models\n",
    "\n",
    "This tutorial showcases how `FlowJudge` can be effectively used to evaluate and iteratively improve RAG pipelines built with Haystack, providing a comprehensive assessment of performance across multiple dimensions including semantic similarity, context relevancy, and faithfulness."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "flow-judge-test",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}