02_Geoparser_FineTuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ead637c6-f8f4-44fc-a6f7-7e28d38b2256",
   "metadata": {},
   "source": [
    "# Irchel Geoparser Workshop: Fine-Tuning a Transformer Model\n",
    "\n",
    "In this second notebook, we will learn how to fine-tune the transformer model used by Geoparser for specific text corpora. This tutorial will guide you through the process of preparing training data, training a custom model, and evaluating its performance.\n",
    "\n",
    "**Objectives:**\n",
    "\n",
    "- Learn how to annotate text data using the Geoparser Annotator web app.\n",
    "- Understand how to prepare annotated data for model training and testing.\n",
    "- Fine-tune the Geoparser model using custom annotations.\n",
    "- Evaluate and compare the performance of the base model and the fine-tuned model.\n",
    "\n",
    "---\n",
    "\n",
    "## 📖 **Documentation**\n",
    "\n",
    "This tutorial requires you to consult the Geoparser documentation: [docs.geoparser.app](https://docs.geoparser.app). Knowing where to find information about the library is crucial for working with it effectively, especially as it is likely that future updates may bring changes to functionality. The documentation will always be the definitive reference for how to use the package.\n",
    "\n",
    "---\n",
    "\n",
    "## 1. Exploring the Geoparser Annotator\n",
    "\n",
    "The Geoparser Annotator is a browser-based tool for creating annotated datasets for training and evaluation. In this section, you will launch the Annotator app, upload texts, and familiarize yourself with its features.\n",
    "\n",
    "### 1.1 Launch the Geoparser Annotator\n",
    "\n",
    "**Objective:** Start the Geoparser Annotator web app on your device.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Open your terminal or command prompt.\n",
    "- Run the command below to launch the Annotator.\n",
    "- Once the app starts, it will automatically open in your browser on `http://127.0.0.1:5000/`.\n",
    "- Familiarize yourself with the Annotator interface.\n",
    "\n",
    "**Hints:**\n",
    "- Ensure that you run the launch command in an environment where you have `geoparser` installed.\n",
    "\n",
    "**Launch command:**\n",
    "```bash\n",
    "python -m geoparser annotator\n",
    "```\n",
    "You can close with command c\n",
    "### 1.2 Upload and Annotate Texts\n",
    "\n",
    "**Objective:** Upload texts to the annotator and familiarize yourself with the annotation process.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Upload the `.txt` files from the previous tutorial to the Annotator.\n",
    "- Correct the pre-recognised toponyms:\n",
    "  - Delete invalid toponyms.\n",
    "  - Adjust faulty string boundaries.\n",
    "  - Tag missed toponyms.\n",
    "- Annotate toponym locations:\n",
    "  - Infer the locations from the surrounding context.\n",
    "  - Use the search and filter functionality to find the correct locations in the database.\n",
    "  - Annotate the location.\n",
    "- Explore the different features:\n",
    "  - Play around with session settings.\n",
    "  - Add and remove documents.\n",
    "  - Download the annotation session file.\n",
    "  - Restart the Annotator and resume a previous session (from file or cache).\n",
    "\n",
    "**Note:** This initial tinkering is to get you comfortable with the Annotator. The quality of annotations is not important for this task.\n",
    "\n",
    "---\n",
    "\n",
    "## 2. Preparing Training Data\n",
    "\n",
    "To fine-tune the transformer model, we need annotated data. We will fine-tune a transformer using annotated tweets to optimize its performance on a tweet corpus.\n",
    "\n",
    "### 2.1 Complete the Annotations\n",
    "\n",
    "**Objective:** Finish annotating the provided annotation session to create a training dataset.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Download the provided annotation session file (`training_annotations_incomplete.json`) that contains tweets with most toponyms already annotated.\n",
    "- Upload the file to the Annotator.\n",
    "- Finish annotating the remaining texts.\n",
    "- Download the final annotations file (e.g. as `training_annotations.json`).\n",
    "\n",
    "### 2.2 Initialize GeoparserTrainer\n",
    "\n",
    "**Objective:** Initialize the GeoparserTrainer with the transformer model we want to fine-tune.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Import the necessary class to use GeoparserTrainer.\n",
    "- Initialize an instance of the `GeoparserTrainer` class.\n",
    "- Specify `dguzh/geo-all-MiniLM-L6-v2` as the transformer model to be fine-tuned.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c1041c5-8359-41e4-96d5-b2fdc24039eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4829fb31-c098-4b61-88ce-78e29a4a514f",
   "metadata": {},
   "source": [
    "### 2.3 Load Training Data into GeoparserTrainer\n",
    "\n",
    "**Objective:** Load your annotated training data into GeoparserTrainer.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Load the completed training annotations file into the GeoparserTrainer.\n",
    "- Store the prepared training documents in a variable.\n",
    "- Use `include_unmatched=True` to avoid excluding annotations not recognized by spaCy.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1fa95bf6-64e7-4d39-90b7-0957867bf3d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from geoparser import GeoparserTrainer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0dc28d5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "trainer = GeoparserTrainer(\n",
    "    spacy_model=\"en_core_web_trf\",\n",
    "    transformer_model=\"dguzh/geo-all-distilroberta-v1\",\n",
    "    gazetteer=\"geonames\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e960c01a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5751e9e076d41439aa4d2080ef73f6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1282 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_docs = trainer.annotate(\"test_annotations.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66cc78c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/skalli-adm/Library/Mobile Documents/com~apple~CloudDocs/Documents/GitHub Repositories/geoparser-workshop\n"
     ]
    }
   ],
   "source": [
    "#define path of this notebook using pathlib. NameError: name '__file__' is not defined\n",
    "from pathlib import Path\n",
    "path = Path.cwd()\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b3a37e44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e35e5929dc794b3789f1abb658b08341",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1282 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[10], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Train model\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[43mtrainer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrain_docs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput_path\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpath/model\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m8\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/trainer/trainer.py:346\u001b[0m, in \u001b[0;36mGeoparserTrainer.train\u001b[0;34m(self, train_docs, output_path, epochs, batch_size)\u001b[0m\n\u001b[1;32m    330\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mtrain\u001b[39m(\n\u001b[1;32m    331\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m    332\u001b[0m     train_docs: t\u001b[38;5;241m.\u001b[39mList[GeoDoc],\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    335\u001b[0m     batch_size: \u001b[38;5;28mint\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m8\u001b[39m,\n\u001b[1;32m    336\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    337\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    338\u001b[0m \u001b[38;5;124;03m    Train the toponym disambiguation model using the prepared training data.\u001b[39;00m\n\u001b[1;32m    339\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    344\u001b[0m \u001b[38;5;124;03m        batch_size (int, optional): Training batch size. Defaults to 8.\u001b[39;00m\n\u001b[1;32m    345\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 346\u001b[0m     train_dataset \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_prepare_training_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrain_docs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    348\u001b[0m     train_loss \u001b[38;5;241m=\u001b[39m losses\u001b[38;5;241m.\u001b[39mContrastiveLoss(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtransformer)\n\u001b[1;32m    350\u001b[0m     training_args \u001b[38;5;241m=\u001b[39m SentenceTransformerTrainingArguments(\n\u001b[1;32m    351\u001b[0m         output_dir\u001b[38;5;241m=\u001b[39moutput_path,\n\u001b[1;32m    352\u001b[0m         num_train_epochs\u001b[38;5;241m=\u001b[39mepochs,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    355\u001b[0m         save_strategy\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mno\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    356\u001b[0m     )\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/trainer/trainer.py:311\u001b[0m, in \u001b[0;36mGeoparserTrainer._prepare_training_data\u001b[0;34m(self, docs)\u001b[0m\n\u001b[1;32m    308\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m correct_location:\n\u001b[1;32m    310\u001b[0m     candidate_ids \u001b[38;5;241m=\u001b[39m toponym\u001b[38;5;241m.\u001b[39mget_candidates()\n\u001b[0;32m--> 311\u001b[0m     candidate_locations \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgazetteer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquery_locations\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcandidate_ids\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    313\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m candidate_location \u001b[38;5;129;01min\u001b[39;00m candidate_locations:\n\u001b[1;32m    314\u001b[0m         description \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgazetteer\u001b[38;5;241m.\u001b[39mget_location_description(\n\u001b[1;32m    315\u001b[0m             candidate_location\n\u001b[1;32m    316\u001b[0m         )\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/gazetteers/gazetteer.py:746\u001b[0m, in \u001b[0;36mLocalDBGazetteer.query_locations\u001b[0;34m(self, location_ids, batch_size)\u001b[0m\n\u001b[1;32m    740\u001b[0m placeholders \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m,\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m?\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m _ \u001b[38;5;129;01min\u001b[39;00m batch)\n\u001b[1;32m    741\u001b[0m query \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m    742\u001b[0m \u001b[38;5;124m    SELECT *\u001b[39m\n\u001b[1;32m    743\u001b[0m \u001b[38;5;124m    FROM locations\u001b[39m\n\u001b[1;32m    744\u001b[0m \u001b[38;5;124m    WHERE \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mlocation_identifier\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m IN (\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mplaceholders\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m)\u001b[39m\n\u001b[1;32m    745\u001b[0m \u001b[38;5;124m\u001b[39m\u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[0;32m--> 746\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute_query\u001b[49m\u001b[43m(\u001b[49m\u001b[43mquery\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    748\u001b[0m columns \u001b[38;5;241m=\u001b[39m [col\u001b[38;5;241m.\u001b[39mname \u001b[38;5;28;01mfor\u001b[39;00m col \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig\u001b[38;5;241m.\u001b[39mlocation_columns]\n\u001b[1;32m    749\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m row \u001b[38;5;129;01min\u001b[39;00m results:\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/gazetteers/gazetteer.py:117\u001b[0m, in \u001b[0;36mLocalDBGazetteer.close.<locals>.call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    116\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcall\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m--> 117\u001b[0m     ret \u001b[38;5;241m=\u001b[39m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    118\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_close_connection()\n\u001b[1;32m    119\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m ret\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/gazetteers/gazetteer.py:82\u001b[0m, in \u001b[0;36mLocalDBGazetteer.connect.<locals>.call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     80\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcall\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m     81\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_initiate_connection()\n\u001b[0;32m---> 82\u001b[0m     ret \u001b[38;5;241m=\u001b[39m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     83\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m ret\n",
      "File \u001b[0;32m~/miniforge3/envs/geoparser/lib/python3.10/site-packages/geoparser/gazetteers/gazetteer.py:170\u001b[0m, in \u001b[0;36mLocalDBGazetteer.execute_query\u001b[0;34m(self, query, params)\u001b[0m\n\u001b[1;32m    168\u001b[0m cursor \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_cursor()\n\u001b[1;32m    169\u001b[0m cursor\u001b[38;5;241m.\u001b[39mexecute(query, params \u001b[38;5;129;01mor\u001b[39;00m ())\n\u001b[0;32m--> 170\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mcursor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfetchall\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Train model\n",
    "trainer.train(train_docs, output_path=\"path/model\", epochs=1, batch_size=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d07008d1-1d5b-4a2f-8355-6ca82b99aeee",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Evaluating the Base Model\n",
    "\n",
    "Before fine-tuning, we'll evaluate the performance of the base model using the test dataset.\n",
    "\n",
    "### 3.1 Load Test Data from the Provided JSON File\n",
    "\n",
    "**Objective:** Load the test annotations to use for model evaluation.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- We've provided a fully annotated test annotations file (`test_annotations.json`).\n",
    "- Load them into the GeoparserTrainer in the same way as the training data. Store the test documents in a separate variable.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8f963e-d189-4fe3-9884-ea0814b6ad8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "455efe7e-05d7-4080-9600-da3b0ea72976",
   "metadata": {},
   "source": [
    "### 3.2 Evaluate the Base Model\n",
    "\n",
    "**Objective:** Evaluate the performance of the base model using the test data.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Resolve the toponyms in the test dataset to generate predictions for them using the base model.\n",
    "- Evaluate these predictions to assess the model's performance.\n",
    "- Store the evaluation results for later comparison.\n",
    "- Print the evaluation results.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f076aaee-2591-4361-9cfc-1e5974d6f2fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1483f8b8-c1b0-464f-8344-7739232ce41f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Fine-Tuning the Model\n",
    "\n",
    "Now we'll fine-tune the Geoparser model using our annotated training data.\n",
    "\n",
    "### 4.1 Train the Model With the Prepared Training Data\n",
    "\n",
    "**Objective:** Fine-tune the Geoparser model with your training data.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Use the initialized `GeoparserTrainer` to train the loaded transformer model using the prepared training documents.\n",
    "- Specify an output path where the fine-tuned model will be saved (e.g., `\"fine_tuned_model\"`).\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1381d395-1427-4397-a8b4-1f6e5fb56346",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3716148-cef9-47dc-82e6-c53a3bd57fce",
   "metadata": {},
   "source": [
    "### 4.2 Evaluate the Fine-Tuned Model\n",
    "\n",
    "**Objective:** Evaluate the performance of the fine-tuned model using the test data.\n",
    "\n",
    "**Instructions:**\n",
    "\n",
    "- Follow the same procedure you used to evaluate the base model.\n",
    "- Use the same test documents you used for evaluating the base model.\n",
    "\n",
    "**Hints:**\n",
    "- After training, the original transformer model loaded in the GeoparserTrainer is automatically replaced with the fine-tuned version.\n",
    "- Resolving toponyms in previously processed GeoDoc objects will overwrite existing predictions with those generated by the new model.\n",
    "\n",
    "**Your Code:**\n",
    "\n",
    "*(Write your code in the cell below.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32f16b43-8789-4661-8e33-452b0403d315",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b1123da-bcdc-4895-9c63-9542563a929d",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Comparing Model Performance\n",
    "\n",
    "Finally, we'll compare the evaluation results of the base model and the fine-tuned model to see if there's an improvement.\n",
    "\n",
    "### 5.1 Compare the Results Before and After Fine-Tuning\n",
    "\n",
    "**Objective:** Analyze the evaluation metrics to determine the impact of fine-tuning.\n",
    "\n",
    "**Analyze the Results:**\n",
    "\n",
    "- Compare the evaluation metrics obtained from both models.\n",
    "- Consider the costs and benefits of fine-tuning a model, and evaluate the scenarios in which it makes sense to undertake this process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3861e39-1f0c-4f93-8b2d-166152addfb6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "geoparser",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}