03a_pydantic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing Pydantic: Pythonic Data Validation\n",
    "\n",
    "### Goal of this notebook \n",
    "This notebook will introduce you to [Pydantic](https://docs.pydantic.dev/latest/), a data validation library that provides a way to define data schemas in a way that is both Pythonic and easy to use. \n",
    "\n",
    "**Note:** *We are using pydantic 2 in this tutorial! Many of the example will not work with pydantic 1.*\n",
    "\n",
    "\n",
    "### Steps you will take in this notebook\n",
    "\n",
    "1. Review creating a class in Python.\n",
    "2. Learn about data classes in Python.\n",
    "3. Learn how to write a class in Pydantic that enforces data types and constraints\n",
    "4. Very briefly introduce `ipyautoui` to uncover an issue with our model of the controls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What problem are we trying to solve?\n",
    "\n",
    "The dashboard we have built so far works, but would be hard to include as part of a larger project, and its individual components would be difficult to reuse. For example, the controls to select a range of years and apply smoothing to the data would be useful in making other graphs. As the code stand now, it would be hard to take those pieces as a unit and integrate them into a different dashboard.\n",
    "\n",
    "Put differently, one might want a `DataSelector` widget that has as its value a range of years, a smoothing windows, and a polynomial order for the smoothing that can be reused.\n",
    "\n",
    "### Our route to the solution is indirect\n",
    "\n",
    "In this notebook we will define a class using Pydantic that does not, by itself, have any widgets attached to it. In the next notebook we will use `ipyautoui` to generate a widget-based user interface from our Pydantic class that will be much easier to reuse.\n",
    "\n",
    "### Goals for our pydantic `DataSelector` class:\n",
    "\n",
    "The class should have these attributes (also called *fields* in the pydantic documentation):\n",
    "\n",
    "+ `year_range`, the range of time selected.\n",
    "+ `window_size`, the size of the smoothing window.\n",
    "+ `polynomial_order`, the order of the polynomial used in smoothing.\n",
    "\n",
    "These attributes also have some important constraints:\n",
    "\n",
    "+ The `window_size` should be an integer larger than one and, to match our earlier example, less than or equal to 100.\n",
    "+ The `polynomial_order` should be an integer less than or equal to 10, and less than the `window_size`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We import our dashboard below to get the magics from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (optional) [Review of Python classes and a (First Draft) `DataSelector` Class](03az_OPTIONAL_intro_classes.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Constructing a (Second Draft) `DataSelector` Class with `dataclass`\n",
    "Writing a class with multiple attributes gets repetitive in Python. Each attribute typically comes with an argument to the class and lines of boilerplate code to set the attributes of the class to the those arguments.\n",
    "\n",
    "Data classes were introduced in Python 3.7 to make that sort of code more compact to write. They leverage *type annotations*, which were added to the language in version 3.0, and allow you to provide some information about the type of a variable.\n",
    "\n",
    "The class below is an implemenation, using data classes, of part of the `DataSelectorPlainPython` we wrote above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "\n",
    "@dataclass \n",
    "class DataSelectorDataClass:\n",
    "    \"\"\"\n",
    "    Partial implementation of a class to hold a data selector widget using dataclasses.\n",
    "    \"\"\"\n",
    "    year_range: tuple = (1800, 2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is already much more compact than the initial class definition above. Python automatically creates an `__init__` method for this class that sets the class up. It also comes with some extras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selector_dc = DataSelectorDataClass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(selector_dc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare this to what we get for our plain Python class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(selector_plain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will see in a few minutes, in addition to getting a nice string representation of the object for free, we also get the ability to test for equality of two instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "1. Extend `DataSelectorDataClass` so that it also has a `window_size` with a default value of 2 and a `polynomial_order` with a default value of 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/09.py\n",
    "\n",
    "@dataclass\n",
    "class DataSelectorDataClass:\n",
    "    \"\"\"\n",
    "    A class to hold a data selector widget using dataclasses.\n",
    "    \"\"\"\n",
    "    year_range: tuple = (1800, 2000)\n",
    "    # Add your code below -- remember to include the default values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Recall that testing for equality did not work the way we wanted for the plain Python version of our data selector. Try comparing the two selectors below with each other and with `selector_dc` to see if equality testing works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_dc_2 = DataSelectorDataClass(year_range=(1991, 2018))\n",
    "sel_dc_3 = DataSelectorDataClass(year_range=(1991, 2018))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/11.py\n",
    "\n",
    "# Put your comparison here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data class summary\n",
    "\n",
    "Using `dataclass` to define our selector has several advantages:\n",
    "\n",
    "+ It is less code.\n",
    "+ It has a human-readable string representation.\n",
    "+ You can check whether instances of the class are equal.\n",
    "\n",
    "We could have done those last two things without `dataclass` by defining a couple of special methods methods in our class definition. However, it's really nice to just have it happen automatically behind the scenes, though!\n",
    "\n",
    "*Note:* There is much more to data classes than we have covered. Read more about them in this [Real Python tutorial](https://realpython.com/python-data-classes/) or in the [Python documentation](https://docs.python.org/3/library/dataclasses.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Progress check\n",
    "\n",
    "The data class was relatively straightforward to write and looks promising for representing our controls:\n",
    "\n",
    "```python\n",
    "@dataclass \n",
    "class DataSelectorDataClass:\n",
    "    \"\"\"\n",
    "    Partial implementation of a class to hold a data selector widget using dataclasses.\n",
    "    \"\"\"\n",
    "    year_range: tuple = (1800, 2000)\n",
    "    window_size: int = 2\n",
    "    polynomial_order: int = 1\n",
    "```\n",
    "\n",
    "There are a couple of issues, though:\n",
    "\n",
    "1. You can set any of the attributes to whatever value you want. This will raise no errors: ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataSelectorDataClass(year_range=5, window_size=\"three\", polynomial_order=-3.14159)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. None of the contraints we wanted are enforced.\n",
    "\n",
    "Pydantic will help us solve these issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing a `DataSelector` Class with Pydantic\n",
    "\n",
    "The `pydantic` library solves several of our problems and gets us a few more abilities for free:\n",
    "\n",
    "+ It is designed to help enforce type requirements. It can do its best to convert values for you, or not if you prefer that.\n",
    "+ Simple constraints like \"this number must be greater than or equal to two\" are easy to express.\n",
    "+ More complicated constraints like \"this number must be smaller than this other one\" are possible to express.\n",
    "+ It is straightforward to save objects to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| default_exp pydantic_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a class using pydantic\n",
    "\n",
    "One way to use pydantic to make a class is to import a class called `BaseModel` from it and subclass that. Classes made using pydantic are often called *models*, a term we will use for pydantic-based classes. It ends up looking a lot like a data class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will export the proper imports shortly, once we are closer to a final model\n",
    "from pydantic import BaseModel\n",
    "\n",
    "class DataSelectorModelDraft1(BaseModel):\n",
    "    year_range: tuple = (1800, 2000)\n",
    "    window_size: int = 2\n",
    "    polynomial_order: int = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like a data class, attributes are defined by adding a type annotation. Unlike data classes, pydantic enforces types. Try running the cell below, which will raise an exception:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%exception\n",
    "\n",
    "selector_pyd = DataSelectorModelDraft1(year_range=5, window_size=\"three\", polynomial_order=-3.14159)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pydantic includes fairly detailed error messages for validation errors. Many users could read this and update their input appropriately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "1. Make a valid instance of `DataSelectorModelDraft1`, i.e. an instance that does not raise an error when you create it. Feel free to try to come up with an instances that might surprise other people."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/13.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More ways to make an instance of a pydantic model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There a few ways of making an instance of a pydantic model:\n",
    "\n",
    "1. Provide arguments when you call the class; this is what we did above.\n",
    "2. From a dictionary of values, using the class method `model_validate`.\n",
    "3. From json.\n",
    "\n",
    "We will come back to the third way later in the notebook. An example of the second way is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_choices = {\n",
    "    \"year_range\": (1900, 1950),\n",
    "    \"window_size\": 10,\n",
    "    \"polynomial_order\": 2,\n",
    "}\n",
    "\n",
    "DataSelectorModelDraft1.model_validate(my_choices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imposing constraints after object creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, pydantic imposes its constraints only when you create the object. Consider this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selector_pyd_simple = DataSelectorModelDraft1()\n",
    "selector_pyd_simple.window_size = \"two\"\n",
    "selector_pyd_simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `window_size` has been set to a string, not an integer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, pydantic can be configured to check types when values are assigned by using the `validate_assignment` configuration. There are many more options available in [pydantic configuration](https://docs.pydantic.dev/latest/concepts/config/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataSelectorModelDraft2(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple = (1800, 2000)\n",
    "    window_size: int = 2\n",
    "    polynomial_order: int = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now an exception is raised when we try to assign a string to `window_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%exception\n",
    "\n",
    "selector_pyd_simple2 = DataSelectorModelDraft2()\n",
    "selector_pyd_simple2.window_size = \"two\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More specific constraints on types\n",
    "\n",
    "You might be surprised to see that the line below raises no error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selector_pyd_simple2.year_range = (\"eightteen eighty 5\", 8+5j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason is that pydantic simply checks to see that a tuple is being assigned to `year_range` -- the contents of the tuple can be anything at all. This would also raise no errors: `selector_pyd_simple2.year_range = (1, 2, 3)`\n",
    "\n",
    "Python type annotations provide a way to provide information about what the tuple should consist of by putting the contents in square brackets, as shown in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataSelectorModelDraft3(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: int = 2\n",
    "    polynomial_order: int = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this change, trying to assign `(\"eightteen eighty 5\", 8+5j)` to `window_size` will fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%exception\n",
    "\n",
    "selector_pyd_simple3 = DataSelectorModelDraft3()\n",
    "selector_pyd_simple3.year_range = (\"eightteen eighty 5\", 8+5j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imposing constraints on field values\n",
    "\n",
    "**Typing** refers to Python's type hinting system that allows you to specify the expected types of variables, function parameters, and return values. Also known as *type hinting*, this feature of Python helps reduce errors and enhance code readability. Later in the tutorial, we will learn more about a particular kind of type hinting called *type aliasing*, which will help us make more complex types easier to reuse and manage.\n",
    "\n",
    "*Attributes in a pydantic model are typically called fields, terminology we will use for the remainder of the notebook.*\n",
    "\n",
    "We have made some progress but we still have not imposed the constraints we want on window size and polynomial order. There are a couple new things we need to do that:\n",
    "\n",
    "+ `Annotated` from Python's typing system lets you add additional information about the type of an item. Here we will use it to add information about the constraint on a field.\n",
    "+ The `Field` class from pydantic is a class you use that contains that extra information. There are a number of possible arguments to `Field`. Here we use `ge`, short for \"greater than or equal to,\" to impose the constraint that the `window_size` be larger greater than or equal to 2. The `Field` class from `pydantic` is somewhat similar to the [`field` class from Python's data classes](https://docs.python.org/3/library/dataclasses.html#dataclasses.field) which also serves the purpose of adding information about a typed field.\n",
    "\n",
    "In the cell below we define a pydantic model that imposes the constraint that the `window_size` must be greater than or equal to 2. It does that by annotating the `window_size` type, `int`, with `Field(ge=2)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Annotated\n",
    "from pydantic import Field\n",
    "\n",
    "class DataSelectorModelDraft4(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2)] = 2\n",
    "    polynomial_order: int = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test this by creating an instance and setting the `window_size` to an integer value that should not be allowed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%exception\n",
    "\n",
    "selector_pyd_simple4 = DataSelectorModelDraft4()\n",
    "\n",
    "selector_pyd_simple4.window_size = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the `window_size` also had an upper limit of 100 in the earlier dashboard we are trying to reproduce. This version of the class adds that upper limit also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataSelectorModelDraft5(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2, le=100)] = 2\n",
    "    polynomial_order: int = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "Add a constraint on the `polynomial_order` that requires is to be greater than or equal to 1 and less than or equal to 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/15.py\n",
    "\n",
    "class DataSelectorModelDraft5(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2, le=100)] = 2\n",
    "    # Modify the line below\n",
    "    polynomial_order: int = 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these changes we have some of the constraints we want. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imposing constraints on multiple fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one more thing we need to do: the polynomial order should be at least 1 less than the window size in addition to being 10 or smaller.\n",
    "\n",
    "To do that we will add a *model validator* to our pydantic class. The model validator has access to all of the proposed model values and can check them in whatever way it wants. If the values are acceptable then the method returns `self`. If the values are not acceptable then the validator should raise a `ValueError`.\n",
    "\n",
    "A draft class with the model validator is below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a little experience with Pydantic we will start exporting the cells we need to make this part of our dashboard package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "from typing import Annotated\n",
    "from pydantic import model_validator, BaseModel, Field"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not quite ready to export our pydantic model -- we will do that later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataSelectorModelDraft6(BaseModel, validate_assignment=True):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2, le=100)] = 2\n",
    "    polynomial_order: Annotated[int, Field(ge=1, le=10)] = 1\n",
    "\n",
    "    # mode=\"after\" means the validator runs after pydantic has checked that the individual\n",
    "    # fields have values that are valid.\n",
    "    @model_validator(mode=\"after\")\n",
    "    def limit_polynomial_order(self):\n",
    "        \n",
    "        if self.polynomial_order > self.window_size - 1:\n",
    "            # Handle a bad polynomial order or window size\n",
    "            raise ValueError(\"Polynomial order must be smaller than window size\")\n",
    "            \n",
    "        # If we got this far the polynomial order is consistent with the window size\n",
    "        # so return self. Failing to return self will end up causing an error.\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "1. Check whether the new validation works by creating a valid `DataSelectorModelDraft6` and then trying to set the `polynomial_order` or `window_size` to inconsistent values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/17.py\n",
    "\n",
    "# Break things here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional benefits of using Pydantic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Easy to save model value to a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One additional benefit of using pydantic to model our control is that pydantic classes come with easy conversion to json, which is in turn easy to save to disk.\n",
    "\n",
    "We make a model in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DataSelectorModelDraft6()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert this model to a couple of different forms:\n",
    "\n",
    "+ The `model_dump` method converts the pydantic model to a dictionary of values.\n",
    "+ The `model_dump_json` method converts the pydantic model to json with the model's values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.model_dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the indent argument causes the json to have line breaks, with the indentation\n",
    "# of each new level given by indent\n",
    "print(model.model_dump_json(indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might not be surprised to learn that you can also create a model instance from json. In the cell below, we take the json from `model` and use it to create a new instance.\n",
    "\n",
    "To do that you use the class method `model_validate_json` to make the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_json = model.model_dump_json()\n",
    "\n",
    "# Use the class method model_validate_json to make a new model\n",
    "new_model = DataSelectorModelDraft6.model_validate_json(model_json)\n",
    "\n",
    "new_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"my_selections.json\", \"w\") as f:\n",
    "    f.write(model_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though we will not have occasion to use it much in this tutorial, the next cell shows how to load the json from disk make a model from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"my_selections.json\") as f:\n",
    "    disk_json = f.read()\n",
    "\n",
    "DataSelectorModelDraft6.model_validate_json(disk_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text representation of the model\n",
    "\n",
    "The `model_json_schema` method produces a json schema for the model, which is a description of the model and its restrictions.\n",
    "\n",
    "While we will not have much occasion to use the json schema directly, it is really important to understand how the next package we will take a look at, `ipyautoui`, generates a user interface from a pydantic model.\n",
    "\n",
    "The json schema for `DataSelectorModelDraft6` is below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add extra keys to the json schema for a field by adding them a dictionary and using that dictionary as an argument in `Field`. The example below adds a key to the schema for `polynomial_order` noting that its value should be constrained by `window_size`, information not currently in the json schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a dictionary....\n",
    "schema_extra = dict(constraint_note=\"Polynomial order should be less than window size\")\n",
    "\n",
    "\n",
    "class DataSelectorModelDraftWithExtra(BaseModel):\n",
    "    year_range: tuple[int, int] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2, le=100)] = 2\n",
    "    # ...provide the dictionary to json_schema_extra\n",
    "    polynomial_order: Annotated[int, Field(ge=1, le=10, json_schema_extra=schema_extra)] = 1\n",
    "\n",
    "    # mode=\"after\" means the validator runs after pydantic has checked that the individual\n",
    "    # fields have values that are valid.\n",
    "    @model_validator(mode=\"after\")\n",
    "    def limit_polynomial_order(self):\n",
    "        \n",
    "        if self.polynomial_order > self.window_size - 1:\n",
    "            # Handle a bad polynomial order or window size\n",
    "            raise ValueError(\"Polynomial order must be smaller than window size\")\n",
    "            \n",
    "        # If we got this far the polynomial order is consistent with the window size\n",
    "        # so return self. Failing to return self will end up causing an error.\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataSelectorModelDraftWithExtra.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A problem with the data selector so far\n",
    "\n",
    "There is a problem with our data selector class that is not obvious. To expose, we are going to use `ipyautoui`, a package that can generate a widget from a Pydantic model, to make a user interface from our model.\n",
    "\n",
    "We will talk about ipyautoui in more detail in the next part of the tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyautoui import AutoUi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AutoUi(DataSelectorModelDraft6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two surprising things here. The first is that there was a validation error for `year_range`. The second is that the control for `year_range` does not look anything like a way to enter a range of years.\n",
    "\n",
    "Resolving the second issue will also, it turns out, address the first.\n",
    "\n",
    "The problem here is that our description of `year_range` is incomplete. Right now we annotate `year_range` as `tuple[int, int]`. That isn't quite right, since there is a constraint on both the first and second integer. Each must be in the range of years covered by the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the year range from the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One other issue is that we have manually put in the start and end year. Let's fix that up now by importing the data to get the range of years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "from pathlib import Path\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "DATA_DIR = 'data'\n",
    "DATA_FILE = 'land-ocean-temp-index.csv'\n",
    "\n",
    "original_data = pd.read_csv(Path(DATA_DIR) / DATA_FILE, escapechar='#')\n",
    "min_year = original_data['Year'].min()\n",
    "max_year = original_data['Year'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixing the slider problem: adding a constraint to `year_range` contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to express to pydantic that each `int` has a restricted range. We'll do that by defining our own type below and then using that custom type in our Pydantic model.\n",
    "\n",
    "We use a feature of type hinting introduced in Python 3.12 called type aliasing to do this. \n",
    "\n",
    "Since this tutorial was written to work in Python 3.11, we need to use the `typing-extensions` package, which is a package that back ports new typing features in Python to versions before the feature was added.\n",
    "\n",
    "The cell below imports`TypeAliasType` from `typing-extensions`to let pydantic and type checkers know that we are defining a new type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "from typing_extensions import TypeAliasType"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define a `ConstrainedInt` type that includes limits on its range. Instead of using fixed values of 1880 and 2023 we read in the data file and use the minimum and maximum from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "ConstrainedInt = TypeAliasType(\n",
    "    \"ConstrainedInt\", \n",
    "    Annotated[\n",
    "        int, \n",
    "        Field(ge=min_year, le=max_year)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Our kernel is running Python 3.11. The above feature looks very different in Python 3.12 than it does in this and earlier versions of Python. The equivalent to the definition of `ConstrainedInt` in Python 3.12 looks like this:\n",
    "\n",
    "```python\n",
    "# Only works in Python 3.12 and up!\n",
    "type ConstrainedInt = Annotated[int, Field(ge=1880, le=2023)]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we write our pydantic model using this new type to `year_range`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#| export\n",
    "\n",
    "class DataSelectorModel(BaseModel, validate_assignment=True):\n",
    "    year_range: Annotated[\n",
    "        # The key change is in the line below\n",
    "        tuple[ConstrainedInt, ConstrainedInt],\n",
    "        # With this change to the type we no longer need to tell ipyautoui\n",
    "        # what kind of widget to use. Field contains just a brief description\n",
    "        Field(description=\"Range of years to plot\")\n",
    "    ] = (1800, 2000)\n",
    "    window_size: Annotated[int, Field(ge=2, le=100)] = 2\n",
    "    polynomial_order: Annotated[int, Field(ge=1, le=10)] = 1\n",
    "\n",
    "    # mode=\"after\" means the validator runs after pydantic has checked that the individual\n",
    "    # fields have values that are valid.\n",
    "    @model_validator(mode=\"after\")\n",
    "    def limit_polynomial_order(self):\n",
    "        \n",
    "        if self.polynomial_order > self.window_size - 1:\n",
    "            # Handle a bad polynomial order or window size\n",
    "            raise ValueError(\"Polynomial order must be smaller than window size\")\n",
    "            \n",
    "        # If we got this far the polynomial order is consistent with the window size\n",
    "        # so return self. Failing to return self will end up causing an error.\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: try out the auto-generated user interface\n",
    "\n",
    "Make an interface using `AutoUi` from the model above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %answer key/03a/19.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks great! We will save our progress here and look into ipyautoui more in the next part of the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export module with `nbdev`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the code we want to reuse in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nbdev.export import nb_export\n",
    "\n",
    "nb_export('03a_pydantic.ipynb', 'dashboard_pydantic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FYI: `pydantic.TypeAdapter` can make any class a Pydantic class\n",
    "\n",
    "Pydantic 2 introduced `TypeAdapter` ([docs are here](https://docs.pydantic.dev/latest/concepts/type_adapter/) and it is also discussed in the [Types documentation ](https://docs.pydantic.dev/latest/concepts/types/#constrained-types)) which essentially takes any class and adds some of the methods you get if you subclass from `BaseModel`. This is very useful if the class you would like to work with is one from a package that you cannot modify to subclass from `BaseModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}