5.dive_into_dec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9b4396dc-4933-45de-8152-efd49e83192b",
   "metadata": {},
   "source": [
    "# 深入学习 DEC 模型\n",
    "\n",
    "第三节，我通过吃百家饭复现了 [DEC 模型](https://arxiv.org/abs/1511.06335)。不知道是不是我的问题，[vlukiyanov/pt-dec](https://github.com/vlukiyanov/pt-dec) 仓库的 DEC 模型跑起来 loss 不降反升（见 [test_ptdec.ipynb](./archived/test_ptdec.ipynb)）。好在该仓库已经实现了 DEC 论文中几个重要的类和函数，将它们拼接一番，也顺利把模型跑起来了。\n",
    "\n",
    "上一节，我们比较了 DEC 模型和传统的 K-Means 模型的准确率。在我的数据集上，它们准确率类似，都在 0.7 左右。这个结果不意外，因为 DEC 本身就是用 K-Means 来初始化聚类中心的。\n",
    "\n",
    "经过前两节的工作，我们对 DEC 模型有了初步的理解，希望在这里停下来总结一下。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb353512-2c40-4a95-92b6-2fd1c7896792",
   "metadata": {},
   "source": [
    "## 1. 模型的创新点\n",
    "\n",
    "这个模型的主要创新点是用软分配分布逼近一个目标分布，使得聚类下的样本特征向聚类中心靠拢，同时远离其他聚类中心。但是在初次使用 K-Means 分配聚类的时候，准确率就非常高了。可能是我数据集或者训练技巧的原因，这个创新点对我结果的提升不太明显。\n",
    "\n",
    "下面来介绍一下，软分配向目标分配逼近的具体过程：\n",
    "\n",
    "1. **聚类中心初始化**：首先做聚类中心的初始化，得到每个聚类的特征向量\n",
    "2. **软分配策略**：然后计算归一化的软分配分布 $q$\n",
    "3. **目标分布优化**：再通过 $q$ 计算分布“更尖锐”的目标分布 $p$\n",
    "4. **模型训练**：最后通过 KL 散度，最小化软分配分布与目标分布的差异，反向更新参数\n",
    "\n",
    "以下是对具体过程更详细的解释。\n",
    "\n",
    "### 1.1 聚类中心初始化\n",
    "\n",
    "通过编码器将原始数据映射到低维特征空间后，用 K-Means 计算 $k$ 个簇的类心。\n",
    "\n",
    "$k$ 个类心组成 $(k, hidden\\_dim)$ 维矩阵，其中 `hidden_dim` 是编码器最后一层的维度大小。\n",
    "\n",
    "### 1.2 软分配策略\n",
    "\n",
    "样本经过编码器映射成一个 `hidden_dim` 维特征向量，因此可以与 $k$  个同样是 `hidden_dim` 维的聚类中心计算欧式距离。\n",
    "\n",
    "我们需要借由学生 t 分布，将样本与各个类心之间的距离，转换成样本分配到各个类心的概率。\n",
    "\n",
    "原始的学生 t 分布如下：\n",
    "\n",
    "$f(t) = \\frac{\\Gamma\\left(\\frac{\\alpha + 1}{2}\\right)}{\\sqrt{\\alpha\\pi} \\cdot \\Gamma\\left(\\frac{\\alpha}{2}\\right)} \\cdot \\left(1 + \\frac{t^2}{\\alpha}\\right)^{-\\frac{\\alpha + 1}{2}}$\n",
    "\n",
    "其中：\n",
    "\n",
    "- $\\Gamma$ 是伽马函数\n",
    "- $\\alpha$ 是自由度\n",
    "\n",
    "t 分布用于计算样本嵌入与聚类中心的相似度，并将其转化为概率分布。具体步骤如下：\n",
    "\n",
    "**1）距离计算**\n",
    "\n",
    "假设样本嵌入为 $z_i$，聚类中心为 $\\mu_i$，计算欧式距离的平方：\n",
    "\n",
    "$d_{ij}^2 = \\|z_i - \\mu_j\\|^2$\n",
    "\n",
    "**2）t 分布概率转换**\n",
    "\n",
    "将距离代入 t 分布的概率密度函数形式，得到非归一化的概率：\n",
    "\n",
    "$q_{ij} \\propto \\left(1 + \\frac{d_{ij}^2}{\\alpha}\\right)^{-\\frac{\\alpha + 1}{2}}$\n",
    "\n",
    "其中 $\\alpha$ 是自由度参数。\n",
    "\n",
    "该公式通过调整距离的权重，使得近邻样本的概率更高，远邻样本的概率更低。\n",
    "\n",
    "**3）归一化**\n",
    "\n",
    "下面是样本 $i$ 分配到聚类 $j$ 的归一化概率：\n",
    "\n",
    "$q_{ij} = \\frac{\\left(1 + \\frac{d_{ij}^2}{\\alpha}\\right)^{-\\frac{\\alpha + 1}{2}}}{\\sum_{j'} \\left(1 + \\frac{d_{ij'}^2}{\\alpha}\\right)^{-\\frac{\\alpha + 1}{2}}}$\n",
    "\n",
    "其中，样本 $i$ 对所有聚类中心的 t 分布概率为分母，对第 $j$ 个聚类中心的 t 分布概率为分子。归一化确保样本 $i$ 对所有聚类中心的概率和为 1.\n",
    "\n",
    "> 软分配 (Soft Assignment) 为每个样本分配一个属于各个聚类中心的概率分布，而不是硬性地将其分配到某个聚类中。\n",
    "\n",
    "### 1.3 目标分布优化\n",
    "\n",
    "目标分布 (Target Distribution) 用于引导模型的训练，使得聚类中心更分离，聚类内样本更紧凑。\n",
    "\n",
    "目标分布的计算公式如下：\n",
    "\n",
    "$p_{ij} = \\frac{q_{ij}^2 / f_j}{\\sum_{j'=1}^k q_{ij'}^2 / f_{j'}}$\n",
    "\n",
    "其中，$f_j = \\sum_{i} q_{ij}$ 是第 $j$ 个聚类的软分配频率。\n",
    "\n",
    "该分布通过提升高频聚类的置信度 并抑制低频聚类的噪声 ，引导模型学习更具判别性的特征。\n",
    "\n",
    "### 1.4 模型训练\n",
    "\n",
    "使用 KL 散度作为损失函数，最小化软分配分布与目标分布之间的差异。具体来说，损失函数定义为：\n",
    "\n",
    "$L = \\text{KL}(P \\| Q) = \\sum_{i,j} p_{ij} \\log \\frac{p_{ij}}{q_{ij}}$\n",
    "\n",
    "其中，$P$ 是目标分布，$Q$ 是软分配分布。通过反向传播算法更新编码器的参数和聚类中心，不断迭代训练模型，直到损失函数收敛或达到预设的训练轮数。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5dba74c-1d94-4da2-8f5b-531d461fa43b",
   "metadata": {},
   "source": [
    "## 2. 训练阶段\n",
    "\n",
    "DEC 模型训练大致分为三个阶段：\n",
    "\n",
    "1. 预训练自编码器\n",
    "2. 初始化聚类中心\n",
    "3. 最小化软分配与目标分布的差异\n",
    "\n",
    "我把训练过程写到本仓库的 [dec.py](./dec.py) 文件中了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a432f36-580b-47a3-a575-6314ad4c6c8e",
   "metadata": {},
   "source": [
    "## 3. 聚类标签匹配问题\n",
    "\n",
    ">  参考：[匈牙利算法](https://zh.wikipedia.org/zh-sg/%E5%8C%88%E7%89%99%E5%88%A9%E7%AE%97%E6%B3%95)\n",
    "\n",
    "由于训练产生的标签与原标签之间可能标号不同，但是聚类下的样本却相同。为了对齐标号，需要求解 **线性分配问题**（Linear Assignment Problem, LAP），即寻找二分图的最小权匹配。其核心作用是将两个集合中的元素进行最优匹配，使得总匹配成本最小。我们可以直接用 scipy 的 [linear_sum_assignment](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html) 函数求解。\n",
    "\n",
    "下面是 [vlukiyanov/pt-dec](https://github.com/vlukiyanov/pt-dec/blob/master/ptdec/utils.py) 仓库中计算聚类准确率的函数，写得挺好的，可以借鉴："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e5da0885-4eca-41a1-b0d2-db6e604398a8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:07:14.250577Z",
     "iopub.status.busy": "2025-03-09T20:07:14.249576Z",
     "iopub.status.idle": "2025-03-09T20:07:14.263925Z",
     "shell.execute_reply": "2025-03-09T20:07:14.261915Z",
     "shell.execute_reply.started": "2025-03-09T20:07:14.250577Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from typing import Optional\n",
    "from scipy.optimize import linear_sum_assignment\n",
    "\n",
    "def cluster_accuracy(y_true, y_predicted, cluster_number: Optional[int] = None):\n",
    "    \"\"\"\n",
    "    Calculate clustering accuracy after using the linear_sum_assignment function in SciPy to\n",
    "    determine reassignments.\n",
    "\n",
    "    :param y_true: list of true cluster numbers, an integer array 0-indexed\n",
    "    :param y_predicted: list  of predicted cluster numbers, an integer array 0-indexed\n",
    "    :param cluster_number: number of clusters, if None then calculated from input\n",
    "    :return: reassignment dictionary, clustering accuracy\n",
    "    \"\"\"\n",
    "    if cluster_number is None:\n",
    "        cluster_number = (\n",
    "            max(y_predicted.max(), y_true.max()) + 1\n",
    "        )  # assume labels are 0-indexed\n",
    "    count_matrix = np.zeros((cluster_number, cluster_number), dtype=np.int64)\n",
    "    for i in range(y_predicted.size):\n",
    "        count_matrix[y_predicted[i], y_true[i]] += 1\n",
    "\n",
    "    row_ind, col_ind = linear_sum_assignment(count_matrix.max() - count_matrix)\n",
    "    reassignment = dict(zip(row_ind, col_ind))\n",
    "    accuracy = count_matrix[row_ind, col_ind].sum() / y_predicted.size\n",
    "    return reassignment, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6bd925-6207-46ff-af2d-0d29ba763028",
   "metadata": {},
   "source": [
    "## 4. 模型推理和优化\n",
    "\n",
    "1）模型推理\n",
    "\n",
    "我在 `./reserved/dec_full.pth` 路径下保留了一个训练过的 DEC 模型用于推理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1eaef55b-6477-44db-a940-5f281ad6e52f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:22:46.529080Z",
     "iopub.status.busy": "2025-03-09T20:22:46.528051Z",
     "iopub.status.idle": "2025-03-09T20:22:46.557767Z",
     "shell.execute_reply": "2025-03-09T20:22:46.556106Z",
     "shell.execute_reply.started": "2025-03-09T20:22:46.528051Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cuda\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([63, 12, 76, 76, 18, 76, 63, 93, 74, 76, 76, 18, 24, 18, 74, 18, 24,\n",
       "       63, 74, 18, 61,  6, 48, 18, 16, 24, 63, 74, 63, 76, 76, 85, 76, 41,\n",
       "       24, 76, 63, 18, 39, 76, 24, 24, 14, 71, 76, 12, 76, 63, 24, 99],\n",
       "      dtype=int64)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dec\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "# 定义设备\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f'Using device: {device}')\n",
    "\n",
    "# 声明模型结构\n",
    "DEC = dec.DEC\n",
    "ClusterAssignment = dec.ClusterAssignment\n",
    "\n",
    "# 加载模型\n",
    "full_model_path = './reserved/dec_full.pth'\n",
    "loaded_model = dec.load_full_model(full_model_path, device=device)\n",
    "\n",
    "# 执行推理\n",
    "new_embeddings = np.random.randn(50, 768).astype(np.float32)\n",
    "y_pred = dec.infer_embeddings(loaded_model, new_embeddings, device=device)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5afe083-e1c2-4244-b48e-373752a6f45b",
   "metadata": {},
   "source": [
    "2）模型优化\n",
    "\n",
    "使用 `dec.py` 训练模型，且进行一个模型优化尝试：在拟合目标分布阶段，冻结编码器参数。\n",
    "\n",
    "PS：这个改造似乎没啥用 ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e7191d21-193a-4846-af3e-e1e0087e2356",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:44:44.534408Z",
     "iopub.status.busy": "2025-03-09T20:44:44.533408Z",
     "iopub.status.idle": "2025-03-09T20:44:44.544241Z",
     "shell.execute_reply": "2025-03-09T20:44:44.542235Z",
     "shell.execute_reply.started": "2025-03-09T20:44:44.533408Z"
    }
   },
   "outputs": [],
   "source": [
    "import copy\n",
    "import torch\n",
    "import numpy as np\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "import dec\n",
    "\n",
    "from torch.utils.data import DataLoader, TensorDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0d5ae47e-0000-4d44-94a7-08bc915ff221",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:44:52.648980Z",
     "iopub.status.busy": "2025-03-09T20:44:52.647978Z",
     "iopub.status.idle": "2025-03-09T20:44:52.659225Z",
     "shell.execute_reply": "2025-03-09T20:44:52.657214Z",
     "shell.execute_reply.started": "2025-03-09T20:44:52.648980Z"
    }
   },
   "outputs": [],
   "source": [
    "# 参数配置\n",
    "TRAIN_CSV_PATH = './data/train_embed_label.csv'\n",
    "\n",
    "# 模型超参数配置\n",
    "config = {\n",
    "    \"dims\": [768, 256, 32],\n",
    "    \"n_clusters\": 100,\n",
    "    \"pretrain_epochs\": 50,\n",
    "    \"soft_dist_epochs\": 100,\n",
    "    \"update_interval\": 10,\n",
    "    \"batch_size\": 256,\n",
    "    \"tol\": 0.001,\n",
    "    \"alpha\": 1.0,\n",
    "    \"save_dir\": \"./model\",\n",
    "    \"args_model_file\": \"dec_args.pth\",\n",
    "    \"full_model_file\": \"dec_full.pth\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5dbfec6c-f1a0-44db-94df-819c5b780a13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:44:52.822553Z",
     "iopub.status.busy": "2025-03-09T20:44:52.821550Z",
     "iopub.status.idle": "2025-03-09T20:44:52.838779Z",
     "shell.execute_reply": "2025-03-09T20:44:52.836768Z",
     "shell.execute_reply.started": "2025-03-09T20:44:52.822553Z"
    }
   },
   "outputs": [],
   "source": [
    "# 模型优化尝试：冻结编码器参数\n",
    "def train_dec(model, data_loader, epochs, device, X, y_true=None, interval=10):\n",
    "    \"\"\"通过目标分布引导聚类优化\"\"\"\n",
    "\n",
    "    # 记录最优模型\n",
    "    best_model, best_acc = None, None\n",
    "\n",
    "    for param in model.encoder.parameters():\n",
    "        param.requires_grad = False\n",
    "    optimizer = optim.Adam([\n",
    "        {'params': model.assignment.parameters(), 'lr': 1e-5}\n",
    "    ])\n",
    "\n",
    "    criterion = F.kl_div\n",
    "    model.train()\n",
    "    for epoch in range(epochs):\n",
    "        total_loss = 0.0\n",
    "        for idx, x in data_loader:\n",
    "            x = x.to(device)\n",
    "            optimizer.zero_grad()\n",
    "            output = model(x)\n",
    "            target = dec.target_distribution(output).detach()\n",
    "            loss = criterion(output.log(), target, reduction='batchmean')\n",
    "            loss.backward()\n",
    "\n",
    "            # 梯度裁剪\n",
    "            torch.nn.utils.clip_grad_norm_(model.parameters(),\n",
    "                                           max_norm=1.0,\n",
    "                                           norm_type=2)\n",
    "\n",
    "            optimizer.step()\n",
    "            total_loss += loss.item()\n",
    "\n",
    "        avg_loss = total_loss / len(data_loader)\n",
    "        if (epoch + 1) % interval == 0:\n",
    "            print(f\"DEC Train Epoch {epoch+1}/{epochs}, Loss: {avg_loss:.4f}\")\n",
    "\n",
    "        if y_true is not None:\n",
    "            # 计算准确率\n",
    "            with torch.no_grad():\n",
    "                input = torch.from_numpy(X).float().to(device)\n",
    "                y_pred = model(input).argmax(1).cpu().numpy()\n",
    "            current_acc = dec.acc(y_true, y_pred)\n",
    "\n",
    "            # 更新最优模型\n",
    "            if best_acc is None or current_acc > best_acc:\n",
    "                best_model = copy.deepcopy(model)\n",
    "                best_acc = current_acc\n",
    "                print(f'===== best_acc: {best_acc:.4f} =====')\n",
    "\n",
    "    return model if best_model is None else best_model, best_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c5c150da-2c8f-44ff-9135-8fe46aa302d8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-03-09T20:44:53.403597Z",
     "iopub.status.busy": "2025-03-09T20:44:53.402602Z",
     "iopub.status.idle": "2025-03-09T20:48:02.082467Z",
     "shell.execute_reply": "2025-03-09T20:48:02.080766Z",
     "shell.execute_reply.started": "2025-03-09T20:44:53.403597Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cuda\n",
      "Pretrain Epoch 10/50, Loss: 0.0009\n",
      "Pretrain Epoch 20/50, Loss: 0.0008\n",
      "Pretrain Epoch 30/50, Loss: 0.0007\n",
      "Pretrain Epoch 40/50, Loss: 0.0007\n",
      "Pretrain Epoch 50/50, Loss: 0.0007\n",
      "init_acc: 0.7061\n",
      "===== best_acc: 0.7060 =====\n",
      "===== best_acc: 0.7061 =====\n",
      "DEC Train Epoch 10/100, Loss: 0.0050\n",
      "DEC Train Epoch 20/100, Loss: 0.0053\n",
      "DEC Train Epoch 30/100, Loss: 0.0057\n",
      "DEC Train Epoch 40/100, Loss: 0.0061\n",
      "DEC Train Epoch 50/100, Loss: 0.0065\n",
      "DEC Train Epoch 60/100, Loss: 0.0069\n",
      "DEC Train Epoch 70/100, Loss: 0.0073\n",
      "DEC Train Epoch 80/100, Loss: 0.0077\n",
      "DEC Train Epoch 90/100, Loss: 0.0081\n",
      "DEC Train Epoch 100/100, Loss: 0.0085\n",
      "\n",
      "Final Clustering Results:\n",
      "ACC: 0.7061\n",
      "NMI: 0.8057\n",
      "ARI: 0.5622\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f'Using device: {device}')\n",
    "\n",
    "# 数据准备\n",
    "X, y_true = dec.load_embed_data(TRAIN_CSV_PATH)\n",
    "dataset = TensorDataset(torch.arange(len(X)), torch.from_numpy(X.astype(np.float32)))\n",
    "pretrain_loader = DataLoader(dataset, batch_size=config[\"batch_size\"], shuffle=True)\n",
    "\n",
    "# ======= 阶段一：训练降噪自编码器 =======\n",
    "# 实例化编码器\n",
    "auto_encoder = dec.Autoencoder(config[\"dims\"]).to(device)\n",
    "\n",
    "# 执行编码器预训练代码\n",
    "auto_encoder = dec.pretrain(autoencoder=auto_encoder,\n",
    "                            data_loader=pretrain_loader,\n",
    "                            epochs=50,\n",
    "                            device=device,\n",
    "                            interval=config[\"update_interval\"])\n",
    "\n",
    "# ======= 阶段二：初始化聚类中心 =======\n",
    "full_loader = DataLoader(dataset, batch_size=1024, shuffle=False)\n",
    "kmeans, y_pred, init_acc = dec.init_cluster_centers(encoder=auto_encoder.encoder,\n",
    "                                                    data_loader=full_loader,\n",
    "                                                    n_clusters=config[\"n_clusters\"],\n",
    "                                                    device=device,\n",
    "                                                    y_true=y_true)\n",
    "print(f'init_acc: {init_acc}')\n",
    "\n",
    "# 代表聚类中心的特征向量\n",
    "cluster_centers = torch.tensor(kmeans.cluster_centers_,\n",
    "                               dtype=torch.float,\n",
    "                               requires_grad=True,\n",
    "                               device=device)\n",
    "\n",
    "# ======= 阶段三：训练 DEC =======\n",
    "# 实例化 DEC\n",
    "dec_model = dec.DEC(\n",
    "    cluster_number=config[\"n_clusters\"],  # 预设的聚类数\n",
    "    hidden_dimension=config[\"dims\"][-1],  # 编码器输出维度\n",
    "    encoder=auto_encoder.encoder,\n",
    "    alpha=config[\"alpha\"],\n",
    "    cluster_centers=cluster_centers\n",
    ")\n",
    "data_loader = DataLoader(dataset, batch_size=config[\"batch_size\"], shuffle=False)\n",
    "dec_model, dec_acc = train_dec(model=dec_model,\n",
    "                               data_loader=data_loader,\n",
    "                               epochs=config[\"soft_dist_epochs\"],\n",
    "                               device=device,\n",
    "                               X=X,\n",
    "                               y_true=y_true,\n",
    "                               interval=config[\"update_interval\"])\n",
    "\n",
    "# 保存最优模型\n",
    "dec.save_full_model(dec_model, config)\n",
    "dec.save_args_model(dec_model, config)\n",
    "\n",
    "# 计算指标\n",
    "y_pred = dec.infer_embeddings(dec_model, X, device=device)\n",
    "print(\"\\nFinal Clustering Results:\")\n",
    "print(f\"ACC: {dec.acc(y_true, y_pred):.4f}\")\n",
    "print(f\"NMI: {dec.nmi(y_true, y_pred):.4f}\")\n",
    "print(f\"ARI: {dec.ari(y_true, y_pred):.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a1cf0f6-523a-4e89-8c80-704fcdb2f49e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}