Demos/04. Hugging Face AutoModel 示例合集.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf6ecadb-1944-49e6-bb79-789276ece92c",
   "metadata": {},
   "source": [
    "# 理解 Hugging Face 的 `AutoModel` 系列：不同任务的自动模型加载类\n",
    "\n",
    "> 指导文章：[05. 理解 Hugging Face 的 `AutoModel` 系列：不同任务的自动模型加载类](https://github.com/Hoper-J/LLM-Guide-and-Demos-zh_CN/blob/master/Guide/05.%20理解%20Hugging%20Face%20的%20%60AutoModel%60%20系列：不同任务的自动模型加载类.md)\n",
    "\n",
    "这里是一些不同类的代码示例。\n",
    "\n",
    "在线链接：[Kaggle](https://www.kaggle.com/code/aidemos/04-hugging-face-automodel) | [Colab](https://colab.research.google.com/drive/1gLTXcvG-tEDOqnR7qM-3-S812qnBUGlh?usp=sharing)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc4392f-446d-42e8-8310-a4d607a4070a",
   "metadata": {},
   "source": [
    "## 安装库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "399141ec-07dd-484c-ae90-1d882133e6d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install transformers\n",
    "!pip install sentencepiece"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ec78d1-eb1f-46cd-a529-95d618fe8ef0",
   "metadata": {},
   "source": [
    "## 设置模型下载镜像"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91155fcb-9c6f-4d57-a7a8-23bc88e7e55f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc629306-1e54-40d3-82ef-83ed6e391290",
   "metadata": {},
   "source": [
    "## 示例 1：文本生成 (`AutoModelForCausalLM`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b41daad1-093e-4431-914a-69256700acbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"gpt2\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name)\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"Once upon a time\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 生成文本\n",
    "outputs = model.generate(**inputs, max_length=50, do_sample=True, top_p=0.95, temperature=0.7)\n",
    "\n",
    "# 解码生成的文本\n",
    "generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
    "print(generated_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a1af4af-5168-4480-b0a8-f1041598b0d0",
   "metadata": {},
   "source": [
    "## 示例 2：填空任务 (`AutoModelForMaskedLM`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6763d23e-b0b4-46d4-94ae-e322eb54c84b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"bert-base-uncased\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForMaskedLM.from_pretrained(model_name)\n",
    "\n",
    "# 输入文本，包含 [MASK] 标记\n",
    "input_text = \"The capital of France is [MASK].\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 获取预测\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "    predictions = outputs.logits\n",
    "\n",
    "# 获取最高得分的预测词\n",
    "masked_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]\n",
    "predicted_token_id = predictions[0, masked_index].argmax(dim=-1).item()\n",
    "predicted_token = tokenizer.decode([predicted_token_id])\n",
    "\n",
    "print(f\"预测结果: {predicted_token}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a97ce34-2da0-41be-8a2b-81406448a518",
   "metadata": {},
   "source": [
    "## 示例 3：序列到序列任务 (`AutoModelForSeq2SeqLM`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f33496a0-a14a-445c-b80a-5777329b7915",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"Helsinki-NLP/opus-mt-en-de\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"Hello, how are you?\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 生成翻译\n",
    "outputs = model.generate(**inputs, max_length=40, num_beams=4, early_stopping=True)\n",
    "\n",
    "# 解码生成的文本\n",
    "translated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
    "print(f\"翻译结果: {translated_text}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb96f479-95cf-4394-965e-d62e5440aea7",
   "metadata": {},
   "source": [
    "## 示例 4：问答系统 (`AutoModelForQuestionAnswering`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36ab7c79-c25c-480e-9ff5-38de61aa84b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForQuestionAnswering\n",
    "import torch\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"distilbert-base-uncased-distilled-squad\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForQuestionAnswering.from_pretrained(model_name)\n",
    "\n",
    "# 输入上下文和问题\n",
    "context = \"Hugging Face is creating a tool that democratizes AI.\"\n",
    "question = \"What is Hugging Face creating?\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer.encode_plus(question, context, return_tensors=\"pt\")\n",
    "\n",
    "# 获取预测\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# 获取答案的起始和结束位置\n",
    "answer_start = torch.argmax(outputs.start_logits)\n",
    "answer_end = torch.argmax(outputs.end_logits) + 1\n",
    "\n",
    "# 解码答案\n",
    "answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0][answer_start:answer_end]))\n",
    "print(f\"答案: {answer}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2737c9-baf0-404f-bbfb-766e470f2d6b",
   "metadata": {},
   "source": [
    "## 示例 5：命名实体识别 (`AutoModelForTokenClassification`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f4db14d-ed18-432d-9226-bb046cf2c69f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"dbmdz/bert-large-cased-finetuned-conll03-english\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForTokenClassification.from_pretrained(model_name)\n",
    "\n",
    "# 标签列表\n",
    "label_list = model.config.id2label\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very close to the Manhattan Bridge.\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 获取模型输出\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# 获取预测分数\n",
    "logits = outputs.logits\n",
    "predictions = torch.argmax(logits, dim=2)\n",
    "\n",
    "# 将预测结果映射到标签\n",
    "tokens = tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0])\n",
    "pred_labels = [label_list[prediction.item()] for prediction in predictions[0]]\n",
    "\n",
    "# 打印结果\n",
    "for token, label in zip(tokens, pred_labels):\n",
    "    print(f\"{token}: {label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd72d2d-108b-4722-beab-df35640dcb15",
   "metadata": {},
   "source": [
    "## 示例 6：文本分类 (`AutoModelForSequenceClassification`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae2d0feb-680d-44af-98af-b31677c8bbb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "\n",
    "# 加载 Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"I love using transformers library!\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 获取模型输出\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# 获取预测分数\n",
    "logits = outputs.logits\n",
    "probabilities = F.softmax(logits, dim=1)\n",
    "\n",
    "# 获取标签\n",
    "labels = ['Negative', 'Positive']\n",
    "prediction = torch.argmax(probabilities, dim=1)\n",
    "predicted_label = labels[prediction]\n",
    "\n",
    "# 打印结果\n",
    "print(f\"文本: {input_text}\")\n",
    "print(f\"情感预测: {predicted_label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "981f413d-09d7-400d-ba2f-f04bc0d5e097",
   "metadata": {},
   "source": [
    "## 示例 7：特征提取 (`AutoModel`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1e45a6-dfef-42e4-a483-e200ae8cd448",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModel\n",
    "import torch\n",
    "\n",
    "# 指定模型名称\n",
    "model_name = \"bert-base-uncased\"\n",
    "\n",
    "# 加载 Tokenizer 和模型\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = AutoModel.from_pretrained(model_name)\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"This is a sample sentence.\"\n",
    "\n",
    "# 编码输入\n",
    "inputs = tokenizer(input_text, return_tensors=\"pt\")\n",
    "\n",
    "# 获取模型输出\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# 获取最后一层隐藏状态\n",
    "last_hidden_states = outputs.last_hidden_state\n",
    "\n",
    "# 输出维度\n",
    "print(f\"Last hidden state shape: {last_hidden_states.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e6009d-3196-42a3-b6cf-8cbf40117ff3",
   "metadata": {},
   "source": [
    "## 查看源码\n",
    "\n",
    "以 `AutoModelForQuestionAnswering` 为例，使用 `inspect` 库查看对应源码:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b22d1db0-f217-438d-b1d0-c3fc553ec1e5",
   "metadata": {},
   "source": [
    "### 查看 `__init__` 方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e3bd9cd-0b0a-4092-a0a5-e89e17742d20",
   "metadata": {},
   "outputs": [],
   "source": [
    "import inspect\n",
    "from transformers import AutoModelForQuestionAnswering\n",
    "\n",
    "# 加载预训练模型\n",
    "model = AutoModelForQuestionAnswering.from_pretrained(\"distilbert-base-uncased-distilled-squad\")\n",
    "\n",
    "# 获取并打印 __init__ 方法的源码\n",
    "init_code = inspect.getsource(model.__init__)\n",
    "print(init_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc2d04f-c1f9-4bba-b7c1-f5daeac3a5d7",
   "metadata": {},
   "source": [
    "### 查看 `forward` 方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37924b48-b057-4621-89d4-6e766874d2f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取并打印 forward 方法的源码\n",
    "forward_code = inspect.getsource(model.forward)\n",
    "print(forward_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c180a0a-f434-45f2-a4dc-4119b0616e1d",
   "metadata": {},
   "source": [
    "### 使用 `help` 快速查看\n",
    "\n",
    "除了 `inspect`，我们还可以使用 Python 内置的 `help` 函数查看模型的文档和方法:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30b51b00-93ef-48e5-aa74-3dab2bb26882",
   "metadata": {},
   "outputs": [],
   "source": [
    "help(AutoModelForQuestionAnswering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "964b87e1-74d7-49b7-a763-4ee8978c299f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}