From a310429d8b57c37b4bc9cb808938685cf7726cf7 Mon Sep 17 00:00:00 2001 From: Eric Date: Mon, 6 May 2024 23:14:08 +0800 Subject: [PATCH 1/3] Conversation template deepseek update (doc only) --- docs/source/examples/DATASETS.md | 1 + .../supported_conversation_template.md | 57 +++++++++++++++---- 2 files changed, 47 insertions(+), 11 deletions(-) diff --git a/docs/source/examples/DATASETS.md b/docs/source/examples/DATASETS.md index 54abd8c92..7e4bc39ca 100644 --- a/docs/source/examples/DATASETS.md +++ b/docs/source/examples/DATASETS.md @@ -143,6 +143,7 @@ Conversations should be formatted before feeding into the model. As of now, we'v | Template Name | Filled Example | Detailed Template | | ------------- | -------------- | ----------------- | | `chatml` | `<\|im_start\|>system`
`You are a chatbot developed by LMFlow team.<\|im_end\|>`
`<\|im_start\|>user`
`Who are you?<\|im_end\|>`
`<\|im_start\|>assistant`
`I am a chatbot developed by LMFlow team.<\|im_end\|>`
`<\|im_start\|>user`
`How old are you?<\|im_end\|>`
`<\|im_start\|>assistant`
`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|im_end\|>`
| [Link](./supported_conversation_template.md#chatml) | +| `deepseek` | `<｜begin▁of▁sentence｜>You are a chatbot developed by LMFlow team.`

`User: Who are you?`

`Assistant: I am a chatbot developed by LMFlow team.<｜end▁of▁sentence｜>User: How old are you?`

`Assistant: I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<｜end▁of▁sentence｜>` | [Link](./supported_conversation_template.md#deepseek) | | `llama3` | `<\|begin_of_text\|><\|start_header_id\|>system<\|end_header_id\|>`

`You are a chatbot developed by LMFlow team.<\|eot_id\|><\|start_header_id\|>user<\|end_header_id\|>`

`Who are you?<\|eot_id\|><\|start_header_id\|>assistant<\|end_header_id\|>`

`I am a chatbot developed by LMFlow team.<\|eot_id\|><\|start_header_id\|>user<\|end_header_id\|>`

`How old are you?<\|eot_id\|><\|start_header_id\|>assistant<\|end_header_id\|>`

`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|eot_id\|>` | [Link](./supported_conversation_template.md#llama-3) | | `llama2` | `[INST] <>`
`You are a chatbot developed by LMFlow team.`
`<>`

`Who are you? [/INST] I am a chatbot developed by LMFlow team.[INST] How old are you? [/INST] I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.` | [Link](./supported_conversation_template.md#llama-2) | | `phi3` | `<\|system\|>`
`You are a chatbot developed by LMFlow team.<\|end\|>`
`<\|user\|>\nWho are you?<\|end\|>`
`<\|assistant\|>`
`I am a chatbot developed by LMFlow team.<\|end\|>`
`<\|user\|>`
`How old are you?<\|end\|>`
`<\|assistant\|>`
`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|end\|>`
`<\|endoftext\|>` | [Link](./supported_conversation_template.md#phi-3) | diff --git a/docs/source/examples/supported_conversation_template.md b/docs/source/examples/supported_conversation_template.md index 15b8d1423..f6bf2caf0 100644 --- a/docs/source/examples/supported_conversation_template.md +++ b/docs/source/examples/supported_conversation_template.md @@ -1,15 +1,17 @@ # Supported Conversation Template -## Table of Contents -- [ChatGLM-3](#chatglm-3) -- [InternLM2](#internlm2) -- [Llama-2](#llama-2) -- [Llama-3](#llama-3) -- [Mixtral 8x22B](#mixtral-8x22b) -- [Mixtral 8x7B](#mixtral-8x7b) -- [Qwen-2](#qwen-2) -- [Phi-3](#phi-3) -- [Yi](#yi) +- [Supported Conversation Template](#supported-conversation-template) + - [ChatGLM-3](#chatglm-3) + - [ChatML](#chatml) + - [DeepSeek](#deepseek) + - [InternLM2](#internlm2) + - [Llama-2](#llama-2) + - [Llama-3](#llama-3) + - [Mixtral 8x22B](#mixtral-8x22b) + - [Mixtral 8x7B](#mixtral-8x7b) + - [Phi-3](#phi-3) + - [Qwen-2](#qwen-2) + - [Yi](#yi) ## ChatGLM-3 @@ -58,6 +60,39 @@ This template is not preseted in LMFlow currently. We are working on it and will ``` +## DeepSeek +**With a system message** +``` +<｜begin▁of▁sentence｜>{{system_message}}\n\nUser: {{user_message_0}}\n\n +``` + +**Without a system message** +``` +<｜begin▁of▁sentence｜>User: {{user_message_0}}\n\n +``` + +**A complete conversation** +``` +<｜begin▁of▁sentence｜>{{system_message}}\n\nUser: {{user_message_0}}\n\nAssistant: {{assistant_reply_0}}<｜end▁of▁sentence｜> +``` + +**Multiple rounds** +``` +<｜begin▁of▁sentence｜>{{system_message}}\n\nUser: {{user_message_0}}\n\nAssistant: {{assistant_reply_0}}<｜end▁of▁sentence｜>User: {{user_message_1}}\n\nAssistant: {{assistant_reply_1}}<｜end▁of▁sentence｜> +``` + +**jinja template** +[[Reference](https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat/blob/941577e8236164bc96829096d20c61568630d7bc/tokenizer_config.json#L34)] +``` +{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{{ bos_token }}{% for message in messages %}{% if message['role'] == 'user' %}{{ 'User: ' + message['content'] + '\n\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'] + eos_token }}{% elif message['role'] == 'system' %}{{ message['content'] + '\n\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %} +``` + +**Filled Example** +``` +<｜begin▁of▁sentence｜>You are a chatbot developed by LMFlow team.\n\nUser: Who are you?\n\nAssistant: I am a chatbot developed by LMFlow team.<｜end▁of▁sentence｜>User: How old are you?\n\nAssistant: I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<｜end▁of▁sentence｜> +``` + + ## InternLM2 ```{admonition} **Work in Progress** :class: info @@ -196,7 +231,7 @@ The conversation template for Mixtral 8x7B is slightly different from the templa <|system|>\n{{system_message}}<|end|>\n<|user|>\n{{user_message_0}}<|end|>\n<|assistant|>\n{{assistant_reply_0}}<|end|>\n<|user|>\n{{user_message_1}}<|end|>\n<|assistant|>\n{{assistant_reply_1}}<|end|>\n<|endoftext|> ``` -**jinja template** +**jinja template** [[Reference]](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/3a811845d89f3c1b3f41b341d0f9f05104769f35/tokenizer_config.json#L338) ``` {{ bos_token }}{% for message in messages %}{{'<|' + message['role'] + '|>' + '\n' + message['content'] + '<|end|>\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>\n' }}{% else %}{{ eos_token }}{% endif %} From 6ea79b8e9ca29b3436005ac4e2544e893e242456 Mon Sep 17 00:00:00 2001 From: Eric Date: Mon, 6 May 2024 23:23:07 +0800 Subject: [PATCH 2/3] DeepSeek conversation template support --- src/lmflow/utils/conversation_template.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/src/lmflow/utils/conversation_template.py b/src/lmflow/utils/conversation_template.py index 20839367d..61715c62b 100644 --- a/src/lmflow/utils/conversation_template.py +++ b/src/lmflow/utils/conversation_template.py @@ -268,6 +268,27 @@ class ChatMLConversationTemplate(ConversationTemplate): TemplateComponent(type='string', content='<|im_start|>system\n{{content}}<|im_end|>\n') ] ) + + +@dataclass +class DeepSeekConversationTemplate(ConversationTemplate): + user_formatter: Formatter = StringFormatter( + template=[ + TemplateComponent(type='string', content='User: {{content}}\n\n') + ] + ) + assistant_formatter: Formatter = StringFormatter( + template=[ + TemplateComponent(type='string', content='Assistant: {{content}}'), + TemplateComponent(type='token', content='eos_token') + ] + ) + system_formatter: Formatter = StringFormatter( + template=[ + TemplateComponent(type='string', content='{{content}}\n\n') + ] + ) + special_starter: TemplateComponent = TemplateComponent(type='token', content='bos_token') @dataclass From c102a78f392a4e8ca97e837dbf17712bbb675075 Mon Sep 17 00:00:00 2001 From: Eric Date: Mon, 6 May 2024 23:34:32 +0800 Subject: [PATCH 3/3] DeepSeek conversation template on hold --- docs/source/examples/DATASETS.md | 1 - docs/source/examples/supported_conversation_template.md | 5 +++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/source/examples/DATASETS.md b/docs/source/examples/DATASETS.md index 7e4bc39ca..54abd8c92 100644 --- a/docs/source/examples/DATASETS.md +++ b/docs/source/examples/DATASETS.md @@ -143,7 +143,6 @@ Conversations should be formatted before feeding into the model. As of now, we'v | Template Name | Filled Example | Detailed Template | | ------------- | -------------- | ----------------- | | `chatml` | `<\|im_start\|>system`
`You are a chatbot developed by LMFlow team.<\|im_end\|>`
`<\|im_start\|>user`
`Who are you?<\|im_end\|>`
`<\|im_start\|>assistant`
`I am a chatbot developed by LMFlow team.<\|im_end\|>`
`<\|im_start\|>user`
`How old are you?<\|im_end\|>`
`<\|im_start\|>assistant`
`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|im_end\|>`
| [Link](./supported_conversation_template.md#chatml) | -| `deepseek` | `<｜begin▁of▁sentence｜>You are a chatbot developed by LMFlow team.`

`User: Who are you?`

`Assistant: I am a chatbot developed by LMFlow team.<｜end▁of▁sentence｜>User: How old are you?`

`Assistant: I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<｜end▁of▁sentence｜>` | [Link](./supported_conversation_template.md#deepseek) | | `llama3` | `<\|begin_of_text\|><\|start_header_id\|>system<\|end_header_id\|>`

`You are a chatbot developed by LMFlow team.<\|eot_id\|><\|start_header_id\|>user<\|end_header_id\|>`

`Who are you?<\|eot_id\|><\|start_header_id\|>assistant<\|end_header_id\|>`

`I am a chatbot developed by LMFlow team.<\|eot_id\|><\|start_header_id\|>user<\|end_header_id\|>`

`How old are you?<\|eot_id\|><\|start_header_id\|>assistant<\|end_header_id\|>`

`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|eot_id\|>` | [Link](./supported_conversation_template.md#llama-3) | | `llama2` | `[INST] <>`
`You are a chatbot developed by LMFlow team.`
`<>`

`Who are you? [/INST] I am a chatbot developed by LMFlow team.[INST] How old are you? [/INST] I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.` | [Link](./supported_conversation_template.md#llama-2) | | `phi3` | `<\|system\|>`
`You are a chatbot developed by LMFlow team.<\|end\|>`
`<\|user\|>\nWho are you?<\|end\|>`
`<\|assistant\|>`
`I am a chatbot developed by LMFlow team.<\|end\|>`
`<\|user\|>`
`How old are you?<\|end\|>`
`<\|assistant\|>`
`I don't age like humans do. I exist as a piece of software, so I don't have a concept of age in the traditional sense.<\|end\|>`
`<\|endoftext\|>` | [Link](./supported_conversation_template.md#phi-3) | diff --git a/docs/source/examples/supported_conversation_template.md b/docs/source/examples/supported_conversation_template.md index f6bf2caf0..c0f6f2261 100644 --- a/docs/source/examples/supported_conversation_template.md +++ b/docs/source/examples/supported_conversation_template.md @@ -61,6 +61,11 @@ This template is not preseted in LMFlow currently. We are working on it and will ## DeepSeek +```{admonition} **Work in Progress** +:class: info + +We are working on building test cases for this template. We will add this template into switch case once the tests pass. +``` **With a system message** ``` <｜begin▁of▁sentence｜>{{system_message}}\n\nUser: {{user_message_0}}\n\n