本地部署之后，AgentFabric上操作就会报错。preview_send_message user_agent = _state['user_agent'] KeyError: 'user_agent'

[yebanliuying]

Initial Checks

• I have searched GitHub for a duplicate issue and I'm sure this is something new
• I have read and followed the docs & demos and still think this is a bug
• I am confident that the issue is with modelscope-agent (not my code, or another library in the ecosystem)

What happened + What you expected to happen

根据https://github.com/modelscope/modelscope-agent/blob/master/docs/local_deploy.md ，本地部署之后，AgentFabric上操作就会报错。 报错信息如下： Starting install nltk data... nltk data installed. setting nltk data path to: /data/work/modelscope-agent/apps/agentfabric/tmp/nltk_data /opt/conda/lib/python3.10/site-packages/pydantic/_internal/_fields.py:184: UserWarning: Field name "function_map" shadows an attribute in parent "Agent"; warnings.warn( Running on local URL: http://0.0.0.0:7860

To create a public link, set share=True in launch().
2024-07-07 14:33:31.463 - modelscope-agent - INFO - | message: builder_cfg | uuid: local_user | details: {'builder_cfg': "Config (path: /tmp/agentfabric/config/local_user/builder_config.json): {'name': '', 'avatar': 'custom_bot_avatar.png', 'description': '', 'instruction': '', 'language': 'zh', 'prompt_recommend': ['你可以做什么？', '你有什么功能？', '如何使用你的功能？', '能否给我一些示例指令？'], 'knowledge': [], 'tools': {'image_gen': {'name': 'Wanx Image Generation', 'is_active': True, 'use': True}, 'code_interpreter': {'name': 'Code Interpreter', 'is_active': True, 'use': False}}, 'model': 'qwen-max'}"} | step: | error:
2024-07-07 14:33:31.493 - modelscope-agent - INFO - | message: using model qwen-max with tool Config (path: ./config/tool_config.json): {'image_gen': {'name': 'Wanx Image Generation', 'is_active': True, 'use': True, 'is_remote_tool': True}, 'code_interpreter': {'name': 'Code Interpreter', 'is_active': True, 'use': False, 'is_remote_tool': False, 'max_output': 2000}, 'web_browser': {'name': 'Web Browsing', 'is_active': True, 'use': False, 'max_browser_length': 2000}, 'amap_weather': {'name': '高德天气', 'is_active': True, 'use': False}, 'paraformer_asr': {'name': 'Paraformer语音识别', 'is_active': True, 'use': False, 'is_remote_tool': True}, 'sambert_tts': {'name': 'Sambert语音合成', 'is_active': True, 'use': False, 'is_remote_tool': True}, 'wordart_texture_generation': {'name': '艺术字纹理生成', 'is_active': True, 'use': False}, 'web_search': {'name': 'Web Searching', 'is_active': True, 'use': False, 'searcher': 'bing'}, 'qwen_vl': {'name': 'Qwen-VL识图', 'is_active': True, 'use': False}, 'style_repaint': {'name': '人物风格重绘', 'is_active': True, 'use': False}, 'image_enhancement': {'name': '追影-放大镜', 'is_active': True, 'use': False}, 'text-address': {'name': '地址解析', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/mgeo_geographic_elements_tagging_chinese_base', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'text-ner': {'name': '命名实体识别', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_raner_named-entity-recognition_chinese-base-cmeee', 'use': False, 'is_active': False, 'is_remote_tool': True}, 'speech-generation': {'name': '语音生成', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/speech_sambert-hifigan_tts_zh-cn_16k', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'video-generation': {'name': '视频生成', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/text-to-video-synthesis', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'text-translation-en2zh': {'name': '英译中', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_csanmt_translation_en2zh', 'use': False, 'is_active': False, 'is_remote_tool': True}, 'text-translation-zh2en': {'name': '中译英', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_csanmt_translation_zh2en', 'use': False, 'is_active': False, 'is_remote_tool': True}} and function list ['image_gen'] | uuid: local_user | details: {'model_config': {'type': 'dashscope', 'model': 'qwen-max', 'length_constraint': {'knowledge': 4000, 'input': 6000}, 'generate_cfg': {'use_raw_prompt': True, 'top_p': 0.5, 'stop': 'Observation'}}} | step: | error:
2024-07-07 14:33:31.496 - modelscope-agent - ERROR - | message: | uuid: local_user | details: {'error_traceback': 'Traceback (most recent call last):\n File "/data/work/modelscope-agent/apps/agentfabric/app.py", line 37, in init_user\n user_agent, user_memory = init_user_chatbot_agent(\n File "/data/work/modelscope-agent/apps/agentfabric/user_core.py", line 46, in init_user_chatbot_agent\n agent = RolePlay(\n File "/data/work/modelscope-agent/modelscope_agent/agents/role_play.py", line 146, in init\n Agent.init(self, function_list, llm, storage_path, name,\n File "/data/work/modelscope-agent/modelscope_agent/agent.py", line 44, in init\n self.llm = get_chat_model(**self.llm_config)\n File "/data/work/modelscope-agent/modelscope_agent/llm/init.py", line 21, in get_chat_model\n return LLM_REGISTRY[registered_model_id](model, model_server, **kwargs)\n File "/data/work/modelscope-agent/modelscope_agent/llm/dashscope.py", line 78, in init\n assert dashscope.api_key, 'DASHSCOPE_API_KEY is required.'\nAssertionError: DASHSCOPE_API_KEY is required.\n'} | step: | error: DASHSCOPE_API_KEY is required.
2024-07-07 14:33:31.501 - modelscope-agent - INFO - | message: using builder model qwen-max | uuid: local_user | details: {} | step: | error:
2024-07-07 14:33:31.502 - modelscope-agent - ERROR - | message: | uuid: local_user | details: {'error_traceback': 'Traceback (most recent call last):\n File "/data/work/modelscope-agent/apps/agentfabric/app.py", line 52, in init_builder\n builder_agent, builder_memory = init_builder_chatbot_agent(uuid_str)\n File "/data/work/modelscope-agent/apps/agentfabric/builder_core.py", line 35, in init_builder_chatbot_agent\n agent = AgentBuilder(llm=llm_config, uuid_str=uuid_str)\n File "/data/work/modelscope-agent/modelscope_agent/agents/agent_builder.py", line 91, in init\n super().init(\n File "/data/work/modelscope-agent/modelscope_agent/agent.py", line 44, in init\n self.llm = get_chat_model(**self.llm_config)\n File "/data/work/modelscope-agent/modelscope_agent/llm/init.py", line 21, in get_chat_model\n return LLM_REGISTRY[registered_model_id](model, model_server, **kwargs)\n File "/data/work/modelscope-agent/modelscope_agent/llm/dashscope.py", line 78, in init\n assert dashscope.api_key, 'DASHSCOPE_API_KEY is required.'\nAssertionError: DASHSCOPE_API_KEY is required.\n'} | step: | error: DASHSCOPE_API_KEY is required.
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/gradio/queueing.py", line 532, in process_events
response = await route_utils.call_process_api(
File "/opt/conda/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "/opt/conda/lib/python3.10/site-packages/gradio/blocks.py", line 1928, in process_api
result = await self.call_function(
File "/opt/conda/lib/python3.10/site-packages/gradio/blocks.py", line 1526, in call_function
prediction = await utils.async_iteration(iterator)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 656, in async_iteration
return await iterator.anext()
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 649, in anext
return await anyio.to_thread.run_sync(
File "/opt/conda/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/opt/conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
return await future
File "/opt/conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 632, in run_sync_iterator_async
return next(iterator)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 815, in gen_wrapper
response = next(iterator)
File "/data/work/modelscope-agent/apps/agentfabric/app.py", line 435, in create_send_message
builder_agent = _state['builder_agent']
KeyError: 'builder_agent'
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/gradio/queueing.py", line 532, in process_events
response = await route_utils.call_process_api(
File "/opt/conda/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "/opt/conda/lib/python3.10/site-packages/gradio/blocks.py", line 1928, in process_api
result = await self.call_function(
File "/opt/conda/lib/python3.10/site-packages/gradio/blocks.py", line 1526, in call_function
prediction = await utils.async_iteration(iterator)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 656, in async_iteration
return await iterator.anext()
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 649, in anext
return await anyio.to_thread.run_sync(
File "/opt/conda/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/opt/conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
return await future
File "/opt/conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 632, in run_sync_iterator_async
return next(iterator)
File "/opt/conda/lib/python3.10/site-packages/gradio/utils.py", line 815, in gen_wrapper
response = next(iterator)
File "/data/work/modelscope-agent/apps/agentfabric/app.py", line 601, in preview_send_message
user_agent = _state['user_agent']
KeyError: 'user_agent'

Versions / Dependencies

最新版本

Reproduction script

GRADIO_SERVER_NAME=0.0.0.0 PYTHONPATH=../../ python app.py

Issue Severity

High: It blocks me from completing my task.

[yebanliuying]

模型配的是:Qwen2-72B-Instruct-AWQ

"qwen2-72b-instruct-awq": {
"type": "openai",
"model": "qwen/Qwen2-72B-Instruct-AWQ",
"api_base": "http://localhost:8000/v1",
"is_chat": true,
"is_function_call": false
}

[yebanliuying]

在环境变量定义DASHSCOPE_API_KEY了之后能往下跑了，但是还是会报错：
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data/work/modelscope-agent/apps/agentfabric/builder_core.py", line 60, in gen_response_and_process
for s in response:
File "/data/work/modelscope-agent/modelscope_agent/llm/dashscope.py", line 20, in stream_output
for trunk in response:
File "/data/work/modelscope-agent/modelscope_agent/llm/dashscope.py", line 142, in stat_last_call_token_info
if not chunk.usage.get('total_tokens'):
AttributeError: 'NoneType' object has no attribute 'get'
| step: | error: llm result is not valid
2024-07-07 14:59:58.225 - modelscope-agent - INFO - | message: frame | uuid: local_user | details: {'frame': '{'error': "llm result is not valid: 'NoneType' object has no attribute 'get'. Please reset and try again."}'} | step: | error:
2024-07-07 14:59:58.233 - modelscope-agent - INFO - | message: frame | uuid: local_user | details: {'frame': "{'error': 'llm result is not valid. parse RichConfig error. Please reset and try again.'}"} | step: | error:
2024-07-07 14:59:58.241 - modelscope-agent - ERROR - | message: | uuid: local_user | details: {} | step: | error: parse RichConfig error
No valid document. Return Empty Response.
2024-07-07 15:00:09.693 - modelscope-agent - INFO - | message: call dashscope generation api | uuid: | details: {'model': 'qwen-max', 'messages': [{'role': 'user', 'content': 'What is the weather like in Boston?'}], 'stop': [{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather in a given location.', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit']}}, 'required': ['location']}}}], 'top_p': 0.8, 'result_format': 'message', 'stream': True} | step: | error:
2024-07-07 15:00:09.700 - modelscope-agent - INFO - | message: call dashscope generation api | uuid: | details: {'model': 'qwen-max', 'messages': [{'role': 'user', 'content': '<|im_start|>system\n\n\n# 知识库\n\nEmpty Response\n\n\n# 工具\n\n## 你拥有如下工具：\n\nimage_gen: image_gen API。AI绘画（图像生成）服务，输入文本描述和图像分辨率，返回根据文本信息绘制的图片URL。 输入参数: {"type": "object", "properties": {"text": {"type": "string", "description": "详细描述了希望生成的图像具有什么内容，例如人物、环境、动作等细节描述"}, "resolution": {"type": "string", "description": "格式是 数字数字，表示希望生成的图像的分辨率大小，选项有[10241024, 7201280, 1280720]"}, "lora_index": {"type": "string", "description": "如果用户要求使用lora的情况下，则使用该参数，没有指定的情况下默认为wanx1.4.5_textlora_huiben2_20240518"}}, "required": ["text", "resolution"]} Format the arguments as a JSON object.\n\n## 当你需要调用工具时，请在你的回复中穿插如下的工具调用命令，可以根据需求调用零次或多次：\n\n工具调用\nAction: 工具的名称，必须是[image_gen]之一\nAction Input: 工具的输入\nObservation: 工具返回的结果\nAnswer: 根据Observation总结本次工具调用返回的结果，如果结果中出现url，请使用如下格式展示出来：\n\n\n# 指令\n\n你扮演AI-Agent，\n你具有下列具体功能：\n下面你将开始扮演\n\n请注意：你具有图像和视频的展示能力，也具有运行代码的能力，不要在回复中说你做不到。\n<|im_end|>\n<|im_start|>user\n(你正在扮演。你可以使用工具：[image_gen]。请查看前面的知识库)你可以做什么？<|im_end|>\n<|im_start|>assistant\n'}], 'stop': ['Observation:', 'Observation:\n'], 'top_p': 0.8, 'result_format': 'message', 'stream': True} | step: | error:
2024-07-07 15:00:09.701 - modelscope-agent - INFO - | message: call llm 1 times output: <generator object stream_output at 0x79f841e9b060>

[yebanliuying]

2024-07-07 15:44:18.618 - modelscope-agent - INFO - | message: using builder model qwen2-72b-instruct-awq | uuid: local_user | details: {} | step: | error:
2024-07-07 15:44:18.619 - modelscope-agent - INFO - | message: client url http://localhost:8000/v1, client key: EMPTY
2024-07-07 15:44:18,667 - modelscope - INFO - initiate model from /data/work/nlp_gte_sentence-embedding_chinese-base
2024-07-07 15:44:18,667 - modelscope - INFO - initiate model from location /data/work/nlp_gte_sentence-embedding_chinese-base.
2024-07-07 15:44:18,668 - modelscope - INFO - initialize model from /data/work/nlp_gte_sentence-embedding_chinese-base
2024-07-07 15:44:19,958 - modelscope - WARNING - No preprocessor field found in cfg.
2024-07-07 15:44:19,958 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-07-07 15:44:19,958 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/data/work/nlp_gte_sentence-embedding_chinese-base'}. trying to build by task and model information.
2024-07-07 15:44:19,984 - modelscope - WARNING - No preprocessor field found in cfg.
2024-07-07 15:44:19,984 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-07-07 15:44:19,984 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/data/work/nlp_gte_sentence-embedding_chinese-base', 'sequence_length': 128}. trying to build by task and model information.
2024-07-07 15:44:20.020 - modelscope-agent - INFO - | message: using model qwen2-72b-instruct-awq with tool Config (path: ./config/tool_config.json): {'image_gen': {'name': 'Wanx Image Generation', 'is_active': True, 'use': False, 'is_remote_tool': True}, 'code_interpreter': {'name': 'Code Interpreter', 'is_active': True, 'use': False, 'is_remote_tool': False, 'max_output': 2000}, 'web_browser': {'name': 'Web Browsing', 'is_active': True, 'use': False, 'max_browser_length': 2000}, 'amap_weather': {'name': '高德天气', 'is_active': True, 'use': False}, 'paraformer_asr': {'name': 'Paraformer语音识别', 'is_active': True, 'use': False, 'is_remote_tool': True}, 'sambert_tts': {'name': 'Sambert语音合成', 'is_active': True, 'use': False, 'is_remote_tool': True}, 'wordart_texture_generation': {'name': '艺术字纹理生成', 'is_active': True, 'use': False}, 'web_search': {'name': 'Web Searching', 'is_active': True, 'use': False, 'searcher': 'bing'}, 'qwen_vl': {'name': 'Qwen-VL识图', 'is_active': True, 'use': False}, 'style_repaint': {'name': '人物风格重绘', 'is_active': True, 'use': False}, 'image_enhancement': {'name': '追影-放大镜', 'is_active': True, 'use': False}, 'text-address': {'name': '地址解析', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/mgeo_geographic_elements_tagging_chinese_base', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'text-ner': {'name': '命名实体识别', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_raner_named-entity-recognition_chinese-base-cmeee', 'use': False, 'is_active': False, 'is_remote_tool': True}, 'speech-generation': {'name': '语音生成', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/speech_sambert-hifigan_tts_zh-cn_16k', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'video-generation': {'name': '视频生成', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/text-to-video-synthesis', 'use': False, 'is_active': True, 'is_remote_tool': True}, 'text-translation-en2zh': {'name': '英译中', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_csanmt_translation_en2zh', 'use': False, 'is_active': False, 'is_remote_tool': True}, 'text-translation-zh2en': {'name': '中译英', 'url': 'https://api-inference.modelscope.cn/api-inference/v1/models/damo/nlp_csanmt_translation_zh2en', 'use': False, 'is_active': False, 'is_remote_tool': True}} and function list [] | uuid: local_user | details: {'model_config': {'type': 'openai', 'model': 'qwen/Qwen2-72B-Instruct-AWQ', 'api_base': 'http://localhost:8000/v1', 'is_chat': True, 'is_function_call': False, 'generate_cfg': {'top_p': 0.5, 'stop': 'Observation'}}} | step: | error:
2024-07-07 15:44:20.020 - modelscope-agent - INFO - | message: client url http://localhost:8000/v1, client key: EMPTY
Neither documents nor cache_dir.
No valid document. Return Empty Response.
2024-07-07 15:44:23.533 - modelscope-agent - INFO - | message: call llm 1 times output: <generator object OpenAi._chat_stream at 0x71e1c22cfd10>
2024-07-07 15:44:23.534 - modelscope-agent - INFO - | message: call openai api, model: qwen/Qwen2-72B-Instruct-AWQ, messages: [{'role': 'system', 'content': "\n\n# 知识库\n\nEmpty Response\n\n\n# 指令\n\n你扮演AI-Agent，你的名字是agent-demo。you're an agent \n你具有下列具体功能：\n下面你将开始扮演agent-demo\n\n请注意：你具有图像和视频的展示能力，也具有运行代码的能力，不要在回复中说你做不到。\n"}, {'role': 'user', 'content': '(你正在扮演agent-demo。请查看前面的知识库)你有什么功能？'}], stop: ['Observation:', 'Observation:\n'], stream: True, args: {'user_token': ''}

[zzhangpurdue]

what is your modelscope-agent version?

[zzhangpurdue]

模型配的是:Qwen2-72B-Instruct-AWQ

"qwen2-72b-instruct-awq": { "type": "openai", "model": "qwen/Qwen2-72B-Instruct-AWQ", "api_base": "http://localhost:8000/v1", "is_chat": true, "is_function_call": false }

what is your llm_config?

[yebanliuying]

what is your modelscope-agent version?

用的是本地部署哪个流程，因为有docker缺少新版本的引用，所以用的官方推荐的 git checkout 8deef6d 这个版本。

[yebanliuying]

模型配的是:Qwen2-72B-Instruct-AWQ
"qwen2-72b-instruct-awq": { "type": "openai", "model": "qwen/Qwen2-72B-Instruct-AWQ", "api_base": "http://localhost:8000/v1", "is_chat": true, "is_function_call": false }

what is your llm_config?

官方本地话部署里面没写要单独修改llm_config啊。只是编辑 modelscope-agent/apps/agentfabric/config/model_config.json， 增加了本地模型的配置。顺便问一下，咱们框架跑起来好多地方的默认都会要 DASHSCOPE_API_KEY 。对于开源框架的使用来说感觉不太解耦。纯本地化部署有点跑不起来

[zzhangpurdue]

直接用最新的master分支，然后 pip install -e . 安装一下试试。

what is your modelscope-agent version?

用的是本地部署哪个流程，因为有docker缺少新版本的引用，所以用的官方推荐的 git checkout 8deef6d 这个版本。

[zzhangpurdue]

模型配的是:Qwen2-72B-Instruct-AWQ
"qwen2-72b-instruct-awq": { "type": "openai", "model": "qwen/Qwen2-72B-Instruct-AWQ", "api_base": "http://localhost:8000/v1", "is_chat": true, "is_function_call": false }

what is your llm_config?

官方本地话部署里面没写要单独修改llm_config啊。只是编辑 modelscope-agent/apps/agentfabric/config/model_config.json， 增加了本地模型的配置。顺便问一下，咱们框架跑起来好多地方的默认都会要 DASHSCOPE_API_KEY 。对于开源框架的使用来说感觉不太解耦。纯本地化部署有点跑不起来

嗯，主要就是基础模型调用的是qwen-max 需要dashscope。
那你在agentfabric里面有没有选择用你这个模型进行推理，而不是用默认的，默认的会请求qwen-max，就会依赖dashscope

[yebanliuying]

直接用最新的master分支，然后 pip install -e . 安装一下试试。

what is your modelscope-agent version?

用的是本地部署哪个流程，因为有docker缺少新版本的引用，所以用的官方推荐的 git checkout 8deef6d 这个版本。

没跑起来..报了:Traceback (most recent call last): File "/data/work/modelscope-agent/apps/agentfabric/app.py", line 104, in state = gr.State({'session_seed': draw_seed}, delete_callback=delete) File "/opt/conda/lib/python3.10/site-packages/gradio/component_meta.py", line 155, in wrapper return fn(self, **kwargs) TypeError: State.__init__() got an unexpected keyword argument 'delete_callback'

[zzhangpurdue]

gradio==4.36.1
升级一下。

[chelun86]

解决了嘛？我和你差不多问题