增加grpo多次工具调用训练 #3503
Open
增加grpo多次工具调用训练 #3503
+405
−72
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
数据集可以放在modelscope上，然后使用model_id进行使用嘛 然后 最外层目录的文件 放置在examples/train/grpo中单读建立个文件夹，然后放置在里面，然后写给文档（最佳实践）来介绍一下不 |
lint过一下，会进行代码的整理 |
好的 |
|
把数据集上传到了modelscope，然后新增一个最佳实践多轮工具调用实践 |
examples/train/rft/rft.py
Outdated
| @@ -22,7 +22,8 @@ def do_sample(model: str, model_type: str, dataset: List[str], iter: int): | |||
| for device in range(device_count): | |||
| sample_cmd = (f'{conda_prefix} CUDA_VISIBLE_DEVICES={device} swift sample ' | |||
| f'--model {model} --model_type {model_type} ' | |||
| f'--dataset {" ".join(dataset)} ' | |||
| f'--dataset {' | |||
scripts/benchmark/exp_utils.py
Outdated
| @@ -122,7 +122,7 @@ def run(self, exp: Experiment): | |||
| exp.runtime = runtime | |||
| envs = deepcopy(runtime.get('env', {})) | |||
| envs.update(os.environ) | |||
| logger.info(f'Running cmd: {runtime["running_cmd"]}, env: {runtime.get("env", {})}') | |||
| logger.info(f'Running cmd: {runtime['running_cmd']}, env: {runtime.get('env', {})}') | |||
|
期待调用工具的GRPO支持 |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
test_grpo_tool.py:训练测试脚本
math_tool.py:测试的工具，定义新运算，接口主要是判断是否继续和给格式奖励，还有online的result输入
相关数据集也放在目录中，比较混乱，主要还是修改gpro_trainer.py
GRPO args需要新增参数:
is_reward_tool_call:是否累加计算每个tool_call的格式奖励，但应该设置上限，否则可能会学到无限调用，但不输出正确答案。
tool_call_weight:tool_call_format奖励的权重