This project is an extended version of the original @OpenManus initiative. Inspired by successful RL tunning for reasoning LLM such as Deepseek-R1, QwQ-32B, we will explore new paradigms for RL-based LLM agent tuning, particularly building upon foundations.
We are committed to regularly updating our exploration directions and results in a dynamic, live-streaming fashion. All progress, including rigorous testing on agent benchmarks such as GAIA, AgentBench, WebShop, and OSWorld, and tuned models, will be openly shared and continuously updated.
We warmly welcome contributions from the broader community—join us in pushing the boundaries of agent reasoning and tool integration!
Code and dataset coming soon! Stay tuned!
- OpenManus-RL
- Running
- Related Work
- Acknowledgement
- Community Group
- Citation
- [2025-03-09] 🍺 We collect and opensource our Agent SFT dataset at Huggingface, go try it!
- [2025-03-08] 🎉 We are collaborating with @OpenManus from Metagpt to work on this project together!
- [2025-03-06] 🥳 We(UIUC-Ulab) are announcing our live-streaming project, OpenManus-RL.
@Kunlun Zhu(Ulab-UIUC), @Jiayi Zhang(MetaGPT), @Xiangxin Zhou, @Yanfei Zhang, @Yingxuan Yang, @Weijia Zhang, @Muxin Tian, @Haofei Yu(Ulab-UIUC)
We wholeheartedly welcome suggestions, feedback, and contributions from the community! Feel free to:
We welcome contributions, including fine-tuning codebase, tuning dataset, environment setup, and computing resources. Create issues for feature requests, bug reports, or ideas. Submit pull requests to help improve OpenManus-RL. Or simply reach out to us for direct collaboration. Important contributors will be listed as co-authors to our paper.
-
Agent Environment Support Setting up LLM agent environment for online RL tunning.
-
Agent Trajectories Data Collection Connect to specialized reasoning models such as deepseek-r1, QwQ-32B for more complex inference tasks to collect comprehensive agent trajectories.
-
RL-Tuning Model Paradigm Provide an RL fine-tuning approach for customizing the agent’s behavior in our agent environment.
-
Test on Agent Benchmarks Evaluate our framework on agentic benchmark such as Webshop, GAIA, OSWorld, AgentBench
Our method proposes an advanced reinforcement learning (RL)-based agent tuning framework designed to significantly enhance reasoning and decision-making capabilities of large language models (LLMs). Drawing inspiration from RAGEN's Reasoning-Interaction Chain Optimization (RICO), our approach further explores novel algorithmic structures, diverse reasoning paradigms, sophisticated reward strategies, and extensive benchmark environments.
To benchmark the reasoning capabilities effectively, we evaluate multiple state-of-the-art reasoning models:
- GPT-O1
- Deepseek-R1
- QwQ-32B
Each model provides unique reasoning capabilities that inform downstream optimization and training strategies.
We experiment with a variety of rollout strategies to enhance agent planning efficiency and reasoning robustness, including:
- Tree-of-Thoughts (ToT): Employs tree-based reasoning paths, enabling agents to explore branching possibilities systematically.
- Graph-of-Thoughts (GoT): Utilizes graph structures to represent complex reasoning dependencies effectively.
- DFSDT (Depth-First Search Decision Trees): Optimizes action selection through depth-first search, enhancing long-horizon planning.
- Monte Carlo Tree Search (MCTS): Explores reasoning and decision paths probabilistically, balancing exploration and exploitation effectively.
These methods help identify optimal rollout techniques for various reasoning tasks.
We specifically analyze and compare several reasoning output formats, notably:
- ReAct: Integrates reasoning and action explicitly, encouraging structured decision-making.
- Outcome-based Reasoning: Optimizes toward explicit outcome predictions, driving focused goal alignment.
These formats are rigorously compared to derive the most effective reasoning representation for various tasks.
We investigate multiple post-training methodologies to fine-tune agent reasoning effectively:
- Supervised Fine-Tuning (SFT): Initializes reasoning capabilities using human-annotated instructions.
- Generalized Reward-based Policy Optimization (GRPO): Incorporates:
- Format-based Rewards: Rewards adherence to specified reasoning structures.
- Outcome-based Rewards: Rewards accurate task completion and goal attainment.
- Proximal Policy Optimization (PPO): Enhances agent stability through proximal updates.
- Direct Preference Optimization (DPO): Leverages explicit human preferences to optimize agent outputs directly.
- Preference-based Reward Modeling (PRM): Uses learned reward functions derived from human preference data.
We train specialized agent reward models using annotated data to accurately quantify nuanced reward signals. These models are then leveraged to guide agent trajectory selection during both training and evaluation phases.
During the inference phase, trajectory scaling methods are implemented, allowing agents to flexibly adapt to varying task complexities, thus enhancing robustness and performance in real-world scenarios.
Agents are equipped with action-space awareness, employing systematic exploration strategies designed to navigate complex action spaces effectively, ultimately maximizing expected rewards.
We integrate insights and methodologies from leading RL tuning frameworks, including:
- Verl
- TinyZero
- OpenR1
- Trlx
Through these frameworks, agents can effectively balance exploration and exploitation, optimize reasoning processes, and adapt dynamically to novel environments.
In summary, our method systematically integrates advanced reasoning paradigms, diverse rollout strategies, sophisticated reward modeling, and robust RL frameworks, significantly advancing the capability and adaptability of reasoning-enhanced LLM agents.
OpenManusRL-Dataset combines agent trajectories from AgentInstruct and Agent-FLAN with features:
- 🔍 ReAct Framework - Reasoning-Acting integration
- 🧠 Structured Training - Separate format/reasoning learning
- 🚫 Anti-Hallucination - Negative samples + environment grounding
- 🌐 6 Domains - OS, DB, Web, KG, Household, E-commerce
| Source | Trajectories | Avg Turns | Key Features |
|---|---|---|---|
| AgentInstruct | 1,866 | 5.24 | Multi-task QA, CoT reasoning |
| Agent-FLAN | 34,442 | 3-35 | Error recovery patterns, diverse real-world tasks |
| Combined | 36,308 | 4-20 | Enhanced generalization, broader task coverage |
- text-generation: ReAct-style instruction following
- conversational-ai: Tool-augmented dialogues
English
ReAct Pattern Example:
{
"id": "os_0",
"conversations": [
{"role": "user", "content": "Count files in /etc"},
{"role": "assistant", "content": "Think: Need reliable counting method\nAct: bash\n```bash\nls -1 /etc | wc -l\n```"},
{"role": "user", "content": "OS Output: 220"},
{"role": "assistant", "content": "Think: Verified through execution\nAct: answer(220)"}
]
}A simplified library for Supervised Fine-Tuning (SFT) and GRPO tunning of language models for agentic system. (developed upon Open-R1 from huggingface) We are still laboriously developing this part, welcome feedback.
First, create a conda environment and activate it:
conda create -n openmanus-rl python=3.10
conda activate openmanus-rlThen, install the required dependencies:
pip install -r requirements.txtSupervised Fine-Tuning (SFT)
Basic Usage
To fine-tune a model on a single GPU:
python -m openmanus_rl.sft \
--model_name_or_path Qwen/Qwen2.5-1.5B-Instruct \
--dataset_name CharlieDreemur/OpenManus-RL \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--max_seq_length 4096 \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--bf16 \
--logging_steps 5 \
--output_dir data/sft-outputDistributed Training with Accelerate
For multi-GPU training using Accelerate:
accelerate launch --config_file=configs/accelerate_configs/zero3.yaml openmanus_rl/sft.py \
--model_name_or_path Qwen/Qwen2.5-1.5B-Instruct \
--dataset_name CharlieDreemur/OpenManus-RL \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--max_seq_length 4096 \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--bf16 \
--logging_steps 5 \
--output_dir data/sft-outputBasic Usage To fine-tune a model using GRPO on a single GPU:
python -m openmanus_rl.grpo \
--model_name_or_path Qwen/Qwen2.5-1.5B-Instruct \
--dataset_name CharlieDreemur/OpenManus-RL-GRPO \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--max_seq_length 4096 \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--bf16 \
--reward_funcs accuracy format tag_count \
--logging_steps 5 \
--output_dir data/grpo-outputDistributed Training with Accelerate For multi-GPU training using Accelerate:
accelerate launch --config_file=configs/accelerate_configs/zero3.yaml openmanus_rl/grpo.py \
--model_name_or_path Qwen/Qwen2.5-1.5B-Instruct \
--dataset_name CharlieDreemur/OpenManus-RL-GRPO \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--max_seq_length 4096 \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--gradient_checkpointing \
--bf16 \
--reward_funcs accuracy format tag_count \
--logging_steps 5 \
--output_dir data/grpo-output- Offline Training of Language Model Agents with Functions as Learnable Weights. [paper]
- FIREACT : TOWARD LANGUAGE AGENT FINE-TUNING. [paper]
- AgentTuning: Enabling Generalized Agent Abilities for LLMs. [paper]
- ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy. [paper]
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents. [paper]
- ATLAS: Agent Tuning via Learning Critical Steps. [paper]
- Toolformer: Language Models Can Teach Themselves to Use Tools. [paper]
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. [paper]
- Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models. [paper]
- AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning. [paper]
- Training Language Models to Follow Instructions with Human Feedback. [paper]
- Deepseekmath: Pushing the Limits of Mathematical Reasoning in Open Language Models. [paper]
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. [paper]
- AgentBench: Evaluating LLMs as Agents. paper
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. paper
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. paper
- WebShop: Towards Scalable Real-World Web Interaction with Autonomous Agents. paper
- GAIA: a benchmark for General AI Assistants. paper
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. paper
- RAGEN: Training Agents by Reinforcing Reasoning. [code]
We extend our thanks to ulab-uiuc (https://ulab-uiuc.github.io/) and Openmanus (https://github.com/mannaandpoem/OpenManus)) team from MetaGPT for their support and shared knowledge. Their mission and community contributions help drive innovations like OpenManus forward. We welcome all developers who are interested in this project can reach out to (kunlunz2@illinois.edu)
Stay tuned for updates and the official release of our repository. Together, let's build a thriving open-source agent ecosystem!
Join our networking group on Wecgat and share your experience with other developers!
Please cite the following paper if you find OpenManus helpful!
@misc{OpenManus,
author = {OpenManus-RL Team},
title = {OpenManus-RL: Open Platform for Generalist LLM Reasoning Agents with RL optimization},
year = {2025},
organization = {GitHub},
url = {https://github.com/OpenManus/OpenManus-RL},
}
