Agent Reinforcement Trainer (ART)

Training agentic LLMs

Agent Reinforcement Trainer (ART)

Reinforcement learning becomes more complex when the “agent” is an LLM. Instead of choosing a simple action like moving left or right—an LLM agent produces multi-step reasoning traces, tool calls, conversations and plans.

Training such agents requires a system that can collect these trajectories, assign rewards and update the model reliably.

ART (Agent Reinforcement Trainer), built by OpenPipe, provides that system.

It is an open-source framework designed specifically for training agentic LLMs from experience. ART handles the pieces that are difficult to engineer manually:

• running the agent to generate full trajectories
• capturing decisions, tool use and reasoning steps
• scoring each trajectory with a custom reward function
• updating the model using reinforcement learning

ART uses a lightweight client that wraps your existing agent with minimal changes. The client communicates with an ART training server, which manages rollouts, reward computation, batching and optimization.

A key feature is ART’s support for Group Relative Policy Optimization (GRPO), an RL algorithm widely used for training LLMs. GRPO allows the model to learn from trajectory-level rewards rather than token-level labels, which is essential for improving behaviors like planning, correction and tool use.

The workflow looks like this:

• You start with your existing agent code—ART simply wraps it so you don’t need to rewrite anything.
• The agent runs and produces a trajectory.
• The trajectory is scored using a reward function.
• ART applies GRPO (or another supported RL method) to update the policy.
• The loop repeats, gradually improving the agent’s behavior.

By handling rollout execution, reward processing and policy optimization, ART lets developers focus on designing effective reward signals and agent strategies rather than building RL infrastructure.

Key takeaways

• Agent RL needs trajectory capture + credit assignment—ART centralizes that service-side.
• GRPO at trajectory scope matches tool-use and planning better than token-only supervision.