Training Reasoning Models in Simulation to Natively Use Real Robots

TL;DR

We trained an open 27B reasoning model to use real robots through reinforcement learning entirely in simulation, without demonstrations. The base model can barely use a robot; after training, it succeeds most of the time, both on simulation benchmarks and on a real robot, without sacrificing its general reasoning, coding and vision capabilities.

The model uses a robot the way a computer-use agent uses a computer: it observes the environment, reasons with a chain of thought, and outputs the arm’s next move, without needing a harness. Our training, Robot-Use RL, applies reinforcement learning to robot tasks in simulation, similar to how frontier models learned computer use. We find that what it learns in simulation carries over to a new robot, to the real world, and to instructions it has never seen.

On LIBERO, Robot-Use RL takes it from 1.6% to 74.4%, close to GPT-6 Astra and GPT-6.1 Sol at max effort. It also follows language instructions it was never trained on: in familiar scenes with a new task, it succeeds 57.4% of the time, and π0.5 only 0.75%. Trained only in simulation, it succeeds in 60% of our trials on a real dual-arm robot without any harness, up from 0% before training. And it keeps its general capabilities: across nine reasoning, coding and vision benchmarks, it shows no measurable loss.

We believe that Robot-Use RL in simulation is a simple recipe for real-world robot-use agents, with a lot of room to grow, and that with reasoning, these agents can generalize far beyond the tasks they were trained on. We hope these results can motivate the community to further explore Robot-Use RL as part of foundation model post-training. We release the model, the code and the full training infrastructure.

All results without a harness

1How it works

Our model uses a robot the way a computer-use agent uses a computer.

ObservesReasonsWritesCarried out by

Computer‑use agentfor comparison

A screenshot

“The Save button is at the top right.”

Reasoning, in text

click(812, 431)

A click or a keystroke

Operating system

Moves the cursor and clicks

Robot-use agent

Three camera views and the arm’s state

“The gripper is right above the eraser. Lower it until the fingers are on either side.”

A chain of thought

{"left_arm": {
   "x": 0.0, "y": 1.0,
   "z": -17.4, "roll": 0.0,
   "pitch": 0.0, "yaw": 0.0},
 …}

The end-effector’s next pose, without harness or skills

IK + controller

Inverse kinematics turns the pose into joint angles; the arm’s controller moves there

Contextthe episode, round by round

InstructionRound 1···Round tRound t+1Round t+2

A computer-use agent[1, 2] looks at a screenshot, reasons about what to do, and writes a click as text, which the operating system then carries out. Our model does the same with a robot[3], without needing a harness. In each round, it observes the camera frames and the arm’s state, reasons in a chain of thought, and writes the end-effector’s next pose as text. Inverse kinematics turns that pose into joint angles, and the arm’s controller moves the arm there (Figure 1).

Nothing else sits in between: there are no perception tools, grasp or motion planners[4], or skill library[5, 6], so the model itself has to find the objects in the camera frames, work out how to reach and manipulate them, and notice when a move has not gone as planned. Earlier rounds stay in its context, so it remembers what it has seen and done and can build each move on the ones before it. It learns all of this from task success alone, with reinforcement learning on robot tasks in simulation, which we call Robot-Use RL.

2Robot-Use Reinforcement Learning

Learning only from whether each task succeeds, our model goes from 1.6% to 74.4% on LIBERO[7], close to GPT-6 Astra[8] and GPT-6.1 Sol[9] at max effort.

RL makes progress even from a near-zero start. On Goal, Spatial and Object, our base model, Qwen3.8-27B[10], succeeds in only about 2% of rollouts, so almost every rollout ends with no reward; even so, the training success rate on these three suites rises to about 70% within 148 steps. We leave LIBERO-Long out at this stage, both because the base model almost never succeeds on it, which would give RL almost no signal, and because holding it out gives us a clean test of transfer.

The skills learned on these three suites also transfer strongly to tasks the model has never trained on. At step 148, before any training on LIBERO-Long, the model already solves 39.0% of it, up from 0.4% for the base model, and its success rate on LIBERO as a whole is 64.0%; we return to this transfer in section 4. Training on all four suites from step 149 then raises the success rate to 69.5% at step 184 and 74.4% at step 225 (Figure 2), each about 5 points above the last.

These results show that a base model with a very low initial success rate can still be trained to a high success rate with RL alone, from 1.6% to 74.4% in this case. The capability it acquires is also general rather than specific to LIBERO’s training tasks: in the following sections, we show that it stays robust when LIBERO’s scenes and tasks are perturbed and follows instructions it was never trained on (section 3), and that it transfers to a new benchmark, a new robot and the real world (section 4).

3Robustness and instruction following

Trained only on LIBERO, our model generalizes beyond its training tasks: it stays robust when they are perturbed, and it follows instructions it has never seen.

3.1Robustness on LIBERO-Plus and LIBERO-Pro

π0.5[11] beats our model on LIBERO, where it is supervised fine-tuned on demonstrations of every task, but collapses when a task’s goal or object layout changes. Both policies are trained only on LIBERO. We then test them on LIBERO-Plus[12] and LIBERO-Pro[13], which perturb LIBERO’s tasks in ways neither policy has seen. LIBERO-Plus mainly perturbs the observation, such as the camera, the lighting, the background and sensor noise, and the two policies lose a similar amount on it: 11.2 points for π0.5 and 12.2 for our model. LIBERO-Pro also changes the goals and the object layouts. Our model again loses about 12 points there, but π0.5 loses 43.6, so its 22.5-point lead on LIBERO turns into an 8.8-point deficit (Figure 3).

4Transfer to a new benchmark, a new robot and the real world

Trained only in simulation, our model succeeds zero-shot on new tasks and on a new robot in a new simulator, and in 60% of our trials on a real robot in the real world, where the base model never succeeds.

4.1Zero-shot transfer to new tasks, robots and benchmarks

Without training on LIBERO-Long episodes, our model reaches 39.0% on LIBERO-Long, where the base model succeeds in 0.4% of episodes (Figure 5). With so few successes, training on LIBERO-Long from the start would have given RL almost no reward to learn from. Transfer from the other three suites changes this: when LIBERO-Long joins training at step 149, the model’s successes on it give RL a signal to train on.

The skills also carry over when both the simulator and the robot change. RoboDojo[14] is a different environment, and its tasks use a dual-arm ARX instead of LIBERO’s single-arm Franka. Trained only on LIBERO, our model succeeds in 39.9% of RoboDojo’s general_pickup episodes, nearly four times the base model’s 10.7%; on align_blocks it reaches 17.0%, against 13.0% (Figure 5). What the model learns on LIBERO is therefore not tied to LIBERO’s simulator or to its arm.

We also evaluate our model zero-shot on RoboLab-120[15], a third simulation benchmark, in which a Franka arm with a Robotiq gripper performs 120 tabletop tasks at three levels of difficulty, with ten episodes per task. Our model has trained with RL only on LIBERO and RoboDojo (section 4.2), and RoboLab’s tasks and scenes are new to it. It succeeds in 37.3% of RoboLab-120’s 1,200 episodes, while the base model succeeds in 5.4% (Figure 6). This is on par with Cosmos3-Nano-Policy[16] at 36.8% and above π0.5[11] at 28.0%, DreamZero[17] at 25.7% and GR00T N1.6[18] at 7.2%, four generalist robot policies on the RoboLab leaderboard[19].

5Stays a general model

RL on robot tasks leaves its other skills in place: no measurable loss across nine benchmarks.

We compare our model with the base model on the same questions, paired. Over nine benchmarks, every 95% interval includes zero (Figure 11, Table A5), in line with the finding that RL on a new task erodes a model’s other skills less than supervised fine-tuning does[20].

Our model, which reaches 74.4% on LIBERO and 75.0% on RoboDojo’s general_pickup and align_blocks, still scores 98.5 on AIME 2026[21], 89.7 on GPQA Diamond[22] and 88.4 on LiveCodeBench[23], and stays within two points of the base model on every vision benchmark: the same model now reasons, writes code, answers questions about images and uses a robot.

6Reasoning effort and inference speed

Although our model is trained at xhigh effort only, the effort setting in the chat template still controls how long it reasons. No RL rollout ever runs at a lower setting, yet at low effort our model thinks 0.86k tokens per round instead of 3.23k (Figure 12), so RL at a single effort has not fixed the length of its reasoning. Its success falls with that length. On the same checkpoint, its LIBERO success rate is 21.9% at low effort against 74.4% at xhigh: with less than a third of the thinking, it keeps less than a third of its success. Most of what it can do on LIBERO therefore comes from reasoning at length before each move.

Because our model reasons this long before every move, how fast it decodes decides how fast it can act. With NVFP4 weights[24] and a DFlash2 drafter[25], our model decodes 52 tokens/s on an M5 Max in Inco’s Splash engine[26], and, with our megakernel[27], 221 tokens/s on a single RTX 5090 and 747 tokens/s on a single GB300 (Figure 12). At 747 tokens/s, a round of xhigh thinking takes about 4.3 s and a round at low effort about 1.1 s, so the effort setting also trades success for speed. We use Inco’s original DFlash2 checkpoint, with an average acceptance length of 3.2 tokens. We believe that a drafter tuned specifically for our model would have a longer acceptance length, which would make decoding faster. The model that uses the robot can therefore run locally, on a single consumer GPU or a laptop.

7Limitations

• We have not been able to scale up our training. Mainly because of limited GPU resources, we stopped after training on two RoboDojo tasks, and the training went smoothly. Training so far also covers only two simulators and a small set of tasks, and it starts from a 27B base model. We believe that scaling up, by training in more diverse simulations and on more diverse tasks and by starting from a stronger foundation model, could lead to much stronger robot-use performance.
• We did not use supervised fine-tuning (SFT), for two reasons. First, we lacked high-quality in-distribution data and were concerned that SFT would harm the model’s general capabilities[20]. Second, we think that robotics data, even when available, should be added at the pretraining and mid-training stages rather than used for SFT on a model that has already been post-trained. This is why we chose to train directly with RL, and with RL alone.
• Our model is still slow at carrying out tasks, because it reasons at length before every move. We apply no length penalty during RL and place no constraint on the length of its chain of thought. Inference itself is fast, at 747 tokens/s on a single GB300, but a round of this thinking still takes about 4.3 s on average (Figure 12), and in a few cases the chain of thought can reach 32k tokens, so that even with inference this fast, a single move takes more than half a minute. Shortening the reasoning without giving up success, for example by rewarding shorter chains of thought during RL, is left to future work.
• Our base model, Qwen3.8-27B, is relatively small, and with its limited capabilities it often lacks knowledge about the embodiment it controls, so this information has to be provided in the prompt. As a result, the task success rate can depend considerably on whether the model can form a clear understanding of the embodiment and the environment from the prompt, and the model reaches its best performance when the prompt gives it enough of the information it needs. For harder tasks, such as those in our demos, our model succeeds only after we describe the embodiment and the task to it in detail in the prompt. Its performance therefore depends heavily on the prompt, but this also demonstrates, to some extent, its ability to follow detailed language instructions.
• Our current model is not yet able to plan action chunks reliably, so we have it output the end-effector’s next pose; inverse kinematics and a controller then move the arm to that pose and stop it there. Accuracy therefore depends partly on the controller: a poorly implemented one can make the arm jump at some positions and miss the target pose, and an inefficient one uses up the task’s step budget and leaves fewer moves, which matters mainly in simulated evaluation. This design also increases the number of rounds a task needs: since each round moves the arm to a single pose, a complex motion that the model could otherwise complete in one round with one action chunk now has to be split into several rounds. We hope future work can improve this.

References

1. 1OpenAI (2025). Computer-Using Agent.
2. 2Anthropic (2024). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku.
3. 3Isola (2026). Robot-Use Agents.
4. 4Yang et al. (2026). ORA: Open Robot-Use Agents.
5. 5Zhang et al. (2026). Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents.
6. 6Lu et al. (2026). ASPIRE: Agentic /Skills Discovery for Robotics.
7. 7Liu et al. (2023). LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning.
8. 8OpenAI (2026). GPT-6 Astra: A new generation of intelligence.
9. 9OpenAI (2026). Introducing GPT-6.1 Sol.
10. 10Qwen Team (2026). Qwen3.8-27B.
11. 11Black et al. (2025). π0.5: a Vision-Language-Action Model with Open-World Generalization.
12. 12Fei et al. (2026). LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models.
13. 13Zhou et al. (2025). LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization.
14. 14Chen et al. (2026). RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies.
15. 15Yang et al. (2026). RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies.
16. 16NVIDIA (2026). Cosmos 3: Omnimodal World Models for Physical AI.
17. 17Ye et al. (2026). World Action Models are Zero-shot Policies.
18. 18NVIDIA GEAR Team (2025). GR00T N1.6: An Improved Open Foundation Model for Generalist Humanoid Robots.
19. 19NVIDIA (2026). RoboLab Leaderboard.
20. 20Shenfeld et al. (2026). RL’s Razor: Why Online Reinforcement Learning Forgets Less.
21. 21MAA (2026). American Invitational Mathematics Examination (AIME).
22. 22Rein et al. (2024). GPQA: A Graduate-Level Google-Proof Q&A Benchmark.
23. 23Jain et al. (2025). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.
24. 24Alvarez et al. (2025). Introducing NVFP4 for Efficient and Accurate Low-Precision Inference.
25. 25Inco AI (2026). DFlash 2: Keep Drafting Parallel.
26. 26Inco AI (2026). Splash: A Local Engine Built Around the Model.
27. 27NVIDIA (2026). Kernel Design Agents (KDA).
28. 28NVIDIA (2024). NVIDIA Model Optimizer.
29. 29Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
30. 30Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale.
31. 31Marek and Ryabinin (2026). Score Centering Stabilizes Off-policy Reinforcement Learning.
32. 32Williams (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.
33. 33Schulman et al. (2017). Proximal Policy Optimization Algorithms.
34. 34Loshchilov and Hutter (2019). Decoupled Weight Decay Regularization.
35. 35Sheng et al. (2025). HybridFlow: A Flexible and Efficient RLHF Framework.
36. 36Liu et al. (2025). LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference.
37. 37Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.
38. 38Jacobs et al. (2024). System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models.
39. 39Shah et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision.
40. 40Zadouri et al. (2026). FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling.
41. 41Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost.
42. 42Mittal et al. (2025). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning.
43. 43Pyatkin et al. (2025). Generalizing Verifiable Instruction Following.
44. 44xAI (2024). Grok-1.5 Vision Preview.
45. 45Gemini Robotics Team et al. (2025). Gemini Robotics: Bringing AI into the Physical World.
46. 46Yue et al. (2024). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI.
47. 47Yue et al. (2025). MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark.
48. 48ModelScope Team (2024). EvalScope: Evaluation Framework for Large Models.
49. 49Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention.
50. 50Anthropic (2026). Introducing Claude Fable 5.2 and Claude Mythos 5.2.
51. 51Bai et al. (2024). DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning.
52. 52Black et al. (2025). π0: A Vision-Language-Action Flow Model for General Robot Control.
53. 53Carion et al. (2026). SAM 3: Segment Anything with Concepts.
54. 54Chu et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training.
55. 55Chu et al. (2026). OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies.
56. 56Fan et al. (2026). CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents.
57. 57Fang et al. (2026). When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs.
58. 58Guo et al. (2025). DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning.
59. 59Hou and Zhao (2026). LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models.
60. 60Huang et al. (2026). Scaling Online Exploration for Robot Agents.
61. 61Khazatsky et al. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset.
62. 62Kim et al. (2026). LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models.
63. 63Kim et al. (2025). OpenVLA: An Open-Source Vision-Language-Action Model.
64. 64Leet et al. (2026). The Bitter Lesson Comes to Robotics.
65. 65Liu et al. (2026). RoboICL: Embodied In-Context Learning with GPT-6 Astra.
66. 66Liu et al. (2025). What Can RL Bring to VLA Generalization? An Empirical Study.
67. 67Liu et al. (2026). AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics.
68. 68OpenAI (2025). Operator System Card.
69. 69Physical Intelligence (2025). openpi.
70. 70Pikus et al. (2026). Pretraining Knows What, Simulation Teaches How: Compositional Generalization for Agentic Robotics.
71. 71Qi et al. (2025). WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning.
72. 72Qin et al. (2025). UI-TARS: Pioneering Automated GUI Interaction with Native Agents.
73. 73RoboDojo Team (2026). RoboDojo Sim Leaderboard, Agent Track.
74. 74SenseNova et al. (2026). Has GPT-6-Astra Achieved Embodied Intelligence?
75. 75Si et al. (2026). Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens.
76. 76Wang et al. (2025). UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning.
77. 77Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.
78. 78Yan et al. (2026). Flex-π: A Multi-Stream World-Action Model with Compute Flexibility.
79. 79Yang et al. (2026). Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy.
80. 80Yuan et al. (2026). Democratizing Robot-Use Agents: A Unified Harness with Open Models.
81. 81Zhang et al. (2026). Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration.
82. 82Zhang et al. (2026). An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond.
83. 83Zhou et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents.
84. 84Zitkovich et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.

Appendix

Training setup, breakdowns by suite, effort and perturbation, and the general benchmarks in full.

BibTeX

@misc{pear2026,
  title  = {Training Reasoning Models in Simulation to Natively Use Real Robots},
  author = {Jiaming Tang and Zhuoyang Zhang and Junxian Guo and Fangzhou Zhao and
            Yilong Zhao and Zekai Li and Xingyang Li and Yuyang Zhao and Song Bian and
            Zhijian Liu and Song Han},
  year   = {2026},
  url    = {https://mit-han-lab.github.io/pear-blog/}
}