Agentic Coding, Reasoning and Evaluation
Agents learn from a loop, not a dataset. This is my attempt to walk through that loop, from the environment an agent works in to the tests that tell us whether it got better, and to share what building each piece in the open has taught us.
Language models got good by reading text. Agents get good by doing things and watching what happens: a command fails, a test passes, a web page turns out to be the wrong one, a reviewer sends the patch back. Once you take that seriously, the hard problems move. The model still matters, of course. But so do four things around it: the environment it works in, the experience it collects there, the signal that turns that experience into learning, and the evaluation that tells us whether any of it worked.
I find it helpful to read most of my recent work as a study of that loop. Some projects build one piece of it in the open: LegoFlow and SWE-Lego for experience, Lego-RL for learning inside real environments, and SWE-Review and SwingArena for review and evaluation. Others ask what goes wrong when the loop runs on images, the web or the physical world. Each paper answers its own question. Put side by side, though, they say a few things none of them says alone: the strongest model is rarely the best teacher, checking answers is harder than producing them, and the more capable agents get, the less we can take the honesty of the loop for granted.
On this page: Environment · Experience · Signal · Evaluation · Beyond the terminal · What ties it together · Open questions
1. The environment is part of the agent
It's easy to think of an agent as "the model", with the tooling around it as plumbing. That picture turned out to be wrong in a way that matters. A coding agent never looks at a repository directly. It sees whatever its harness decides to show it: which files and tool outputs make it into the context, when the conversation gets compacted, how errors are reported, what happens after a command fails. When we plugged the same starting model into three widely used harnesses (OpenHands, Claude Code and OpenCode) for Lego-RL, it already behaved noticeably differently in each one before we had trained anything. Change the harness and you have, in a real sense, a different agent.
That shaped how we built the system. Lego-RL trains inside the real, unmodified harness instead of a simplified copy, and treats the harness as part of the environment rather than part of the model. The obvious follow-up was whether one model could serve several harnesses at once. Training on a mix of all three got close to the average of three separately trained models and narrowed some of the gaps between them. That's encouraging, but every harness in those runs was one the model had seen during training. How an agent copes with a harness it has never met is still an open question.
The same thing shows up outside code. In multimodal search, the tools are the harness. MMSearch-Plus gives a browsing agent a set-of-mark tool for marking, cropping and zooming into an image before it searches, and that one design choice changes what evidence the agent can use at all. In robotics, RoboCodeX uses code as the bridge between what the robot sees and what it does, which makes its behavior something you can read, check and move to another robot.
2. Experience: what is worth learning from
Everything starts with tasks the environment can actually check. LegoFlow mines merged pull requests, rewrites each one so it doesn't give away the answer, builds an isolated environment for it, and keeps it only if the original code fails the tests and the reference fix passes. Passing that check tells you a task is usable. It doesn't tell you the task is worth training on, so we also score difficulty with a simple rubric: how big the patch is, how much new logic it adds, how much of the codebase it touches, how involved the tests are and how detailed the instructions are. SWE-Lego mixes real tasks, which bring realism and depth, with synthetic ones created by injecting bugs, which bring scale. Terminal-Lego does the same for command-line work, starting from Stack Overflow questions.
The lessons that surprised us most were about trajectories, the recorded attempts a teacher agent makes on these tasks. The usual assumption is simple: pick the strongest agent you can afford as the teacher. Terminal-Lego found that this can be backwards. Trajectories that visibly look around, act on what they found and then check the result taught students far better than trajectories from stronger agents that jump straight to the answer. The paper puts it bluntly: "the best solver is the worst teacher." LegoFlow found something similar inside the reasoning itself. A teacher that thinks a little on every turn teaches less than one that thinks hard on the turns that need it, and most of the benefit comes from throwing out the shallowest traces, not from keeping only the longest ones.
There's also an honesty problem. Stronger teachers are better at finding shortcuts: tracking down the upstream pull request, copying the patch, or reading the fix straight out of Git history. A successful trajectory isn't necessarily an honest one. What worked for us was a combination: tell the agent plainly not to look up solutions, and then make it impossible anyway, by rewriting history and cutting off network access while the agent works and only restoring them for grading.
What we took away
- Pick trajectories for how they reason and how closely they engage with the environment, not just for whether they succeeded.
- Treat answer leakage as something to remove from the environment, not something to ask the agent nicely to avoid.
Read more: Lessons learned from LegoFlow-SWE · Terminal-Lego · SWE-Lego
3. Signal: from imitation to verification
If experience is what the agent tries, the signal is what it learns from those tries. What I've come to believe is that different signals teach genuinely different things, and that a well-aimed signal beats simply having more of it.
Imitation, done carefully. SWE-Lego was our test of how far plain supervised fine-tuning can go if you treat the loss with as much care as the data. With error masking, the model still sees the whole trajectory, mistakes included, but gets no gradient on the wrong steps, so it learns from its errors without learning to repeat them. A curriculum uses trajectory length as a rough stand-in for difficulty and works from easy to hard, mixing earlier data back in so nothing is forgotten. And when it comes to picking the best of several attempts, a verifier that simply answers "yes" or "no" in text does better than one with a scoring head, because it can lean on what the model already learned during pre-training.
RL in the real environment. Rewards from running tests sound naturally honest. They are only honest, though, if what the trainer optimizes matches what the model actually did. Harnesses quietly rewrite their own history, so a saved transcript isn't the exact token sequence the model produced. Lego-RL records tokens, probabilities and mixture-of-experts routing at the moment of generation, and replays that routing during training. The unnerving part is that mismatches here make no noise: training runs, curves move, nothing errors out. Difficulty also turns out to be relative. Tasks the model always solves, or never solves, teach it nothing, and the useful middle shifts as the agent improves. After training, the agents read more code before editing, reread their own edits and run the tests more often, at the cost of longer sessions. A newer recipe, single-rollout optimization, learns from just one attempt per task with a learned critic, but only once that critic has been warmed up enough to give sensible guidance.
Beyond pass or fail. Lots of useful tasks don't have one checkable answer. RLVRR swaps a single pass/fail check for a chain of smaller checks against good reference answers, covering both content and style, which brings some of the efficiency of supervised data into RL for open-ended writing. On-policy Verbal Distillation works with teachers that only give text feedback: their scores rank the student's own partial attempts, the good beginnings are kept and the weak continuations replaced, so the student keeps exploring instead of just copying.
The same tension in vision. With vision-language models, long chain-of-thought SFT and RL help in different places. SFT helps most on hard, multi-step questions but makes answers long-winded; RL helps across the board and keeps answers short. Every way we tried to combine them traded one benefit for the other instead of adding them up, which is what the synergy dilemma paper is about. SRPO takes a more focused route: first show the model what a good reflection looks like, then reward reflections that are short and actually change its reasoning.
Read more: Lego-RL · RL across harnesses · Single-rollout optimization
4. Evaluation and review: a mirror that's hard to fool
Here's the uncomfortable symmetry: the same skill that lets a teacher find shortcuts while we collect data lets an agent find shortcuts while we evaluate it. So as agents get stronger, I think benchmarks need to be built with the care of a security boundary, and ideally with more realism too. SwingArena (ICLR Oral) moves evaluation into continuous integration. Models take turns submitting patches and acting as reviewers who write the tests, across several programming languages, with retrieval to help them cope with very large codebases. REAgent (ICSE) starts one step earlier: it treats the issue description as a requirement that might be incomplete, and checks and improves it before trying to fix anything.
Review is where evaluation stops being a final exam and becomes part of the loop. SWE-Review trains a reviewer agent that explores the repository on its own, figures out the root cause, compares that with the submitted patch and writes back structured feedback. One-shot generation becomes generate, review, revise. Two things from that work stay with me. Review data turns out to make models better at fixing issues too. And because picking among a few attempts is cheaper than generating many more, a good reviewer makes it cheaper to spend extra compute at test time.
Multimodal benchmarks have a shortcut of their own: questions you can answer without really looking. MMSearch-Plus is built so the answer depends on small visual clues that have to be carried through several rounds of search. Most failures there come from not finding the right page or confusing similar-looking events, not from the last reasoning step. PhyX shows models leaning on memorized formulas and surface patterns rather than real physical understanding, and NAVERO looks at how video-language models handle relationships that change over time.
5. Beyond the terminal
When the environment becomes physical, the loop stays the same; what changes is the interface. RoboCodeX breaks a high-level instruction down, step by step, into actions on specific objects, each annotated with how the object can be grasped or moved and what is safe, and then writes those actions as code. Code plays the same role here that a harness plays for software agents: it's the layer where behavior becomes explicit, checkable and reusable from one robot to the next. PhyX measures the kind of physical reasoning these agents will need, and our survey of autoregressive models in vision traces how visual generation, from pixels to tokens to multi-scale models, is starting to reach into embodied settings.
What ties it together
- The strongest model is rarely the best teacher. Grounded traces beat stronger solvers in Terminal-Lego, stronger teachers cheat more in LegoFlow, and the same mismatch between how strong a teacher is and what a student can learn is behind our work on matching each prompt to the right teacher. What transfers is behavior the student can see and copy.
- Checking is the scarce resource. Verified tasks, verifiers that rank attempts, reviewers that diagnose patches, CI-based arenas: they're all ways of producing judgments we can trust. Generating attempts is cheap. Choosing between them reliably is what scales.
- The trainer has to see what the agent actually did. If the tokens, probabilities or routing used for training differ from what happened, learning quietly optimizes the wrong thing. In agent RL, you can't separate the systems work from the learning.
- Stronger agents find more shortcuts. Git history, upstream patches, visual questions you can answer from text alone: capable models find these more often. Honesty has to be built into the environment and the benchmark; a line in the prompt isn't enough.
- Depth beats coverage. Thinking hard where it's needed beats thinking a little everywhere. A useful reflection beats a long one. And long chains of thought have a real cost; they aren't free.
Open questions
- Can an agent trained on several harnesses do well in one it has never seen?
- When a single pass or fail arrives after dozens of actions, how do we tell which actions actually helped?
- What changes when an agent has to ask the user questions and go back and forth, instead of closing a ticket on its own?
- Can supervised and reinforcement signals be combined so that their benefits add up instead of trading off?
- How do these lessons carry over to much longer tasks, like building a whole repository from a written specification?
Open resources
- LegoFlow: code · data
- SWE-Lego: code · data and models
- Terminal-Lego: code · data
- Lego-RL: code · models and data
- SWE-Review: code · models and data
Papers (15)
-
REAgent: Requirement-Driven LLM Agents for Software Issue Resolution
Shiqi Kuang, Zhao Tian, Kaiwei Lin, Chaofan Tao, Shaowei Wang, Haoli Bai, Lifeng Shang, Junjie Chen
REAgent turns issue descriptions into structured, information-rich requirements, detects and iteratively refines low-quality requirements, and uses the refined specifications to guide more accurate software patches.
-
SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving
Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, Haoli Bai
SWE-Lego presents a strong SFT-only recipe (new dataset + curriculum/error masking) and studies verifier-based test-time scaling, pushing open 8B/32B models to SOTA on SWE-bench Verified.
-
LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
Yiming Du, Yuxin Jiang, Tao Yuan, Jianbo Dai, Shaowei Wang, Jierun Chen, Chaofan Tao, Xianzhi Yu, Lifeng Shang, Kam-Fai Wong, Xiaohui Li, Haoli Bai
LEGO-RL trains coding agents with reinforcement learning inside unmodified native harnesses, combining token-faithful proxying with MoE routing replay, sandboxed execution with stage-wise anti-hacking defenses, and observable training with live trajectory diagnostics.
-
SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review
Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, Haoli Bai
SWE-Review turns one-shot patch generation into a generate-review-revise loop: a reviewer agent explores the repository, traces the root cause on its own, decides whether to accept the PR, and returns structured feedback; review trajectories also transfer to issue resolution and make test-time scaling cheaper.
-
What Makes Interaction Trajectories Effective for Training Terminal Agents?
Sidi Yang, Chaofan Tao, Jierun Chen, Tiezheng Yu, Ruoyu Wang, Yuxin Jiang, Yiming Du, Wendong Xu, Jing Xiong, Taiqiang Wu, Lifeng Shang, Xiaohui Li, Ngai Wong, Haoli Bai
Using Terminal-Lego, a pipeline that turns real-world issues into environment-verified terminal tasks, the paper finds a pedagogical paradox: stronger agents are not better teachers. Environment-grounded inspect-act-verify trajectories transfer best, letting Qwen3-32B reach 24.3% on Terminal-Bench 2.0 with only 15.3k trajectories, about 30× less data than prior results.
-
SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving
Wendong Xu, Jing Xiong, Chenyang Zhao, Qiujiang Chen, Haoran Wang, Hui Shen, Zhongwei Wan, Jianbo Dai, Taiqiang Wu, He Xiao, Chaofan Tao, Z Morley Mao, Ying Sheng, Zhijiang Guo, Hongxia Yang, Bei Yu, Lingpeng Kong, Quanquan Gu, Ngai Wong
SwingArena is a CI-driven competitive benchmark where LLMs take turns as submitters and reviewers, paired with retrieval-augmented code generation to handle long-context, multi-language GitHub issue solving.
-
MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
Xijia Tao, Yihua Teng, Xinxing Su, Xinyu Fu, Jihao Wu, Chaofan Tao, Ziru Liu, Haoli Bai, Rui Liu, Lingpeng Kong
MMSearch-Plus is a 311-task benchmark and agent framework with set-of-mark tools that require provenance-aware, vision-in-the-loop search and cross-validation under retrieval noise.
-
OVD: On-policy Verbal Distillation
Jing Xiong, Hui Shen, Shansan Gong, Yuxin Cheng, Jianghan Shen, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Ngai Wong
On-policy Verbal Distillation replaces token-level matching with trajectory-level distillation using discrete teacher scores, reducing memory and improving student performance on web QA and math reasoning.
-
From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation
Yuxin Jiang, Yufei Wang, Qiyuan Zhang, Xingshan Zeng, Liangyou Li, Jierun Chen, Chaofan Tao, Haoli Bai, Lifeng Shang
RLVRR turns single verifiable end rewards into ordered reference-derived reward chains (content + style) to make RL for open-ended generation more efficient and less prone to reward hacking.
-
PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
Hui Shen, Taiqiang Wu, Qi Han, Yunta Hsieh, Jizhou Wang, Yuyue Zhang, Yuxin Cheng, Zijian Hao, Yuansheng Ni, Xin Wang, Zhongwei Wan, Kai Zhang, Wendong Xu, Jing Xiong, Ping Luo, Wenhu Chen, Chaofan Tao, Zhuoqing Mao, Ngai Wong
PhyX introduces a 3K-question multimodal benchmark spanning six physics domains to test physics-grounded reasoning, revealing large gaps between top models and human experts.
-
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, Peize Sun, Haibao Yu, Chao Yang, Wenqi Shao, Wenhai Wang, Jifeng Dai, Yu Qiao, Mingyu Ding, Ping Luo
RoboCodeX is a tree-structured multimodal code generation framework that decomposes high-level instructions into object-centric manipulation units with affordance and safety constraints, generalizing across robot platforms and reaching state-of-the-art results on manipulation and navigation tasks in simulation and on real robots.
-
Autoregressive Models in Vision: A Survey
Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, Chaofan Tao, Shen Yan, Huaxiu Yao, Lingpeng Kong, Hongxia Yang, Mi Zhang, Guillermo Sapiro, Jiebo Luo, Ping Luo, Ngai Wong
A comprehensive survey of visual autoregressive modeling, covering representation strategies and applications across image, video, 3D, and multimodal generation, plus open challenges.
-
SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, Chaofan Tao, Yangfan He, Mi Zhang, Shen Yan
SRPO is a reflection-aware RL framework that teaches MLLMs self-reflection and then optimizes concise, meaningful reflections with GRPO-style rewards to improve multimodal reasoning.
-
The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-training Techniques for Reasoning VLMs
Jierun Chen, Tiezheng Yu, Haoli Bai, Lewei Yao, Jiannan Wu, Kaican Li, Fei Mi, Chaofan Tao, Lei Zhu, Manyi Zhang, Xiaohui Li, Lu Hou, Lifeng Shang, Qun Liu
The study compares long-CoT SFT vs. RL for reasoning VLMs and finds their combination often fails to yield additive gains, instead trading off accuracy, verbosity, and style.
-
NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality
Chaofan Tao, Gukyeong Kwon, Varad Gunjal, Hao Yang, Zhaowei Cai, Yonatan Dukler, Ashwin Swaminathan, R Manmatha, Colin Jon Taylor, Stefano Soatto
NAVERO introduces the AARO benchmark and negative-text augmentation with a negative-augmented matching loss to improve video-language compositional understanding and retrieval.
Related: Efficient ML and AI infrastructure