Projects
Selected work on post-training code models and agents, spanning training data, model training, and reinforcement learning.
RL for a Code-Understanding Subagent
Ongoing · RL environments · Reward design
I’m leading a proof of concept on training a code-understanding subagent with RL. I own the benchmark, reward, model, and algorithm choices, and design and run the experiments.
Mid-Training on Agent Trajectories
Megatron-LM · SFT · Next-token prediction · Distillation
I built a training pipeline for turn-level model routing in code agents: collecting agent trajectories, designing privileged-information distillation data, and training 4B dense models with SFT and next-token prediction in Megatron-LM. I revived the SFT setup and halved training time through TP/FSDP layout and batch-size tuning.
The routed models did not outperform the baselines. My analysis pointed to insufficient turn-level routing signal in the agent trajectories.
RL in Interactive Environments
veRL · SDPO · GRPO · τ-bench
In a one-week team hackathon, I integrated τ-bench into our veRL-based RL infrastructure and designed the feedback signal for SDPO’s self-distillation pass. We compared SDPO and GRPO on an 8B dense model; the project reached a 13 percentage point improvement over the base model on held-out tasks.
RouteLLM Reproduction
XLM-RoBERTa · Hugging Face Trainer · Class rebalancing · Model routing
I reproduced the BERT-based router from RouteLLM and found majority-class collapse in the released checkpoints (macro F1 of 0.23–0.35). I addressed this with class rebalancing and built a reproducible training and evaluation pipeline using Hugging Face Trainer and XLM-RoBERTa-base.
The rebalanced router improved on the released checkpoints in several benchmark comparisons, using about 7× fewer training examples (19k versus 130k+). Evaluations covered MMLU, GSM8K, and MT-Bench.