Location: Mech. Eng., CMU, PA (16-831 Introduction to Robot Learning)
Timeframe: Spring 2026
Introduction
This project investigates reinforcement learning for contact-rich tabletop manipulation: can current algorithm classes learn to grasp and lift an object through trial-and-error interaction alone? We study the PickCube task, in which a robot must reach a cube on a table, grasp it with a parallel-jaw gripper, and lift it to a goal position above its starting location. Despite its apparent simplicity, the task captures the core challenges of prehensile manipulation: the agent must coordinate end-effector positioning with gripper closure timing, maintain a stable grasp during lifting, and generalize across randomized cube poses.
The Ridgeback UR10e in the ManiSkill3 PickCube environment.
Our experiments run in the ManiSkill3 GPU-parallelized simulator on the Ridgeback UR10e, a mobile manipulator with a UR10e arm and parallel-jaw gripper (base held static). Observations are expressed in the end-effector’s local frame, making the policy invariant to base pose. Every method trains under identical conditions: the same control mode, 64 vectorized environments, and a 300k environment-step budget, so comparisons isolate the algorithm rather than the compute.
We benchmark four algorithm classes on the task (a random agent, on-policy PPO, off-policy SAC, and model-based MBPO) and propose two modifications targeting the benchmark’s weakest points.
Design & Development
The baseline dense reward decomposes the task into four gated stages: reaching, a binary grasp bonus, lifting gated on grasp, and a static bonus at the goal. Because the lift and goal rewards are zero until the engine’s binary grasp flag fires, there is no continuous gradient between “not grasped” and “grasped.” This sparse barrier between reaching and grasping shaped both of our modifications.
Modification 1 replaces MBPO with TD-MPC2 to compare two fundamentally different model-based RL paradigms. MBPO trains an ensemble of dynamics models and generates short imagined rollouts to augment a model-free learner, a data-augmentation paradigm. TD-MPC2 instead learns a latent-space world model and performs explicit Model Predictive Control at every action-selection step, a trajectory-optimization paradigm. Running both under an otherwise identical setup reduces the comparison to algorithm class alone.
Modification 2 redesigns the reward to provide continuous shaping through the grasp transition, applied to the strongest learner from Modification 1. Three changes are central: a continuous gripper-closure signal amplified near the cube, a proxy-grasp indicator built from position and finger closure that does not depend on the engine’s binary grasp flag, and explicit lift and height components that shape upward motion directly.
Evaluation
The learning curves sort the methods into clear tiers. SAC achieves the highest return, plateauing around 8.0 by 150k steps through aggressive reuse of off-policy data. The two TD-MPC2 variants form a stable second tier near 5.5 to 5.8. PPO learns the same reaching behavior roughly an order of magnitude more slowly, reaching about 2.5 by 300k steps. MBPO collapses entirely: its dynamics ensemble, fit on a low-reward replay buffer, predicts near-zero reward almost everywhere, and the synthetic rollouts drive the actor to a degenerate policy that drifts away from the table. The sharpest result is this within-paradigm contrast: under identical conditions, trajectory-optimization MBRL learns stably where data-augmentation MBRL degenerates.
Mean training return versus environment steps for all six configurations on PickCube.
Return, however, is not success. Evaluation success is approximately zero for every method; only SAC ever crosses the threshold, transiently peaking near 6% before falling back. Decomposing the returns shows each method is earning reward through the dense reaching component rather than task completion, so the return ranking measures how well each method exploits reaching, not grasping.
Evaluation success rate versus training steps: near zero for every configuration.
Video inspection separates the failure modes. SAC reaches fastest but its closure timing imparts lateral momentum that pushes the cube out of the grasp. TD-MPC2 with the original reward attempts stationary grasps whose force closure is unstable. TD-MPC2 with the redesigned reward changes behavior in a counterproductive direction: partial credit for closing the fingers near the cube teaches it to close early during approach, shoving the cube sideways. PPO never completes the alignment that precedes grasping. That three different algorithm classes and two reward designs all converge to zero success isolates the gripper’s low-level controller as the residual bottleneck: the Ridgeback UR10e is not the platform these benchmarks were tuned for, and its contact dynamics cannot reliably secure a 4 cm cube.
Reaching without Grasping
SAC: lateral push at closure
TD-MPC2: unstable force closure
Mod 2: premature closure
Failing before the Grasp stage
PPO: never centers the cube
MBPO: degenerate drift-away policy
Conclusion
Three takeaways summarize the study. First, algorithm class matters within model-based RL but is not the dominant bottleneck: switching from MBPO to TD-MPC2 turned a collapsing learner into a stable one, yet still did not solve the task. Second, reward shaping changes behavior without necessarily improving it; our continuous closure shaping redirected the policy into premature gripping. Third, converging failures across every configuration point below the policy: the gripper’s low-level controller cannot convert closure commands into stable grasps on this embodiment.
These conclusions define the follow-ups: tuning or learning a low-level grasp controller for the UR10e gripper, demonstration-augmented model-based RL and potential-based reward shaping to bridge the sparse grasp barrier in a principled way, and adding a latent-imagination world model (DreamerV3) to complete the three-paradigm model-based comparison.
