Sim-to-Real Transfer: Training RL Agents in Simulation for Real-World Robotics

Sim-to-real transfer remains the central bottleneck for deploying reinforcement learning in robotics. You can train a dexterous manipulation policy to 95% success in Isaac Sim or MuJoCo in a few hours, only to watch it fail completely on the physical robot because of a 12ms latency you didn't model or a friction coefficient that was off by 0.15. In my experience, the gap is rarely about algorithm choice — Deep RL Algorithms: DQN, PPO and SAC Compared matters less here than your physics modeling — and almost always about how you structure randomization, identification, and the hardware interface itself. This article distills what I've learned moving RL policies from simulation to real Franka, UR5, and quadruped platforms, focusing on the engineering decisions that actually determine transfer success.

The Reality Gap: Why Contact Dynamics Break Naive Simulation Training

Most RL practitioners coming from locomotion or Atari backgrounds underestimate how differently contact dynamics behave in simulation versus reality. Free-space motion — a robotic arm moving without payload — transfers trivially. The moment you introduce contact, whether it's a parallel-jaw gripper closing on a deformable object or a quadruped foot striking uneven terrain, the reality gap widens by an order of magnitude.

Simulators like MuJoCo, Isaac Gym, and PyBullet all solve constrained rigid-body dynamics, but they make different compromises on contact solvers. MuJoCo uses a soft contact model with convex optimization that is stable but introduces interpenetration artifacts. Isaac Sim's PhysX backend prioritizes speed for massively parallel rollouts, sacrificing contact accuracy at high contact counts. In my experience, policies trained with excessively stiff contact settings learn to exploit solver errors — they discover that they can "vibrate" an object into a stable grasp that would be physically impossible due to micro-slips the solver averages away.

Identifying Exploitable Physics Artifacts

I've found that the best diagnostic is to log contact forces and penetration depths during training. If you see impulsive forces exceeding 5-10x the robot's payload capacity, or sustained penetrations above 2-3mm, your reward function is almost certainly rewarding solver exploitation. Add a penalty term for excessive contact force and penetration depth early. It slows initial learning but prevents a class of failures that domain randomization alone cannot fix.

Another subtle failure mode is actuator modeling. Most Reinforcement Learning Fundamentals: MDPs, Policies and Value Functions tutorials model actions as direct torque or position commands. Real motors have current loops, gear backlash, temperature-dependent friction, and communication delays. A policy that assumes instantaneous torque execution will oscillate on hardware. You need to model the low-level controller explicitly in simulation, even if it's just a PD controller with delay.

Friction and Compliance Are Not Scalars

Beginners randomize a single friction coefficient. Real surfaces have anisotropic friction — sliding a box along its long edge versus its short edge produces different resistance. Patch contact also matters: a fingertip with a soft silicone pad has a contact patch, not a point, which creates torsional friction that point-contact models ignore. If you are doing in-hand manipulation or pushing, model the contact patch compliance. In MuJoCo, this means tuning solimp and solref parameters rather than just friction.

Architecting Domain Randomization Pipelines That Generalize

Domain randomization (DR) is the most reliable tool for sim-to-real, but its implementation dictates whether you get robust generalization or a policy that masters nothing. The naive approach — uniformly randomizing every physical parameter over a wide range — creates an impossible learning problem. The policy becomes overly conservative, or worse, fails to converge because the MDP is no longer stationary in a useful way.

I've found that staged randomization works far better. Start training with narrow randomization, then widen distributions as performance plateaus. This is similar to curriculum learning, but automated based on success rate thresholds. Visual DR and dynamics DR should also be decoupled. Randomizing camera textures at the same time you randomize mass and damping forces the policy to solve perception and control robustness simultaneously, which dramatically increases sample complexity.

# Staged domain randomization with automatic curriculum
import numpy as np

class StagedDomainRandomization:
    def __init__(self, env):
        self.env = env
        self.stage = 0
        self.randomization_ranges = {
            0: {"mass": (0.95, 1.05), "friction": (0.7, 0.9), "latency_ms": (0, 2)},
            1: {"mass": (0.8, 1.2), "friction": (0.5, 1.1), "latency_ms": (0, 8)},
            2: {"mass": (0.6, 1.4), "friction": (0.3, 1.3), "latency_ms": (0, 20),
                "joint_damping": (0.8, 1.2), "actuator_gain": (0.9, 1.1)}
        }
        self.success_threshold = 0.85
        self.eval_window = 100

    def sample_params(self):
        ranges = self.randomization_ranges[self.stage]
        return {k: np.random.uniform(v[0], v[1]) for k, v in ranges.items()}

    def maybe_advance_stage(self, recent_success_rate):
        if recent_success_rate > self.success_threshold and self.stage < 2:
            self.stage += 1
            print(f"Advancing to DR stage {self.stage}: {self.randomization_ranges[self.stage]}")

Visual Randomization Without Destroying Sample Efficiency

For vision-based policies, randomize in order of importance: lighting first, then textures, then camera pose. I've seen teams randomize 100 textures on day one and then wonder why training takes 3x longer. Start with 4-6 plausible textures and strong lighting variation. The policy needs to learn shape and spatial reasoning before it can handle being texture-invariant. If you use depth images, add correlated per-pixel noise and dropout that mimics real stereo or structured-light artifacts — uncorrelated Gaussian noise is not representative and teaches the wrong invariances. For implementation details on wrapping these observation perturbations correctly, see Designing Custom Gymnasium Environments for RL Training.

Dynamics Randomization: What to Randomize and What to Fix

Randomize parameters your policy can actually observe or compensate for. Mass, center-of-mass offset, and joint friction are good candidates because their effects are observable through proprioceptive history. Do not randomize gravity or timestep — they break physical consistency without teaching useful robustness. In my recent quadruped work, randomizing PD gains by ±15% and adding 0-15ms actuation delay covered more real-world variance than randomizing link masses by ±30%.

System Identification Versus Domain Randomization: Selecting the Transfer Strategy

There are two fundamentally different philosophies: identify the real world accurately and train in that accurate simulator (system identification), or train across such a wide distribution that the real world is just another sample (domain randomization). In practice, you need a hybrid, but choosing the right balance saves weeks.

System identification shines when you have a small number of dominant parameters and can measure them well. For a 7-DoF arm with known kinematics, identifying joint friction, armature inertia, and actuator delay from a few minutes of real trajectory data yields a simulator that transfers with minimal randomization. Domain randomization shines when parameters are numerous, hard to measure, or vary per episode — like object mass, friction, and deformation for bin picking.

My default approach now is hybrid: run system identification to find the mean of each parameter distribution, then add randomization around that mean with width proportional to identification uncertainty. For example, if you identify joint damping as 0.42 ± 0.05, randomize in [0.32, 0.52] rather than a blind [0.1, 1.0]. This keeps training tractable while covering real variation. The arXiv Machine Learning literature on adaptive domain randomization formalizes this as learning the distribution parameters themselves, which is worth reading if you plan to automate tuning.

Adapting Proprioceptive and Tactile Observations for Contact-Rich Transfer

Policies that rely solely on vision and joint positions often fail on contact-rich tasks because contact is fundamentally partially observable. A vision system cannot tell you if a grasp has 2N or 10N of normal force. If you train with access to privileged simulator state — exact contact forces, object pose, friction coefficients — you must handle the privileged information gap carefully. The teacher-student distillation approach is the current standard.

Train a teacher policy with full privileged state, then distill it into a student that only sees deployable observations (history of proprioception, noisy depth, tactile signals if available). I've found that adding a short history window (0.3-0.5 seconds at 30-60 Hz) of joint positions, velocities, and actions is surprisingly effective as a substitute for explicit force sensing. The policy learns to infer contact events from the mismatch between commanded and achieved positions.

# Teacher-student with privileged information and history
import torch
import torch.nn as nn

class PrivilegedTeacher(nn.Module):
    def __init__(self, obs_dim, priv_dim, action_dim):
        super().__init__()
        # Teacher sees true object pose, contact forces, friction
        self.net = nn.Sequential(
            nn.Linear(obs_dim + priv_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, action_dim)
        )
    
    def forward(self, obs, priv_info):
        return self.net(torch.cat([obs, priv_info], dim=-1))

class HistoryStudent(nn.Module):
    def __init__(self, obs_dim, history_len=15, action_dim=7):
        super().__init__()
        # Student sees only deployable history: q, qd, last actions
        self.temporal_encoder = nn.GRU(obs_dim, 128, batch_first=True)
        self.mlp = nn.Sequential(
            nn.Linear(128, 256), nn.ReLU(),
            nn.Linear(256, action_dim)
        )
    
    def forward(self, obs_history):
        # obs_history: [B, T, obs_dim]
        _, h_n = self.temporal_encoder(obs_history)
        return self.mlp(h_n.squeeze(0))

# Distillation loss: student mimics teacher actions without privilege
# See PyTorch Documentation for GRU and distillation utilities

When to Add Real Tactile Sensors

If your task requires force regulation below ~5N or slip detection, do not try to infer it from proprioception alone. Low-cost tactile sensors like DIGIT or GelSight simulation in TACTO or Taxim can be integrated into the simulator, but calibrate the simulated tactile response. In my experience, sim tactile images are too clean — add marker displacement noise and lighting variation in the elastomer simulation, or the student will overfit to perfect tactile edges that don't exist on hardware.

Latent Deployment Killers: Latency, Actuator Dynamics, and Middleware Jitter

This is the section most papers omit and most real deployments fail on. Your policy runs at 30 Hz in simulation with zero-latency, perfectly synchronous observations and actions. On hardware, you have image capture latency (15-30ms), inference latency (5-15ms), ROS or middleware jitter (2-10ms variance), and actuator communication delay. The total loop delay is often 35-60ms, which is 1-2 full control steps at 30 Hz.

A policy trained without latency randomization will oscillate or go unstable on hardware, even if it was robust to dynamics variation. The fix is to model observation and action delays explicitly. At training time, buffer observations and delay actions by a randomized number of steps sampled from your measured hardware distribution. The PyTorch Documentation profiling tools are useful here for measuring your actual inference latency on the deployment device, not just on your training workstation — I've seen 8ms on an A100 become 22ms on a Jetson Orin.

# Latency-aware environment wrapper for sim-to-real
from collections import deque
import random

class LatencyWrapper:
    def __init__(self, env, obs_delay_range=(1, 2), act_delay_range=(1, 2)):
        self.env = env
        self.obs_buffer = deque(maxlen=5)
        self.act_buffer = deque(maxlen=5)
        self.obs_delay_range = obs_delay_range
        self.act_delay_range = act_delay_range

    def step(self, action):
        # Randomize per-episode latency to cover hardware jitter
        obs_delay = random.randint(*self.obs_delay_range)
        act_delay = random.randint(*self.act_delay_range)

        self.act_buffer.append(action)
        if len(self.act_buffer) > act_delay:
            delayed_action = self.act_buffer[-act_delay - 1]
        else:
            delayed_action = action

        obs, reward, done, info = self.env.step(delayed_action)

        self.obs_buffer.append(obs)
        if len(self.obs_buffer) > obs_delay:
            delayed_obs = self.obs_buffer[-obs_delay - 1]
        else:
            delayed_obs = obs

        return delayed_obs, reward, done, info

Action Filtering and Rate Limiting

Real actuators cannot track high-frequency action changes. A policy that outputs bang-bang or chattering actions will burn out motors and excite unmodeled resonances. Apply an exponential moving average or low-pass filter to actions in simulation that matches your low-level controller bandwidth. I typically use an EMA with alpha=0.2-0.3 for 30 Hz policies on Franka arms. Also enforce rate limits — if your joint velocity limit is 1.5 rad/s, your action delta per step cannot exceed that divided by the control frequency.

Evaluation Protocols That Predict Hardware Success Before Touching the Robot

You cannot iterate on hardware quickly. Each real evaluation costs minutes and risks hardware damage. You need simulation-based metrics that actually correlate with transfer.

First, measure performance across the full randomization distribution, not just the mean. Report success rate at the 5th percentile of your randomization range, not the average. A policy with 96% mean success but 45% success at the distribution tails will fail on hardware when it encounters that tail configuration.

Second, run adversarial perturbations: add 10% randomization width beyond your training distribution and see where performance collapses. The collapse point tells you which parameter the policy is most brittle to. In my runs, friction and latency are usually the first to cause collapse, which guides where to invest in better identification.

The Sim-to-Sim Proxy Test

Before sim-to-real, do sim-to-sim: train in MuJoCo and evaluate in Isaac Sim (or vice versa) with identical randomization. If performance drops >15% across simulators, you are exploiting simulator-specific artifacts. Fix the reward and randomization before going to hardware. I've caught multiple cases where a grasping policy transferred between simulators at only 62% despite 91% in the training simulator — every one of those cases also failed on hardware until we added contact force penalties and history-based observations.

Logging Real Trajectories for Iterative Refinement

Once you do deploy, log everything: commanded actions, achieved joint states, end-effector poses from motion capture or kinematics, and success/failure labels with video. Compare the distribution of achieved states on hardware to the simulated distribution. If real trajectories consistently lie outside your simulated state visitation, your randomization is too narrow in that dimension. I maintain a simple script that overlays real and sim joint trajectories after each deployment batch and flags divergence — it has saved more hardware time than any algorithmic improvement.

Frequently Asked Questions