AGPL-3.0 · commercialMuJoCo · Isaac Sim

See through the camera you will deploy¶

Simulators render through a perfect pinhole; robots see through real lenses. TwinRobo renders simulated scenes through the actual optics of off-the-shelf cameras, so perception trained in simulation meets the same image on the robot. See why TwinRobo?

Try it live Get started Browse cameras

A robot's wrist camera over a bin of groceries, through a real camera's lens (ray-traced)

The same view through the simulator's pinhole

Pinhole Real camera

Drag to compare: a robot's wrist camera in MuJoCo, the same frame, pose and field of view, through the simulator's pinhole and through a real camera's lens.

Change the camera, and a robot policy drops from 85 % to 0 %¶

OpenVLA solves 85 % of its LIBERO tasks through the pinhole camera it was trained with, and none through a real camera placed where that pinhole was. TwinRobo shows you that before your robot does. See the benchmark

OpenVLA on the same LIBERO task from the same start, seen through its own (third-person) camera: through the simulator's pinhole it succeeds (left); through a real camera at the same place it fails (right).

One environment, any real camera¶

Real lens optics

Lenses traced with DeepLens: depth- and field-dependent blur, the lens' own distortion, chromatic aberration and vignetting, from a prescription, not a hand-tuned filter.

Optics and rendering
Two rendering methods

A fast 2.5D PSF renderer, plus a ray caster that traces every pixel's rays through the lens into the scene, exact even around defocused foreground objects.

Rendering methods
Drop-in for simulators

MuJoCo, LIBERO, RoboCasa and NVIDIA Isaac Sim. Wrap a camera and policies, eval scripts and recorded demos run unchanged, now through a real camera.

Simulators
An open camera catalog

RealSense, ZED, OAK-D, a C920 webcam and the Raspberry Pi camera, as single cameras or rectified stereo pairs. Every camera works in every simulator and method, and a real unit can be calibrated.

Camera catalog

Support at a glance¶

	MuJoCo (incl. LIBERO, RoboCasa)	NVIDIA Isaac Sim 6.x
Adapter	`MujocoCameraTwin`	`IsaacCameraTwin`
`psf`: pinhole + lens blur and distortion
`raycast`: lens rays cast into the scene
RGB on the GPU, ground-truth depth on request
Cameras on robot links, stereo modules		any USD camera prim
Setup	`pip install`	Docker recipe included

Five lines to a real camera¶

MuJoCoIsaac SimLIBEROAny RGB-D

from twinrobo import CameraTwin
from twinrobo.plugins.mujoco import MujocoCameraTwin, MujocoRenderer

twin = CameraTwin.from_catalog("stereolabs/zed-x/2.2mm")
cam = MujocoCameraTwin(twin, MujocoRenderer(model, data), camera="wrist")
frame = cam.get_frame()          # frame.rgb: the camera's image, on the GPU

from twinrobo import CameraTwin
from twinrobo.plugins.isaac import IsaacCameraTwin

twin = CameraTwin.from_catalog("stereolabs/zed-x/2.2mm", build_psf=False)
cam = IsaacCameraTwin(twin, "/World/Camera", render="raycast")
frame = cam.get_frame()          # every pixel ray-traced through the lens

from twinrobo import CameraTwin
from twinrobo.datasets.libero import CameraTwinLiberoEnv, make_env

env, task, init_states = make_env("libero_spatial", 0, resolution=128)
env = CameraTwinLiberoEnv(env, {"agentview": CameraTwin.from_catalog("stereolabs/zed-x/4mm")})
obs = env.reset()                # obs["agentview_image"] now comes from the twin

from twinrobo import CameraTwin

twin = CameraTwin.from_catalog("stereolabs/zed-x/2.2mm")
frame = twin.process(rgb, depth) # in: linear RGB [B,3,H,W], metric depth [B,1,H,W]

Validated in both simulators¶

46 dB

PSF renderer vs DeepLens' reference renderer at 1920×1200.
< 0.1 px

Points land where the lens' chief rays point (Isaac Sim).
~0.03 vs ~0.13

Error on a defocused foreground edge: ray cast vs pinhole + PSF (Isaac Sim; MuJoCo ~0.05).

Your camera is missing? Add it¶

The catalog is the heart of TwinRobo, and it is open. A camera is a camera.yaml and a lens file; once merged it works in every simulator, rendering method and dataset replay, for everyone.

How to add a camera Request a camera

Camera makers: become a partner¶

Robot teams now choose and validate cameras in simulation. Ship manufacturer_verified models of your products, get listed as a partner and showcased, or sponsor the open environment your customers use.

The partner program