MuJoCo¶
MuJoCo camera -> CameraTwin frames.
MujocoCameraTwin(twin: CameraTwin, backend: RenderBackend, camera: str | None = None, match_fov: bool = True, resolution: tuple[int, int] | None = None, device: str | device | None = None, rectify: bool = False, render: str = 'psf', rays_per_pixel: int = 8, pupil_views: int = 7, shading: str = 'corrected', view_oversample: float = 1.0)
A CameraTwin attached to a MuJoCo camera.
The camera keeps its pose. While rendering, its vertical FoV is set to the
twin's (match_fov=True) and the image is rendered at the twin's
resolution. The model is restored afterwards. For a twin with lens
distortion this is the twin's wider source pinhole (CameraTwin.render_intrinsics),
which the twin warps onto its sensor. With match_fov=False the scene
camera's own pinhole geometry is kept and only blur is applied.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
twin
|
CameraTwin
|
The CameraTwin ( |
required |
backend
|
RenderBackend
|
|
required |
camera
|
str | None
|
MuJoCo camera name (or call |
None
|
match_fov
|
bool
|
Render with the twin's FoV. |
True
|
resolution
|
tuple[int, int] | None
|
|
None
|
device
|
str | device | None
|
Torch device for the CameraTwin pipeline. |
None
|
rectify
|
bool
|
Undistort the output like vendor SDKs (see |
False
|
render
|
str
|
|
'psf'
|
rays_per_pixel
|
int
|
Rays traced per pixel and wavelength (ray cast). |
8
|
pupil_views
|
int
|
Shading views rendered across the lens' entrance pupil (ray cast). |
7
|
shading
|
str
|
|
'corrected'
|
view_oversample
|
float
|
Resolution of the pupil views relative to the sensor; above 1 resolves detail thinner than a pixel (ray cast). |
1.0
|
Ideal render at the twin's resolution and FoV: (rgb uint8, depth m), upright.
get_frame(timestamp: float | None = None, status=None, force_depth: bool = False) -> CameraFrame
Render and process one frame. frame.rgb is linear [1, 3, H, W] on device.
frame.depth is readable only if the camera's depth output is simulated
(outputs.depth in its spec) or with force_depth=True (simulator ground truth).
mujoco.Renderer backend for plain MuJoCo (MjModel/MjData) scenes.
Do not use it inside robosuite/LIBERO: a second GL context in the same
process corrupts robosuite's own offscreen renders (use RobosuiteRenderer).
robosuite backend: renders in robosuite's own GL context.
Pass the robosuite env (or anything with .sim) rather than the sim:
robosuite may rebuild env.sim on reset, so it is looked up on every call.
property
¶
robosuite's render option (which geom groups it draws); None before first render.
Bases: Protocol
Renders one MuJoCo camera. Returns images upright (row 0 = top).
render returns (rgb uint8 [H, W, 3], depth float32 [H, W]), with depth
as metric z-depth in meters.
Encode a linear [1, 3, H, W] frame for consumers: sRGB, uint8 [h, w, 3], upright.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
size
|
tuple[int, int] | None
|
Output |
None
|
crop
|
str
|
|
'center'
|
Custom camera mounts: a camera attached to any body (robot link, gripper, ...) or the world.
A compiled MuJoCo model cannot gain cameras, and rebuilding a LIBERO scene per
edit takes seconds. So a mount is rendered by temporarily re-pointing an
existing host camera: its parent body, pose and FoV are replaced, the camera
is rendered, and everything is restored (mounted). Link-mounted cameras
therefore follow the robot through a replay.
Orientation: yaw, pitch and roll in degrees, in the parent body's frame. At zero the camera looks along the body's +x with +z up. Yaw turns about the body's z, pitch tilts down (positive = look down), and roll turns about the viewing axis.
dataclass
¶
Unit quaternion (w, x, y, z) of rotation matrix R (Shepperd's method), w >= 0.
Bodies a camera can be attached to: the world, then robot, gripper, mount and mobile base links.
mounted(model, data, mount: CameraMount, host: str = HOST_CAMERA)
Temporarily turn camera host into mount; yields the host camera's name.
Inverse of CameraMount.rotation: body-frame camera axes -> (yaw, pitch, roll) degrees.
mount_world_pose(model, data, mount: CameraMount) -> tuple[ndarray, ndarray]
World position and camera axes (columns x, y, z; MuJoCo convention) of mount.
reattach(model, data, mount: CameraMount, body: str) -> CameraMount
mount re-parented to body with its world pose unchanged.
World camera axes (MuJoCo: looks along -z, +y up) looking along forward, no roll.
snap_to_body(model, data, mount: CameraMount, body: str, target: ndarray, standoff_m: float = 0.015) -> CameraMount
Put mount on the surface of body, aimed at target (world).
Candidate spots all around the link are tried; the camera goes where its view
of target (e.g. the gripper's workspace) is least blocked by the robot,
preferring the side of the link facing where the camera was. It sits
standoff_m out from the surface, upright. world keeps the current
world pose.
mount_module(mount: CameraMount, modules=None)
The stereo module of a mount, at the mount's baseline (baseline_mm) if it sets one.
Distance (m) between two cameras of a compiled model, in their parent body's frame.
Move a compiled stereo pair's eyes to baseline_m apart; returns the old baseline.
The eyes move symmetrically about their midpoint, along the line between them,
so the module stays centered where the robot mounts it. This adjusts a stereo pair
built into a robot model (e.g. the multi-camera arms' wrist module) without
rebuilding the model; call mj_forward (or sim.forward()) before rendering.
Both cameras must share a parent body.
expand_mount(mount: CameraMount, modules=None) -> dict[str, CameraMount]
The cameras a mount renders: itself, or a stereo module's two eyes.
A stereo mount's pose is its left eye's; the eyes are <name>_L / <name>_R
on the same parent body, so they move together.
Lens-ray rendering of MuJoCo scenes (see twinrobo.optics.lensrender).
The ray-cast method ("raycast") is simulator independent; this module is its
MuJoCo scene: the shading views rasterized by MuJoCo from the camera shifted across the
pupil, the camera pose from data, and the drawn geoms as ray-cast triangles.
A MuJoCo camera as a lens scene (twinrobo.optics.lensrender).
LensRayRenderer(rays: LensRays, method: str = 'raycast', views: int = 7, shading: str = 'corrected', chunk: int = 4000000, oversample: float = 1.0)
Bases: LensRayRenderer
twinrobo.optics.lensrender.LensRayRenderer with a MuJoCo render shortcut.
(rgb [1, 3, H, W] linear, depth [1, 1, H, W] m) of the current MuJoCo state.
Local-frame triangles of geom g (as MuJoCo draws it), or None if unsupported.