Core

CameraTwin

CameraTwin(spec: CameraSpec | None = None, optics: OpticsModel | None = None, sensor: SensorModel | None = None, isp: ISPModel | None = None, intrinsics: CameraIntrinsics | None = None, distortion: RadialDistortion | None = None, reference: OpticsModel | None = None)

A deployed camera configuration: optics, sensor and ISP.

outputs_depth property

outputs_depth: bool

Whether the camera's depth output is simulated (outputs.depth).

True without a spec (nothing says otherwise).

render_intrinsics property

render_intrinsics: CameraIntrinsics | None

The pinhole a simulator should render for this twin.

Without distortion these are the camera intrinsics. With distortion it is a wider, centered pinhole with the same central focal length, which process warps onto the distorted sensor.

from_spec classmethod

from_spec(path: str | Path | CameraSpec, device: str | device | None = None, cache: PSFCache | None = None, build_psf: bool = True) -> CameraTwin

Load a spec; builds and caches its PSF bank on first use (slow), then reuses it.

build_psf=False skips the PSF bank (identity optics), for twins rendered only with the lens-ray methods (twinrobo.plugins.mujoco.lensrender).

from_catalog classmethod

from_catalog(camera_id: str, registry: CatalogRegistry | None = None, device: str | device | None = None, cache: PSFCache | None = None, build_psf: bool = True) -> CameraTwin

A catalog camera (vendor/model/...). build_psf=False skips the PSF bank, which only the psf rendering method needs (the lens-ray methods trace the lens).

sensor_color

sensor_color(rgb: Tensor) -> Tensor

The light a sensor of this camera records: luminance for mono sensors.

A mono sensor has one intensity channel; it is returned in all three ([B, 3, H, W], equal channels) so frames keep the RGB convention. Color sensors are unchanged.

lens_geometry

lens_geometry() -> dict[str, Any]

Keyword arguments of lens_rays giving the lens-ray methods this camera's geometry.

The spec's intrinsics and distortion (lens.geometry: calibration, the default), so every rendering method has the catalog's field of view and distortion. Empty, keeping the traced lens' own geometry, for lens.geometry: lens, for calibration.distortion.model: physical and when the spec gives no intrinsics.

process

process(rgb: Tensor, depth: Tensor, timestamp: float | None = None, exposure: float = 1.0, metadata: dict[str, Any] | None = None, rectify: bool = False, force_depth: bool = False) -> CameraFrame

Turn an ideal simulator RGB-D observation into a CameraTwin frame.

With distortion, an observation rendered at render_intrinsics is warped onto the sensor first; frame.rgb_ideal is then the undistorted pinhole view with the camera's intrinsics and frame.depth the depth seen through the lens. Observations of any other size skip the warp (blur only).

rectify: undistort the output (as vendor SDKs do by default): frame.rgb and frame.depth are then pinhole images with the camera's intrinsics, the lens blur kept. No effect without distortion.

force_depth: expose frame.depth even if the camera outputs none (outputs_depth false): the simulator's ground truth, which the real camera would not give. Without it, reading such a frame's depth raises DepthUnavailableError. depth is still required: the lens blur uses it.

CameraSpec dataclass

CameraSpec(schema_version: str, id: str, manufacturer: str | None, product: str | None, sensor: SensorSpec, lens: LensSpec, calibration: dict[str, Any] = dict(), sensor_model: dict[str, Any] = dict(), isp: dict[str, Any] = dict(), outputs: OutputsSpec = OutputsSpec(), validation: ValidationSpec = ValidationSpec(), source_path: Path | None = None)

from_yaml classmethod

from_yaml(path: str | Path, **overrides: Any) -> CameraSpec

Load a spec file; keyword overrides go to apply_overrides (width=, ...).

apply_overrides

apply_overrides(data: dict[str, Any], *, width: int | None = None, height: int | None = None, focus_distance_m: float | None = None, near_m: float | None = None, far_m: float | None = None, depth_samples: int | None = None, field_samples: tuple[int, int] | None = None) -> dict[str, Any]

A copy of raw spec data with common changes applied: the same camera, re-configured.

• width / height: the same sensor read out at another resolution. Explicit intrinsics and the pixel pitch scale with it (pixel-edge convention), so the field of view is unchanged.
• focus_distance_m: refocus the lens.
• near_m, far_m, depth_samples, field_samples (w, h): the PSF bank's depth range and sampling (calibration.psf).

CatalogRegistry

CatalogRegistry(roots: list[str | Path] | None = None, use_env: bool = True, use_builtin: bool = True)

Resolve catalog IDs to CameraSpec files on the local filesystem.

list

list() -> list[str]

All camera IDs available across roots (first root wins on duplicates).

frame

CameraFrame and TwinRobo tensor conventions.

Tensor conventions (the single source of truth for all TwinRobo modules):

• rgb: torch.Tensor [B, 3, H, W], float, linear radiance-like values nominally in [0, 1] (not sRGB-encoded). Channel order R, G, B.
• depth: torch.Tensor [B, 1, H, W], float, metric depth in meters, positive, measured along the optical axis (z-depth, not ray distance). Values that are non-finite (inf/nan, e.g. simulator background) or <= 0 are invalid; consumers must mask or clamp them explicitly. Only cameras whose depth output is simulated (outputs.depth in their spec) expose it; none of the catalog does yet (see ROADMAP.md). Reading frame.depth of any other camera raises DepthUnavailableError, unless the frame was made with force_depth=True: the simulator's ground truth (labels, evaluation), not a depth sensor's reading.
• Image coordinates: row 0 is the top of the image, column 0 is the left.
• Device: tensors stay on the producing device (normally CUDA). TwinRobo never moves per-frame data to the CPU unless the caller explicitly asks for it.

Backend-specific units (e.g. DeepLens uses millimeters and negative object z) are converted only inside the corresponding adapter (twinrobo.optics.deeplens).

CameraFrame dataclass

CameraFrame(rgb: Tensor, rgb_ideal: Tensor | None = None, rgb_optical: Tensor | None = None, raw: Tensor | None = None, depth: Tensor | None = None, timestamp: float | None = None, metadata: dict[str, Any] | None = None, has_depth: bool = True)

One simulated camera observation.

depth is readable only when has_depth is true: the camera outputs depth, or the frame was made with force_depth=True. Otherwise reading it raises DepthUnavailableError (a real camera of that kind gives no depth).

to

to(device: str | device) -> CameraFrame

Return a copy with all tensors moved to device (explicit transfer only).

check_rgb

check_rgb(rgb: Tensor) -> None

Raise ValueError if rgb does not follow the [B, 3, H, W] convention.

check_depth

check_depth(depth: Tensor, rgb: Tensor | None = None) -> None

Raise ValueError if depth does not follow the [B, 1, H, W] convention.

valid_depth_mask

valid_depth_mask(depth: Tensor) -> Tensor

Boolean mask of valid depth pixels (finite and strictly positive).

CameraIntrinsics dataclass

CameraIntrinsics(width: int, height: int, fx: float, fy: float, cx: float, cy: float)

Pinhole intrinsics in pixels (pixel-index coordinates: pixel (0, 0) center is 0).

For a CameraTwin these are the paraxial intrinsics of the lens at the spec resolution. They are the right pinhole for the simulator camera while distortion is not rendered.

stereo

Stereo camera modules: two eyes in one housing, placed as one unit.

A module file (<catalog root>/<id>/module.yaml) names the per-eye CameraSpecs and the right eye's pose in the left eye's camera frame. The module's pose is the LEFT eye's (Stereolabs convention), so placing a module means placing its left eye; the right eye follows rigidly (on a robot link it follows the link).

Simulator side, a module is expanded into two ordinary cameras: eye_mounts turns a module mount into left/right CameraMounts, which render like any other custom camera. Rectified output (what vendor SDKs deliver) is handled per eye by CameraTwin.process(rectify=True); with identical eye intrinsics and a pure translation the rectified pair is row-aligned.

StereoModule dataclass

StereoModule(id: str, left: Eye, right: Eye, manufacturer: str | None = None, product: str | None = None, housing_mm: tuple[float, float, float] | None = None, output: str = 'rectified', source_path: Path | None = None)

with_baseline

with_baseline(baseline_m: float) -> StereoModule

The same module with another baseline: the right eye moved along the same direction from the left eye (the housing widened or narrowed with it).

Robots differ in how far apart they mount a stereo pair; this lets one catalog module stand for any of them.

Eye dataclass

Eye(spec: Path | None, translation_m: tuple[float, float, float] = (0.0, 0.0, 0.0), rotation_rpy_deg: tuple[float, float, float] = (0.0, 0.0, 0.0))

ModuleRegistry

ModuleRegistry(roots: list[Path] | None = None)

Stereo modules in the catalog roots (same roots as CatalogRegistry).

eye_poses

eye_poses(module: StereoModule, pos, R_left: ndarray) -> dict[str, tuple[ndarray, ndarray]]

{eye: (position, camera axes)} in the parent frame, from the left eye's pose.

Camera axes follow MuJoCo (columns x right, y up, z backwards).

eye_name

eye_name(module_camera: str, eye: str) -> str

exceptions

TwinRobo exception hierarchy.

CameraTwinError

Bases: Exception

Base class for all TwinRobo errors.

SpecError

Bases: CameraTwinError

A CameraSpec is malformed or uses an unsupported schema version.

CatalogError

Bases: CameraTwinError

A camera ID cannot be resolved in the catalog.

OpticsBackendError

Bases: CameraTwinError

An optics backend (e.g. DeepLens) is unavailable or failed.

DepthUnavailableError

Bases: CameraTwinError

A frame's depth was read; its camera has no simulated depth output (outputs.depth).

SimulatorError

Bases: CameraTwinError

A simulator adapter (e.g. Isaac Sim) is unavailable or failed.