Optics

base

Optics backend interface.

OpticsModel

Maps an ideal scene observation to the optical image on the sensor.

rgb / depth follow the conventions in twinrobo.frame ([B, 3, H, W] linear, [B, 1, H, W] meters). Returns [B, 3, H, W].

IdentityOptics

Bases: OpticsModel

Level 0 (ideal pinhole) optics: returns the input unchanged.

DeepLensOptics

DeepLensOptics(model_path: str | Path, sensor_resolution: tuple[int, int] | None = None, focus_distance_m: float | None = None, device: str | device | None = None, dtype: dtype = float32, psf_ks: int = 65)

Bases: OpticsModel

Physical optics backed by a DeepLens geometric lens (GeoLens).

Parameters:

Name	Type	Description	Default
model_path	str | Path	DeepLens lens file (.json, .zmx or .seq).	required
sensor_resolution	tuple[int, int] | None	(width, height) in pixels. DeepLens keeps the sensor diagonal fixed and adapts the sensor size to this aspect ratio. When None, the lens file's sensor is used.	None
focus_distance_m	float | None	Object focus distance in meters (None keeps the focus stored in the lens file; float('inf') focuses at infinity).	None
device	str | device | None	Torch device. Defaults to CUDA when available.	None
dtype	dtype	Torch dtype for the lens.	float32
psf_ks	int	PSF kernel size in pixels. DeepLens centers even-size PSFs between pixels (on-axis centroid at ks/2 - 0.5), while PSF convolution treats index ks // 2 as zero shift. generate_psf therefore traces an even size one larger and drops the last row and column, so every size is centered on ks // 2.	65

sensor_resolution property

sensor_resolution: tuple[int, int]

(width, height) in pixels.

focal_length_mm property

focal_length_mm: float

Paraxial effective focal length computed by DeepLens (not a lens-file header).

f_number property

f_number: float

F-number computed by DeepLens from its paraxial pupil and EFL.

intrinsics

intrinsics() -> CameraIntrinsics

Paraxial pinhole intrinsics at the sensor resolution (no distortion).

set_focus

set_focus(focus_distance_m: float) -> None

Refocus the lens to an object distance in meters (inf = infinity).

generate_psf

generate_psf(field_positions: Tensor | Sequence[Sequence[float]], depths: Tensor | Sequence[float], wavelengths: Sequence[float] | None = None, ks: int | None = None, spp: int | None = None) -> Tensor

Compute PSFs on a field × depth × wavelength grid.

Parameters:

Name	Type	Description	Default
field_positions	Tensor | Sequence[Sequence[float]]	[N, 2] normalized field coordinates (x, y) in [-1, 1] (DeepLens convention, +y up).	required
depths	Tensor | Sequence[float]	[D] object depths in meters (positive; inf is traced at DEFAULT_FAR_M, DeepLens' "approximately infinity").	required
wavelengths	Sequence[float] | None	Wavelengths in micrometers. Defaults to the lens RGB wavelengths, giving an RGB PSF stack.	None
ks	int | None	Kernel size in pixels (defaults to self.psf_ks).	None
spp	int | None	Rays per point source (DeepLens default when None).	None

Returns:

Type	Description
Tensor	[D, N, C, ks, ks] tensor on self.device; each PSF sums to 1.

sample_depths

sample_depths(near_m: float, far_m: float, num_depths: int) -> Tensor

Depth layers in meters, far -> near, uniform in disparity.

When the focus distance is known (focus_distance_m) and lies inside [near_m, far_m], it is an explicit layer, with the remaining layers split in proportion to the disparity range on each side. This avoids interpolating two defocused PSFs where the image should be sharpest. DeepLens' own focal-plane estimate (calc_focal_plane) is not used: it is Monte Carlo and unstable for lenses focused near infinity.

bank_key

bank_key(grid: tuple[int, int], near_m: float, far_m: float, num_depths: int, ks: int | None = None, spp: int | None = None, seed: int = 0) -> dict[str, Any]

Content-based cache key for build_psf_bank (see twinrobo.optics.cache).

lens_rays_key

lens_rays_key(rays_per_pixel: int, seed: int = 0, overfill: float = 1.15) -> dict

Content-based identity of trace_lens_rays output (lens, focus, sensor, sampling).

trace_lens_rays

trace_lens_rays(rays_per_pixel: int = 8, seed: int = 0, overfill: float = 1.15, chunk_rays: int = 4000000, progress=None, candidates: int = 4)

Trace every sensor pixel through the lens into object space (see LensRays).

Each pixel gets rays_per_pixel rays per RGB wavelength: a jittered point inside the pixel aimed at a stratified point of the disk on the exit pupil plane through which that pixel's light passes (_ray_aim; the pupil enlarged by overfill bounds it), so the stop blocks few of them. Rays are traced backward (sensor -> object) with DeepLens' own surface tracer.

candidates: each pixel traces candidates x rays_per_pixel rays and keeps rays_per_pixel that pass the lens, in random order. Strongly vignetted pixels then still get their full ray count (less noise); their brightness comes from the separately traced relative illumination.

build_psf_bank

build_psf_bank(grid: tuple[int, int] = (17, 11), near_m: float = 0.1, far_m: float = inf, num_depths: int = 16, ks: int | None = None, spp: int | None = None, seed: int = 0) -> PSFBank

Precompute the runtime PSFBank on a (Gw, Gh) field grid x depth layers.

DeepLens samples rays at random; seed fixes them, so a bank is the same every time it is built on a device (the caller's random state is left untouched).

render_reference

render_reference(rgb: Tensor, depth: Tensor, psf_grid: tuple[int, int] = (17, 11), near_m: float = 0.3, far_m: float = DEFAULT_FAR_M, num_layers: int = 16, depths_m: Tensor | Sequence[float] | None = None) -> Tensor

High-fidelity offline reference render, built from DeepLens primitives.

The full DeepLens PSF map is computed at every depth layer (psf_map_rgb), then DeepLens' block-wise, depth-interpolated PSF-map convolution is applied (conv_psf_map_depth_interp, disparity interpolation). It has no occlusion handling and uses one PSF per grid block. PSFs are recomputed on every call, so it is far too slow for per-frame simulation. Use it for validation and reference data only. The runtime renderer is twinrobo.optics.dense_depth.DenseDepthRenderer.

Invalid depth (non-finite or <= 0, e.g. simulator background) is replaced by far_m. Depth outside the layer range is clamped.

Parameters:

Name	Type	Description	Default
rgb	Tensor	[B, 3, H, W] linear RGB; (W, H) must equal self.sensor_resolution.	required
depth	Tensor	[B, 1, H, W] depth in meters.	required
psf_grid	tuple[int, int]	(grid_w, grid_h) PSF map blocks.	(17, 11)
near_m	float	Nearest depth layer (m), via sample_depths unless depths_m is given. Use the same values as the PSFBank being compared.	0.3
far_m	float	Farthest depth layer (m).	DEFAULT_FAR_M
num_layers	int	Number of depth layers.	16

Returns:

Type	Description
Tensor	[B, 3, H, W] optically rendered image.

render

render(rgb: Tensor, depth: Tensor, metadata: dict[str, Any] | None = None) -> Tensor

OpticsModel.render: the slow offline reference renderer.

For runtime use, build a PSFBank with build_psf_bank and render with twinrobo.optics.dense_depth.DenseDepthRenderer.

PSFBank dataclass

PSFBank(psfs: Tensor, depths_m: Tensor, sensor_resolution: tuple[int, int], wavelengths_um: list[float] | None = None, metadata: dict[str, Any] = dict())

grid property

grid: tuple[int, int]

(Gw, Gh).

PSFCache

PSFCache(root: str | Path | None = None)

get_or_build

get_or_build(key: dict[str, Any], build: Callable[[], PSFBank], device=None, source: str | Path | None = None) -> PSFBank

Load or build the bank for key; source (a preset file) is recorded as a user.

release

release(source: str | Path) -> list[Path]

Drop source from every bank's users; delete banks nobody uses any more.

Returns the deleted .pt files. Legacy banks (no used_by) are kept.

DenseDepthRenderer

DenseDepthRenderer(bank: PSFBank, eps: float = 0.0001, padding: str = 'reflect')

Bases: OpticsModel

Occlusion-aware, spatially varying, depth-dependent PSF rendering on the GPU.

Parameters:

Name	Type	Description	Default
bank	PSFBank	PSF bank. Its sensor_resolution must equal the image size.	required
eps	float	Normalization threshold. Where the blurred cumulative alpha is below eps, a layer contributes nothing. This avoids amplifying FFT round-off, and it drops at most ~eps of a pixel's energy.	0.0001
padding	str	F.pad mode for image borders ("reflect" or "replicate").	'reflect'

render_differentiable

render_differentiable(rgb: Tensor, depth: Tensor) -> Tensor

render without no_grad: gradients flow to rgb (and to bank.psfs through the blurred layers). The cached unit response H_k(1) is detached, so rebuild the renderer if the bank PSFs change.

distortion

Parametric lens distortion (OpenCV / Brown-Conrady) and its rendering warp.

Geometry and blur are modelled separately. Blur comes from the PSF bank (PSFs recentered on the chief ray, so it carries no geometric shift); geometry comes from the spec's intrinsics plus this distortion model, which is what a calibration (e.g. a vendor factory calibration file) measures.

Rendering: the simulator renders a wider pinhole source image with the same focal length at the center; DistortionWarp resamples it onto the distorted sensor (bilinear for color, nearest for depth, so no false mid-depths appear at edges). The source covers the undistorted field of every sensor pixel.

Coordinates follow OpenCV: normalized x = (u - cx) / fx, y = (v - cy) / fy with pixel (0, 0) at the center of the top-left pixel and +y down.

RadialDistortion dataclass

RadialDistortion(k1: float = 0.0, k2: float = 0.0, p1: float = 0.0, p2: float = 0.0, k3: float = 0.0)

OpenCV distortion (k1, k2, p1, p2, k3): normalized undistorted -> distorted.

from_dict classmethod

from_dict(d: dict[str, Any] | None) -> RadialDistortion | None

From calibration.distortion; None unless model is parametric.

distort

distort(x, y)

Undistorted normalized coordinates -> distorted (numpy arrays or tensors).

undistort

undistort(xd, yd, iters: int = 30)

Distorted -> undistorted by fixed-point iteration (OpenCV's scheme).

DistortionWarp

DistortionWarp(intrinsics: CameraIntrinsics, distortion: RadialDistortion, margin_px: int = 2)

Resample a wide pinhole source render onto a distorted sensor.

Parameters:

Name	Type	Description	Default
intrinsics	CameraIntrinsics	The camera's (sensor) intrinsics.	required
distortion	RadialDistortion	Its distortion model.	required
margin_px	int	Extra source pixels around the needed field (bilinear support).	2

pinhole

pinhole(img: Tensor, mode: str = 'bilinear') -> Tensor

The undistorted pinhole view with the sensor's intrinsics (for comparison).

rectify

rectify(img: Tensor, mode: str = 'bilinear') -> Tensor

Undistort a sensor image onto the pinhole with the sensor's intrinsics.

This is what vendor SDKs deliver as "rectified": the lens blur stays, the geometry becomes pinhole. Pixels whose ray falls outside the sensor (corners, for barrel distortion) repeat the border.

field_of_view

field_of_view(intr: CameraIntrinsics, dist: RadialDistortion | None) -> dict[str, float]

Real horizontal / vertical / diagonal FoV in degrees (through the principal point).

lensrays

Object-space rays of a real lens: sensor pixel -> lens -> scene (simulator independent).

LensRays holds, for every sensor pixel, n rays per wavelength that leave the sensor, pass through the lens, and enter the scene. They are traced once by an optics backend (DeepLensOptics.trace_lens_rays) and then used by the ray-based renderers (twinrobo.plugins.mujoco.lensrender). They depend only on the lens, focus and sensor, never on the scene, so they are cached.

Conventions. The camera frame is MuJoCo's and Isaac's (USD): x right, y up, and the camera looks along -z. A ray is o + s * (tx, ty, -1):

• origin (ox, oy) in meters on the camera's z = 0 plane (the lens front vertex plane; for the bundled lenses the stop is there).
• tan (tx, ty): direction slopes. They are stored as a residual on top of the pixel's paraxial pinhole direction (intrinsics), which keeps fp16 exact to well below a pixel.
• weight: 0 for rays the lens blocks, else cos^4 of the sensor-side angle. A pixel's color is the weight-normalized mean of its rays' radiance times illumination: the pixel's relative illumination (vignetting and cos^4 falloff, 1 on axis), traced densely on a coarse grid. Splitting brightness from blur keeps a few rays per pixel from turning vignetting into noise.

Pixel (u, v) (row 0 = top) is the image a camera outputs, i.e. upright; the optical inversion on the sensor is handled when tracing.

LensRays dataclass

LensRays(intrinsics: CameraIntrinsics, wavelengths_um: list[float], origin: Tensor, tan_res: Tensor, weight: Tensor, chief_tan: Tensor, illumination: Tensor, pupil_radius_m: float, metadata: dict[str, Any] = dict())

pinhole_tan

pinhole_tan(dtype=float32) -> Tensor

Paraxial pinhole slopes of the pixel centers, [H, W, 2].

rays

rays(channel: int, rows: slice = slice(None)) -> tuple[Tensor, Tensor, Tensor]

(origin [h, W, n, 2], tan [h, W, n, 2], weight [h, W, n]) as fp32 for rows.

max_tan

max_tan() -> tuple[float, float]

Largest |tx|, |ty| of any unblocked ray (the field a source view must cover).

with_geometry

with_geometry(intrinsics: CameraIntrinsics, distortion=None) -> LensRays

These rays with a calibrated camera's geometry and the traced lens' blur.

Every ray of a pixel is turned by the same small angle, so that the pixel's chief ray points where the calibration (intrinsics and OpenCV distortion, a RadialDistortion or None) says it does. The spread of the rays around it (blur, lateral color) and the vignetting stay the lens' own. For a surrogate lens this keeps the catalog's geometry instead of the surrogate design's distortion.

rectify_grid

rectify_grid(iterations: int = 6) -> Tensor

grid_sample grid [1, H, W, 2] undistorting a lens image to the paraxial pinhole.

For each rectified pixel (paraxial direction t*), finds the sensor position whose chief ray has direction t* by Newton iteration on the chief-ray map. This is what a stereo SDK's rectification does, computed from the traced lens instead of a fitted distortion model.

concentric_disk

concentric_disk(u: Tensor, v: Tensor) -> tuple[Tensor, Tensor]

Shirley-Chiu map of the unit square to the unit disk (area preserving, low distortion).

stratified_samples

stratified_samples(n: int, count: int, generator: Generator, device) -> tuple[Tensor, Tensor]

count x n jittered stratified samples of the unit square (n split into a grid).

lensrender

Lens-ray rendering: sensor -> real lens -> scene, for any simulator.

"raycast" traces every sensor pixel's rays through the lens (LensRays, per RGB wavelength) and intersects each with the scene's triangles on the GPU (NVIDIA Warp BVH) for its exact hit point. The hit's color is the simulator's own shading (no PBR): the entrance pupil is split into views cells, the simulator rasterizes one wide pinhole view (RGB + depth) from each cell's center, and a hit takes its color from the nearest view in which it is visible (a depth test). This gives exact per-ray visibility with the simulator's shading, which is view-dependent only through specular highlights.

A simulator plugs in through a lens scene (LensRayRenderer.render_scene):

• render_views(views) -> (rgb [k, 3, h, w] linear, depth [k, 1, h, w] m): the views across the pupil, pinholes of views.fovy_deg at views.width x views.height, from the camera shifted by each views.centers offset (m, camera frame);
• camera_pose() -> (R [3, 3], p [3]): camera axes as columns (x right, y up, looking along -z) and position, world frame, meters;
• mesh(device) -> a TriangleMesh of the scene at the current state.

Adapters: twinrobo.plugins.mujoco.lensrender (MuJoCo) and twinrobo.plugins.isaac.lens (Isaac Sim). Rays depend on lens, focus and sensor only, so they are traced once (lens_rays) and cached on the GPU.

PupilViews

PupilViews(rays: LensRays, k: int, margin: float = 0.06, oversample: float = 1.0)

k rasterized views (linear RGB + z-depth) from the pupil cell centers.

set_views

set_views(rgb: Tensor, depth: Tensor) -> None

Install rendered views: rgb [k, 3, h, w] linear, depth [k, 1, h, w] m.

depth_at

depth_at(k: Tensor, tan: Tensor) -> Tensor

Nearest-tap z-depth of per-ray views k along tan.

shade

shade(k: Tensor, tan: Tensor, channel: int, zref: Tensor)

Color of per-ray views k [M] along tan, using only taps at depth zref.

A lookup beside a silhouette would otherwise blend the occluder into the surface behind it (and rasterizers antialias color, so silhouette pixels are themselves blends). Preference: matching depth off the silhouette ("clean"), then matching depth on it, then the nearest tap. Returns (color, clean, seen): whether a clean tap was used, and whether any tap matches zref.

color_at

color_at(k: Tensor, tan: Tensor, channel: int) -> Tensor

Plain bilinear color of per-ray views k along tan (no depth test).

TriangleMesh

TriangleMesh(local: ndarray, faces: ndarray, owner: ndarray, device)

Scene triangles as one GPU BVH (NVIDIA Warp), posed per frame and refit.

local [V, 3] vertices in their owners' frames, faces [F, 3], owner [V]: the index of each vertex's rigid frame. update poses all of them.

update

update(R: Tensor, t: Tensor) -> None

Pose every owner: world = R[owner] @ local + t[owner].

R [P, 3, 3] rotations (with scale, if any) and t [P, 3] translations.

cast

cast(origins: Tensor, dirs: Tensor, max_t: float = 100.0) -> Tensor

Hit distance along unit dirs (1e30 for a miss), [M].

cast_faces

cast_faces(origins: Tensor, dirs: Tensor, max_t: float = 100.0) -> tuple[Tensor, Tensor]

Hit distance along unit dirs (1e30 for a miss) and the face hit (-1).

face_owner

face_owner(face: Tensor) -> Tensor

The owner (rigid frame) of each face [M] (faces of one owner share it).

face_normal

face_normal(face: Tensor) -> Tensor

World-frame unit normals of faces [M] at the last update.

LensRayRenderer

LensRayRenderer(rays: LensRays, method: str = 'raycast', views: int = 7, shading: str = 'corrected', chunk: int = 4000000, oversample: float = 1.0)

Render a camera through a real lens by ray casting ("raycast").

Parameters:

Name	Type	Description	Default
rays	LensRays	traced LensRays of the camera's lens and sensor.	required
method	str	"raycast" (the only lens-ray method).	'raycast'
views	int	number of pupil cells / rasterized shading views.	7
oversample	float	resolution of the views relative to the sensor. Above 1 resolves detail thinner than a sensor pixel (the views are the shading source).	1.0
shading	str	"corrected" divides out relative illumination, as a camera ISP's lens-shading correction does; "raw" keeps sensor irradiance falloff.	'corrected'
chunk	int	rays per GPU batch.	4000000

render_scene

render_scene(scene) -> tuple[Tensor, Tensor]

(rgb [1, 3, H, W] linear, depth [1, 1, H, W] m) of a lens scene's current state.

lens_rays

lens_rays(reference, rays_per_pixel: int = 8, status=None, intrinsics=None, distortion=None) -> LensRays

Traced rays of a DeepLensOptics (cached; the first trace takes seconds).

With intrinsics (and optionally distortion), the rays take that calibrated geometry and keep the lens' blur (LensRays.with_geometry); see CameraTwin.lens_geometry. Without, the traced lens' own distortion is kept.

pupil_centers

pupil_centers(rays: LensRays, k: int, iters: int = 12) -> Tensor

k cell centers [k, 2] (m) on the entrance pupil: k-means of the ray origins.

check_method

check_method(method: str, allowed=METHODS) -> None

Raise a clear error for an unknown or removed rendering method.

lens_frame

lens_frame(twin, renderer: LensRayRenderer, rgb: Tensor, depth: Tensor, ideal: Tensor, rectify: bool, timestamp=None, metadata: dict | None = None, force_depth: bool = False)

A CameraFrame from a lens render: sensor and ISP, then optional rectification.

rgb/depth come from LensRayRenderer.render_scene; ideal is the paraxial pinhole view of the same sensor (renderer.rays.intrinsics), linear. rectify undistorts with the lens' own chief rays, as stereo SDKs deliver. force_depth: expose depth even if the camera outputs none (see CameraTwin.process).