LIBERO and RoboCasa¶
LIBERO integration: CameraTwin cameras in LIBERO environments.
LIBERO runs on robosuite, which runs on MuJoCo. This package provides:
CameraTwinLiberoEnv: wraps a LIBERO env and replaces camera observations (e.g.agentview_image) with CameraTwin frames, keeping each key's shape, dtype and image convention, so policies and eval scripts run unchanged.make_env/make_env_from_bddl: LIBERO envs with any registered robot (twinrobo.datasets.libero.robots), andreplayhelpers for exact demo replay.
Setup. LIBERO is not pip-installable as a package, since its libero/ has
no __init__.py. Point LIBERO_ROOT at a LIBERO checkout. Also set
LIBERO_CONFIG_PATH to a directory holding LIBERO's config.yaml:
without it, LIBERO prompts on stdin at first import. setup_libero checks
both and makes import libero work.
Make import libero work without LIBERO's interactive first-run prompt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
libero_root
|
str | Path | None
|
LIBERO checkout (the directory containing |
None
|
config_path
|
str | Path | None
|
Directory with LIBERO's |
None
|
(env, task, init_states) for a benchmark task, as LIBERO's eval scripts build it.
A LIBERO env for a bddl path as stored in LIBERO datasets (.../<suite>/<task>.bddl).
LIBERO env wrapper that swaps camera observations for CameraTwin frames.
CameraTwinLiberoEnv(env, cameras: dict[str, CameraTwin], crop: str = 'center', match_fov: bool = True, keep_ideal: bool = False)
Wrap a LIBERO env (OffScreenRenderEnv) so its cameras are CameraTwin cameras.
Each camera name in cameras (a robosuite camera such as
"agentview") is rendered at the twin's resolution and FoV from the same
camera pose, run through the twin's optics, and written back into
obs[f"{name}_image"] at the key's original size, dtype and orientation.
Code downstream of the env is unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
OffScreenRenderEnv
|
LIBERO env, e.g. from |
required |
cameras
|
dict[str, CameraTwin]
|
|
required |
crop
|
str
|
How the twin frame (e.g. 16:10) becomes the observation (e.g.
square): |
'center'
|
match_fov
|
bool
|
Render with the twin's paraxial FoV. |
True
|
keep_ideal
|
bool
|
Also return the untouched observation as |
False
|
CameraTwin observation of camera name: uint8 [h, w, 3], robosuite convention.
Additional robots for LIBERO scenes, registered by TwinRobo (LIBERO is not modified).
LIBERO builds robots=[name] as "Mounted" + name and only ships
MountedPanda. The classes here wrap robosuite's own arms the same way. They
inherit everything (init pose, gripper, mount, controller) and add only the
table-placement offsets LIBERO scenes look up.
module-attribute
¶
ROBOT_CATALOG = [{'name': 'Panda', 'label': 'Franka Panda', 'category': 'Arms', 'base': 'Panda'}, {'name': 'Sawyer', 'label': 'Rethink Sawyer', 'category': 'Arms', 'base': 'Sawyer'}, {'name': 'UR5e', 'label': 'Universal Robots UR5e', 'category': 'Arms', 'base': 'UR5e'}, {'name': 'IIWA', 'label': 'KUKA LBR iiwa', 'category': 'Arms', 'base': 'IIWA'}, {'name': 'Jaco', 'label': 'Kinova Jaco', 'category': 'Arms', 'base': 'Jaco'}, {'name': 'Kinova3', 'label': 'Kinova Gen3', 'category': 'Arms', 'base': 'Kinova3'}, {'name': 'PandaMultiCam', 'label': 'Franka Panda · wrist stereo + mast', 'category': 'Multi-camera arms', 'base': 'Panda'}, {'name': 'UR5eMultiCam', 'label': 'UR5e · wrist stereo + mast', 'category': 'Multi-camera arms', 'base': 'UR5e'}, {'name': 'Kinova3MultiCam', 'label': 'Kinova Gen3 · wrist stereo + mast', 'category': 'Multi-camera arms', 'base': 'Kinova3'}, {'name': 'PandaOmron', 'label': 'Franka Panda on Omron mobile base', 'category': 'Mobile manipulators', 'base': 'PandaOmron', 'simulator': 'robocasa', 'policy_size': 256, 'robot_cameras': ['agentview_left', 'agentview_right', 'eye_in_hand']}, *[{'name': f'{arm}Omron', 'label': f'{label} on Omron mobile base', 'category': 'Mobile manipulators', 'base': f'{arm}Omron', 'simulator': 'robocasa', 'policy_size': 256, 'robot_cameras': ['agentview_left', 'agentview_right', 'eye_in_hand'], 'robosuite': {'robots': arm, 'base_types': 'OmronMobileBase'}, 'keep_pose': True} for arm, label in [('UR5e', 'Universal Robots UR5e'), ('Kinova3', 'Kinova Gen3'), ('IIWA', 'KUKA LBR iiwa'), ('Jaco', 'Kinova Jaco'), ('Sawyer', 'Rethink Sawyer'), ('XArm7', 'UFACTORY xArm 7'), ('PandaDexRH', 'Franka Panda with a dexterous hand')]], {'name': 'GR1FloatingBody', 'label': 'Fourier GR1 humanoid', 'category': 'Humanoids', 'base': 'GR1FloatingBody', 'simulator': 'robocasa', 'policy_size': 256, 'robot_cameras': ['agentview_center', 'eye_in_right_hand', 'eye_in_left_hand'], 'robosuite': {'robots': 'GR1FloatingBody'}, 'keep_pose': False}]
The arm whose recorded demos name can replay (itself for plain arms).
Robots selectable in scenes of simulator ("libero" or "robocasa").
Register Mounted<Robot> for every non-Panda robot in ROBOTS (idempotent).
Exact replay of recorded LIBERO episodes.
A recorded states / qpos trajectory does not fully determine the scene.
LIBERO fixtures (cabinet, stove, ...) have no joints, and env.reset()
randomizes their placement. Their pose lives only in the episode's model XML
(the demo's model_file attribute, or a run record's scene.xml). That XML cannot
be loaded as is: it references the recording machine's asset paths. So the
static top-level bodies' poses are read from it and applied to the live model.
{body name: (pos, quat wxyz)} for the worldbody's direct children in an MJCF string.
Set the pose of every jointless top-level body from the episode's MJCF. Returns their names.
Bodies with joints (objects, the robot) are driven by the recorded state and are left alone.
RoboCasa integration: realistic kitchen scenes (robosuite 1.5, MuJoCo) for CameraTwin.
RoboCasa ships procedurally assembled kitchens (layouts x styles, textured
fixtures, appliances, Objaverse objects) and human demonstrations as LeRobot
datasets. Every episode stores its exact generated kitchen (model.xml.gz),
its simulator states (states.npz) and ep_meta.json, so an episode is
replayed exactly by loading that kitchen and applying the recorded states.
Environment. RoboCasa needs robosuite 1.5 and LIBERO needs 1.4.1, so they
cannot share a Python process: use the twinrobo-robocasa environment
(see README). Assets are the RoboCasa asset pack, which lives outside the
package; setup_robocasa links it into the installed package when missing.
Check RoboCasa is importable and its assets are present; returns the assets dir.
asset_pack (or $ROBOCASA_ASSETS): a RoboCasa asset pack directory
(with objects/, textures/, generative_textures/, fixtures/).
Missing top-level asset folders in the installed package are symlinked to it;
nothing in the pack is modified.
A RoboCasa env from a dataset's env_args (offscreen rendering, no camera obs).
robosuite's arguments for a catalog robot, e.g. UR5eOmron: a UR5e on the Omron base
(twinrobo.datasets.libero.robots.ROBOT_CATALOG); other names are robosuite's own.
An episode's kitchen rebuilt around another robot: RoboCasa regenerates the layout,
style and objects from ep_meta (the episode's model XML holds the recording robot).
The recorded robot base pose is kept for robots on the same mobile base; others are placed by RoboCasa (a humanoid stands higher).
Load an episode's exact kitchen (as RoboCasa's own playback script does).
(states [T, S], model_xml, ep_meta) of a RoboCasa episode (episode_000004).