ManiSkill2-Real2Sim

This repository is forked from the original ManiSkill2 repo, with the following changes: - Environment (mani_skill2_real2sim/envs): We removed all environments irrelevant to real-to-sim evaluation, and we implemented real-to-sim evaluation environments under mani_skill2_real2sim/envs/custom_scenes. These custom environments act as an independent component of ManiSkill2, allowing for automatic integration into the original ManiSkill2 repository without necessitating any modifications. - Robot agents: We added new robot implementations in mani_skill2_real2sim/agents/configs and mani_skill2_real2sim/agents/robots. The corresponding robot assets (URDFs) are in mani_skill2_real2sim/assets/descriptions. - Controllers: We modified pd_joint_pos.py, pd_ee_pose.py, and __init__.py under mani_skill2_real2sim/agents/controllers/, along with base_controller.py and utils.py under ManiSkill2_real2sim/mani_skill2_real2sim/agents/, to support more controller implementations. These scripts can be automatic integrated into the original ManiSkill2 repository. - Object assets: We added custom objects in data/custom and custom scenes in data/hab2_bench_assets for real-to-sim evaluation purposes. Additionally, we use MS2_REAL2SIM_ASSET_DIR to specify the asset directory for custom objects and scenes (if this environment variable is not set, we will use this repo's data directory). - Demo manual control script (mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py): The script is modified from mani_skill2_real2sim/examples/demo_manual_control.py of the original ManiSkill2 repo to support custom real-to-sim environment creationg and visualization. See the script details for usage.

To install, run pip install -e .

(Original ManiSkill2 docs: https://haosulab.github.io/ManiSkill2)

Example in interactive python:

```python import maniskill2real2sim.envs, gymnasium as gym import numpy as np from transforms3d.euler import euler2quat from sapien.core import Pose

env1 = gym.make('GraspSingleOpenedCokeCanInScene-v0', obsmode='rgbd', prepackagedconfig=True) obs1, resetinfo1 = env1.reset() instruction1 = env1.getlanguageinstruction() image1 = obs1['image']['overheadcamera']['rgb'] obs1alt, resetinfo1alt = env1.reset(options={'objinitoptions': {'initxy': [-0.35, -0.02], 'orientation': 'laidvertically'}}) instruction1alt = env1.getlanguageinstruction() image1alt = obs1alt['image']['overhead_camera']['rgb']

env2 = gym.make('MoveNearGoogleBakedTexInScene-v0', obsmode='rgbd', robot='googlerobotstatic', simfreq=513, controlfreq=3, controlmode='armpdeedeltaposealigninterpolatebyplannergripperpdjointtargetdeltaposinterpolatebyplanner', maxepisodesteps=80, scenename='googlepickcokecan1v4', cameracfgs={"addsegmentation": True}, rgboverlaypath='./data/realinpainting/googlemovenearrealeval1.png', rgboverlaycameras=['overheadcamera'], urdfversion='recolortabletopvisualmatching_1' )

remove "rgboverlaypath", "rgboverlaycameras", and "urdf_version" if you do not want to overlay the real background

obs2, resetinfo2 = env2.reset(options={ 'robotinitoptions': { 'initxy': np.array([0.35, 0.21]), 'initrotquat': (Pose(q=euler2quat(0, 0, -0.09)) * Pose(q=[0, 0, 0, 1])).q, }, 'objinitoptions': {'episodeid': 0} }) instruction2 = env2.getlanguageinstruction() image2 = obs2['image']['overhead_camera']['rgb']

env3 = gym.make('CloseDrawerCustomInScene-v0', obsmode='rgbd', robot='googlerobotstatic', simfreq=513, controlfreq=3, maxepisodesteps=113, controlmode='armpdeedeltaposealigninterpolatebyplannergripperpdjointtargetdeltaposinterpolatebyplanner', scenename='modernofficenoroof', stationname='mkstation2', # cabinet model shaderdir='rt', # enable raytracing, slow for non RTX gpus ) obs3, resetinfo3 = env3.reset(options={ 'robotinitoptions': { 'initxy': np.array([0.75, 0.00]), }, }) instruction3 = env3.getlanguage_instruction() image3 = obs3['image']['overhead_camera']['rgb']

env4 = gym.make('PutSpoonOnTableClothInScene-v0', obsmode='rgbd', robot='widowx', simfreq=500, controlfreq=5, controlmode='armpdeetargetdeltaposealign2gripperpdjointpos', maxepisodesteps=60, scenename='bridgetable1v1', cameracfgs={"addsegmentation": True}, rgboverlaypath='./data/realinpainting/bridgerealeval1.png', rgboverlaycameras=['3rdviewcamera'], )

remove "rgboverlaypath", "rgboverlaycameras", and "urdf_version" if you do not want to overlay the real background

obs4, resetinfo4 = env4.reset(options={ 'objinitoptions': {'episodeid': 0} }) instruction4 = env4.getlanguage_instruction() image4 = obs4['image']['3rdviewcamera']['rgb'] ```