MuJoCo AR

mujoco_ar is a plugin for MuJoCo that lets you control frames using your iOS/Android device's AR data.

Recent Updates

• Android support thanks to @Lr-2002's contribution.
• Trigger vibrations on your iOS device for haptic feedback (release log).

MuJoCo Demos

Position Control
MuJoCo Fruits Pick and Place		MuJoCo PushT
Position and Rotation Control
MuJoCo Study Desk		MuJoCo Blocks Stacking

Real Demo

Examples of MuJoCo AR linked to the end-effectors of multiple manipulators can be found in this fork of Mink.

Installation

You can install mujoco_ar package using pip:

bash pip install mujoco_ar

You can download the app from the App Store for iOS devices or here for Android devices.

Usage

Quick MuJoCo Setup

This setup allows you to directly control a MuJoCo frame (body, geom, or site), with the frame's position and orientation matching the ARKit data received from the connected iOS device.

```python from mujoco_ar import MujocoARConnector

Initialize the connector with your desired parameters

mjar = MujocoARConnector(mujocomodel=mymodel,mujocodata=my_data)

Link a MuJoCo frame (linkbody(), linkgeom() or link_site())

mjar.linkbody(name="eef_target")

Start the connector

mj_ar.start() ```

Full MuJoCo Setup

In addition to what the quick setup allows you to do, this setup allows you to automate the applying of a translation, rotation or scaling of the recieved pose. Additionally, you can pass functions to buttonfn and togglefn to be triggered when the button or toggle are activated

```python from mujoco_ar import MujocoARConnector

Initialize the connector with your desired parameters

mjar = MujocoARConnector( mujocomodel=mymodel, mujocodata=my_data, port=8888, # Optional, defaults to 8888 if not provided debug=False # Optional, defaults to False if not provided )

Link a MuJoCo frame (linkbody(), linkgeom() or link_site())

mjar.linkbody( name="eeftarget", scale=1.0, # Optional, defaults to 1.0 if not provided positionorigin=np.array([0.0, 0.0, 0.0]), # Optional, defaults to [0, 0, 0] if not provided rotationorigin=np.identity(3), # Optional, defaults to I(3) if not provided togglefn=mytogglefunction, # Optional, calls nothing if not provided buttonfn=mybuttonfunction, # Optional, calls nothing if not provided disablepos=False, # Optional, defaults to False if not provided disable_rot=False # Optional, defaults to False if not provided )

Start the connector

mj_ar.start() ```

Flexible Setup (works without MuJoCo):

You can retrieve the ARKit data including the position, rotation, button, and toggle states directly from a connected iOS device, making it flexible for usage in various applications beyond physics simulations. Try running mjpython demos/flexible_setup.py.

```python from mujoco_ar import MujocoARConnector

Initialize the connector with your desired parameters

connector = MujocoARConnector()

Start the connector

connector.start()

Retrieve the latest AR data (after connecting the iOS device, see the guide below)

data = connector.getlatestdata() # Returns {"position": (3, 1), "rotation": (3, 3), "button": bool, "toggle": bool} ```

Additional Functions

```python connector.vibrate(sharpness=0.8, intensity=0.4, duration=0.01) # Trigger a vibration on the connected device connector.pauseupdates() # Temporarily stops receiving updates from the connected device. connector.resumeupdates() # Resumes receiving updates from the connected device. connector.reset_position() # Resets the current position as the origin (0,0,0).

```

FAQ

How can I reduce latency?

• If you're experiencing latency, try connecting your PC to your device's hotspot. This should significantly reduce latency if you're far from a router since the communication happens locally via WebSockets.

Can I use it for a non-MuJoCo application?

• Yes, check the Flexible Setup out where you can retrive the pure ARKit position and rotation and use it as you wish. You wouldn't need to pass in the MuJoCo model and data in such a case.

How to contribute?

Report any bugs you encounter to the issue tracker. Also, feel free to suggest any features. Those could be regarding the app (iOS swift code) or the python package.

Acknowledgements

• Thanks to @Lr-2002 for creating an Android app that works with the package.
• Thanks to @kevinzakka for the mink and mjctrl libraries, which are used in the provided demos.

Citation

If you use mujoco_ar in your research, please cite it as follows:

bibtex @software{ayyan2024mujocoar, author = {Rayyan, Omar}, license = {Apache-2.0}, title = {{MuJoCo AR: Phone Teleoperation for Robots}}, url = {https://github.com/omarrayyann/mujocoar}, version = {1.3.0}, year = {2024}, month = aug, }