Data

Tools

Indexes

Applications

Experiments

Open Source Science

visionproteleop

VisionOS App + Python Library to stream head / wrist / finger tracking data from Vision Pro to any robots.

https://github.com/improbable-ai/visionproteleop

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.6%) to scientific vocabulary
Last synced: 6 months ago · JSON representation ·

Repository

VisionOS App + Python Library to stream head / wrist / finger tracking data from Vision Pro to any robots.

Basic Info
  • Host: GitHub
  • Owner: Improbable-AI
  • License: mit
  • Language: Swift
  • Default Branch: main
  • Homepage:
  • Size: 65.9 MB
Statistics
  • Stars: 513
  • Watchers: 13
  • Forks: 29
  • Open Issues: 9
  • Releases: 0
Created about 2 years ago · Last pushed almost 2 years ago
Metadata Files
Readme License Citation

README.md

VisionProTeleop

CleanShot 2024-03-03 at 13 55 11@2x

Wanna use your new Apple Vision Pro to control your robot? Wanna record how you navigate and manipulate the world to train your robot? This VisionOS app and python library streams your Head + Wrist + Hand Tracking result via gRPC over a WiFi network, so any robots connected to the same wifi network can subscribe and use.

For a more detailed explanation, check out this short paper.

How to Use

If you use this repository in your work, consider citing:

@software{park2024avp,
    title={Using Apple Vision Pro to Train and Control Robots},
    author={Park, Younghyo and Agrawal, Pulkit},
    year={2024},
    url = {https://github.com/Improbable-AI/VisionProTeleop},
}

Step 1. Install the app on Vision Pro

This app is now officially on VisionOS App Store! You can search for Tracking Streamer from the App Store and install the app.

If you want to play around with the app, you can build/install the app yourself too. To learn how to do that, take a look at this documentation. This requires (a) Apple Developer Account, (b) Vision Pro Developer Strap, and (c) a Mac with Xcode installed.

Step 2. Run the app on Vision Pro

After installation, click on the app on Vision Pro and click Start. That's it! Vision Pro is now streaming the tracking data over your wifi network.

Tip Remember the IP address before you click start; you need to specify this IP address to subscribe to the data. Once you click start, the app will immediately enter into pass-through mode. Click on the digital crown to stop streaming.

Step 3. Receive the stream from anywhere

The following python package allows you to receive the data stream from any device that's connected to the same WiFi network. First, install the package:

pip install avp_stream

Then, add this code snippet to any of your projects you were developing:

```python from avpstream import VisionProStreamer avpip = "10.31.181.201" # example IP s = VisionProStreamer(ip = avp_ip, record = True)

while True: r = s.latest print(r['head'], r['rightwrist'], r['rightfingers']) ```

Available Data

python r = s.latest

r is a dictionary containing the following data streamed from AVP:

python r['head']: np.ndarray # shape (1,4,4) / measured from ground frame r['right_wrist']: np.ndarray # shape (1,4,4) / measured from ground frame r['left_wrist']: np.ndarray # shape (1,4,4) / measured from ground frame r['right_fingers']: np.ndarray # shape (25,4,4) / measured from right wrist frame r['left_fingers']: np.ndarray # shape (25,4,4) / measured from left wrist frame r['right_pinch_distance']: float # distance between right index tip and thumb tip r['left_pinch_distance']: float # distance between left index tip and thumb tip r['right_wrist_roll']: float # rotation angle of your right wrist around your arm axis r['left_wrist_roll']: float # rotation angle of your left wrist around your arm axis

Axis Convention

Refer to the image below to see how the axis are defined for your head, wrist, and fingers.

Hand Skeleton used in VisionOS

Refer to the image above to see what order the joints are represented in each hand's skeleton.

Acknowledgements

We acknowledge support from Hyundai Motor Company and ARO MURI grant number W911NF-23-1-0277.

Owner

  • Name: Improbable-AI
  • Login: Improbable-AI
  • Kind: organization

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Park"
  given-names: "Younghyo"
  orcid: "https://orcid.org/0000-0000-0000-0000"
title: "Teleopeation System using Apple Vision Pro"
version: 0.1.0
url: "https://github.com/Improbable-AI/VisionProTeleop"

GitHub Events

Total
  • Issues event: 5
  • Watch event: 219
  • Issue comment event: 7
  • Fork event: 11
Last Year
  • Issues event: 5
  • Watch event: 219
  • Issue comment event: 7
  • Fork event: 11

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 2
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 2
  • Total pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 2
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 2
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • DiTo97 (2)
  • durul (1)
  • Augston (1)
  • BBBBBBob (1)
  • Xingyu-Lin (1)
  • fnuabhimanyu8713 (1)
  • mrhouse-sweet (1)
  • bethalageetachandraraju (1)
  • acoudgo (1)
  • Steve-Tod (1)
  • GuoZhuoRan (1)
  • StoneT2000 (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

setup.py pypi
  • grpcio *
  • grpcio-tools *
  • numpy *
Tracking Streamer.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved swiftpm
  • github.com/apple/swift-atomics 1.2.0
  • github.com/apple/swift-collections 1.1.0
  • github.com/apple/swift-http-types 1.0.3
  • github.com/apple/swift-log 1.5.4
  • github.com/apple/swift-nio 2.63.0
  • github.com/apple/swift-nio-extras 1.21.0
  • github.com/apple/swift-nio-http2 1.30.0
  • github.com/apple/swift-nio-ssl 2.26.0
  • github.com/apple/swift-nio-transport-services 1.20.1
  • github.com/apple/swift-protobuf 1.25.2
  • github.com/apple/swift-system 1.2.1
  • github.com/grpc/grpc-swift 1.21.1