yolo_ros

ROS 2 wrap for YOLO models from Ultralytics to perform object detection and tracking, instance segmentation, human pose estimation and Oriented Bounding Box (OBB). There are also 3D versions of object detection, including instance segmentation, and human pose estimation based on depth images.

Table of Contents

1. Installation
2. Docker
3. Models
4. Usage
5. Demos

Installation

shell cd ~/ros2_ws/src git clone https://github.com/mgonzs13/yolo_ros.git pip3 install -r yolo_ros/requirements.txt cd ~/ros2_ws rosdep install --from-paths src --ignore-src -r -y colcon build

Docker

Build the yolo_ros docker.

shell docker build -t yolo_ros .

Run the docker container. If you want to use CUDA, you have to install the NVIDIA Container Tollkit and add --gpus all.

shell docker run -it --rm --gpus all yolo_ros

Models

The compatible models for yolo_ros are the following:

• YOLOv3
• YOLOv4
• YOLOv5
• YOLOv6
• YOLOv7
• YOLOv8
• YOLOv9
• YOLOv10
• YOLOv11
• YOLOv12
• YOLO-World
• YOLOE

Usage

Topics

• /yolo/detections: Objects detected by YOLO using the RGB images. Each object contains a bounding box and a class name. It may also include a mark or a list of keypoints.
• /yolo/tracking: Objects detected and tracked from YOLO results. Each object is assigned a tracking ID.
• /yolo/detections_3d: 3D objects detected. YOLO results are used to crop the depth images to create the 3D bounding boxes and 3D keypoints.
• /yolo/debug_image: Debug images showing the detected and tracked objects. They can be visualized with rviz2.

Parameters

These are the parameters from the yolo.launch.py, used to launch all models. Check out the Ultralytics page for more details.

• model_type: Ultralytics model type (default: YOLO)
• model: YOLO model (default: yolov8m.pt)
• tracker: tracker file (default: bytetrack.yaml)
• device: GPU/CUDA (default: cuda:0)
• yolo_encoding: Encoding to convert input image before using YOLO (default: bgr8)
• enable: whether to start YOLO enabled (default: True)
• threshold: detection threshold (default: 0.5)
• iou: intersection Over Union (IoU) threshold for Non-Maximum Suppression (NMS) (default: 0.7)
• imgsz_height: image height for inference (default: 480)
• imgsz_width: image width for inference (default: 640)
• half: whether to enable half-precision (FP16) inference speeding up model inference with minimal impact on accuracy (default: False)
• max_det: maximum number of detections allowed per image (default: 300)
• augment: whether to enable test-time augmentation (TTA) for predictions improving detection robustness at the cost of speed (default: False)
• agnostic_nms: whether to enable class-agnostic Non-Maximum Suppression (NMS) merging overlapping boxes of different classes (default: False)
• retina_masks: whether to use high-resolution segmentation masks if available in the model, enhancing mask quality for segmentation (default: False)
• inputimagetopic: camera topic of RGB images (default: /camera/rgb/image_raw)
• image_reliability: reliability for the image topic: 0=system default, 1=Reliable, 2=Best Effort (default: 1)
• inputdepthtopic: camera topic of depth images (default: /camera/depth/image_raw)
• depthimagereliability: reliability for the depth image topic: 0=system default, 1=Reliable, 2=Best Effort (default: 1)
• inputdepthinfo_topic: camera topic for info data (default: /camera/depth/camera_info)
• depthinforeliability: reliability for the depth info topic: 0=system default, 1=Reliable, 2=Best Effort (default: 1)
• target_frame: frame to transform the 3D boxes (default: base_link)
• depthimageunits_divisor: divisor to convert the depth image into meters (default: 1000)
• maximumdetectionthreshold: maximum detection threshold in the z-axis (default: 0.3)
• use_tracking: whether to activate tracking after detection (default: True)
• use_3d: whether to activate 3D detections (default: False)
• use_debug: whether to activate debug node (default: True)

Lifecycle Nodes

Previous updates add Lifecycle Nodes support to all the nodes available in the package. This implementation tries to reduce the workload in the unconfigured and inactive states by only loading the models and activating the subscriber on the active state.

These are some resource comparisons using the default yolov8m.pt model on a 30fps video stream.

| State | CPU Usage (i7 12th Gen) | VRAM Usage | Bandwidth Usage | | -------- | ----------------------- | ---------- | --------------- | | Active | 40-50% in one core | 628 MB | Up to 200 Mbps | | Inactive | ~5-7% in one core | 338 MB | 0-20 Kbps |

YOLO 3D

shell ros2 launch yolo_bringup yolov8.launch.py use_3d:=True

Demos

Object Detection

This is the standard behavior of yolo_ros which includes object tracking.

shell ros2 launch yolo_bringup yolo.launch.py

Instance Segmentation

Instance masks are the borders of the detected objects, not all the pixels inside the masks.

shell ros2 launch yolo_bringup yolo.launch.py model:=yolov8m-seg.pt

Human Pose

Online persons are detected along with their keypoints.

shell ros2 launch yolo_bringup yolo.launch.py model:=yolov8m-pose.pt

3D Object Detection

The 3D bounding boxes are calculated by filtering the depth image data from an RGB-D camera using the 2D bounding box. Only objects with a 3D bounding box are visualized in the 2D image.

shell ros2 launch yolo_bringup yolo.launch.py use_3d:=True

3D Object Detection (Using Instance Segmentation Masks)

In this, the depth image data is filtered using the max and min values obtained from the instance masks. Only objects with a 3D bounding box are visualized in the 2D image.

shell ros2 launch yolo_bringup yolo.launch.py model:=yolov8m-seg.pt use_3d:=True

3D Human Pose

Each keypoint is projected in the depth image and visualized using purple spheres. Only objects with a 3D bounding box are visualized in the 2D image.

shell ros2 launch yolo_bringup yolo.launch.py model:=yolov8m-pose.pt use_3d:=True