HBB2OBB: Horizontal to Oriented Bounding Box Conversion and Evaluation Tool

HBB2OBB is a Python tool designed to convert horizontal bounding boxes (HBBs), also known as axis-aligned bounding boxes, into oriented bounding boxes (OBBs), also referred to as rotated bounding boxes, using segmentation models from the SAM (Segment Anything Model) family. This tool addresses a critical need in object detection tasks where objects appear in arbitrary orientations, such as in aerial imagery, satellite data, or traffic monitoring scenarios. The conversion utilizes user-provided HBB annotations as prompts for SAM models, leveraging their state-of-the-art segmentation capabilities to accurately delineate object boundaries and generate precise OBBs that better encapsulate non-upright objects.

The conversion process employs a model ensemble approach that combines masks from multiple segmentation models through majority voting, resulting in enhanced accuracy and robustness. The system implements spatial constraint techniques including region-specific masking and contour refinement to ensure the segmentation remains within relevant object boundaries. The library supports flexible scaling of input HBBs (both positive and negative factors) to accommodate for potentially cropped object parts or overly conservative annotations. If no valid mask is detected, a fallback strategy maintains the original HBB as the OBB, ensuring consistent outputs.

Beyond conversion, HBB2OBB offers comprehensive evaluation tools to assess OBB accuracy against ground truth annotations, hyperparameter optimization capabilities to fine-tune the conversion process for specific datasets, and utilities for format conversion between COCO JSON and YOLO TXT. The package includes intuitive visualization features that render the conversion process transparently, displaying the progression from original HBBs through segmentation masks to final OBBs. Designed with ease of use in mind, HBB2OBB provides both an intuitive command-line interface and a flexible Python API for seamless integration into existing workflows.

Features

• Conversion from HBB to OBB: Automatically converts YOLO format horizontal bounding boxes to oriented bounding boxes
• Segmentation-Based Approach: Uses state-of-the-art segmentation models for accurate object boundary detection
• Multiple Model Support: Compatible with various SAM model variants (SAM, SAM2, SAM2.1, Mobile SAM, and FastSAM families, see ultralytics documentation for details)
• Model Ensemble: Ability to combine outputs from multiple segmentation models for enhanced accuracy through majority voting
• Evaluation Tools: Includes tools to evaluate OBB accuracy against ground truth using IoU metrics
• Hyperparameter Optimization Tool: Finds optimal hyperparameters for HBB2OBB conversion by evaluating different combinations of SAM inference resolutions and scale factors used to enlarge/shrink HBBs
• Visualization Tools: Tools to visualize the conversion process, including HBBs, segmentation masks, derived contours, and the resulting OBBs, as well as the evaluation results
• Format Conversion Utilities: Tools to convert between COCO JSON and YOLO TXT formats for both HBB and OBB annotations

Installation

It is recommended to create and activate a Python Virtual Environment (Python >= 3.9) first using e.g., Miniconda3: bash conda create -n hbb2obb python=3.11 -y conda activate hbb2obb

Then, install the hbb2obb library using one of the following options:

Option 1: Install from PyPI

bash pip install hbb2obb

Option 2: Install from Local Source

You can also clone the repository and install the package from the local source:

bash git clone https://github.com/rfonod/hbb2obb.git cd hbb2obb && pip install .

If you want the changes you make in the repo to be reflected in your install, use pip install -e . instead of pip install ..

CLI Usage

Converting HBB to OBB

To convert HBBs to OBBs using default parameters (single SAM model sam_b), run:

bash hbb2obb /path/to/images --hbb_dir /path/to/hbb/annotations

For enhanced accuracy using multiple segmentation models (model ensemble):

bash hbb2obb /path/to/images --hbb_dir /path/to/hbb/annotations --sam_models sam_b sam_l sam2_b sam2.1_b

To adjust scale factors (useful for recovering cropped object parts or handling conservative HBBs):

```bash

Positive scale factor to expand HBBs (helps recover cropped parts)

hbb2obb /path/to/images --scale_factors 0.1

Negative scale factor to shrink HBBs (useful when HBBs are too conservative)

hbb2obb /path/to/images --scale_factors -0.02

Different scale factors for short and long sides of the HBB

hbb2obb /path/to/images --scale_factors 0.1 0.05 ```

To visualize the conversion process, add the --save_img flag:

bash hbb2obb /path/to/images --save_img

Evaluating OBB Predictions

To evaluate the accuracy of OBB predictions against ground truth annotations:

bash hbb2obb-eval /path/to/ground_truth /path/to/predictions

Python API Usage

Utility Scripts

Format Conversion

COCO JSON to YOLO TXT (supports both HBB and OBB annotations):

bash python scripts/json2yolo.py /path/to/json -mp /path/to/label_map.yaml The -mp flag is optional and can be used to specify a label map file. If not provided, the script will create a default (first-come-first-serve) label map.

YOLO TXT to COCO JSON (supports both HBB and OBB annotations): bash python scripts/yolo2json.py /path/to/yolo /path/to/label_map.yaml Here, the label map file is required to convert the numerical class IDs to class names in the JSON output. The output JSON format is compatible with annotation tools like LabelMe.

Hyperparameter Optimization

To find the optimal hyperparameters for the default SAM model (sam_b), run:

bash python scripts/optimize_hbb2obb.py /path/to/images path/to/ground_truth_annotations

This evaluates different combinations of: - SAM inference resolutions - Scale factors to enlarge/shrink HBBs

The script can be run for different SAM models or combinations of models. For example, to evaluate multiple SAM models: bash python scripts/optimize_hbb2obb.py /path/to/images path/to/ground_truth_annotations -sm sam_b sam_l sam2_b sam2.1_b -n multi_sam

To visualize optimization results: bash python scripts/plot_optimization_results.py /path/to/optimization_results

Data Format

HBB Annotations (Input)

HBB annotations should be in YOLO TXT format (one file per image): class_id x_center y_center width height The coordinates can be in relative format (0-1) or absolute pixel coordinates.

OBB Annotations (Output)

OBB annotations are saved in the following YOLO TXT format (one file per image): class_id x1 y1 x2 y2 x3 y3 x4 y4 Where (x1,y1), (x2,y2), (x3,y3), (x4,y4) are the four corner coordinates of the rotated bounding box in absolute pixel coordinates.

Label Map (Optional)

Label map is a YAML file mapping class IDs to class names. For example: ```yaml 0: Car 1: Bus 2: Truck 3: Motorcycle

...

```

Example Workflow

Basic Workflow

Below is a simple example of how to use the HBB2OBB tool for converting HBB annotations to OBB annotations and evaluating the results. This example assumes you have a dataset with images and HBB annotations in YOLO format. Steps 2-4 are optional and can be skipped if you only want to convert HBBs to OBBs.

1. Prepare HBB annotations in YOLO format dataset/ ├── images/ │ ├── img1.jpg │ ├── img2.jpg │ └── ... ├── labels_hbb/ │ ├── img1.txt │ ├── img2.txt │ └── ... ├── labels_obb_gt/ (optional) │ ├── img1.txt │ ├── img2.txt │ └── ... └── classes.yaml (optional)

💡 Note: The data folder in this repository contains a sample dataset to test the conversion and evaluation processes as well as the parameter optimization and visualization scripts. The README file inside the data folder contains detailed instructions and commands on how to reproduce the results.

1. Convert HBB to OBB annotations and visualize the transformation using default parameters bash hbb2obb data/images --save_img

2. Evaluate OBB predictions against ground truth annotations bash hbb2obb-eval data/labels_obb_gt data/labels_obb -lm data/classes.yaml

3. Optimize hyperparameters for HBB2OBB conversion using a light-weight SAM model bash python scripts/optimize_hbb2obb.py data/images data/labels_obb_gt -sm sam2_s -n sam2_s

4. Visualize optimization results bash python scripts/plot_optimization_results.py data/benchmark_results/sam2_s

Complete Workflow with LabelMe JSON Annotations

Technical Details

The HBB to OBB conversion process involves the following steps:

1. Load HBB annotations: Parse YOLO TXT format annotations
2. Scale bounding boxes: Scale HBB slightly to ensure complete object coverage
   • Positive scale factors: Expand HBBs to recover potentially cropped object parts
   • Negative scale factors: Shrink HBBs when they are overly conservative
   • Different scale factors can be applied to shorter vs. longer sides of the HBB
3. Segmentation: Use SAM model(s) to generate object masks based on the HBB prompts
4. Mask aggregation:
   • When using multiple models (model ensemble), masks are combined through majority voting
   • The aggregated mask is clipped to the scaled HBB region
   • Morphological opening is applied to refine the mask
5. Contour extraction: Extract contours from the largest refined mask per object
6. OBB computation: Calculate minimum-area oriented bounding boxes from the contours
7. Fall-back strategy: If no valid mask is detected inside an HBB, the original HBB is used as the OBB
8. Visualization (optional): Generate images with HBBs, segmentation masks, contours, and OBB overlays

Key characteristics: - Label preservation: OBBs inherit the class labels from their corresponding HBBs (no re-classification) - Corrective effects: The transformation may correct errors in the original HBBs by: - Recovering cropped object parts through positive scale factors - Creating tighter bounding boxes through precise segmentation

Best Practices

• For optimal results, use a combination of SAM models, e.g., --sam_models sam_b sam_l sam2_b sam2.1_b
• Experiment with different scale factors and inference resolutions based on your dataset characteristics
• Run the hyperparameter optimization script to find the best settings for your specific data
• Use class-agnostic evaluation when comparing with manually annotated ground truth that might have different class labels than the original HBBs
• Visualize the conversion process to understand how the model is interpreting the HBBs and generating OBBs
• Regularly check for updates to the library and SAM models for improved performance and new features

Limitations

• The tool relies on the quality of the HBB annotations and the SAM models used for segmentation. Poorly annotated HBBs or low-quality segmentation models may lead to inaccurate OBBs.
• The conversion process may not work well for highly occluded or complex objects where the HBB does not provide sufficient context for the SAM model to generate accurate masks.

Contributing

Contributions are welcome! If you encounter any issues or have suggestions for improvements, please open a GitHub Issue or submit a pull request.

License

This project is distributed under the MIT License. See the LICENSE file for more details.