SAC Implementation in NVIDIA Isaac Sim

Prerequisites

• NVIDIA GPU (RTX series recommended)
• NVIDIA Omniverse
• Isaac Sim
• ROS2 (Humble)

Installation

Clone the repository: bash git clone https://github.com/ch33nchan/isaac-sim

Algorithm Overview

Soft Actor-Critic (SAC) is an off-policy maximum entropy deep reinforcement learning algorithm that optimizes a stochastic policy in continuous action spaces.

Mathematical Formulation

The objective is to maximize the expected return and entropy:

J(π) = ∑ E(st,at)~ρπ [r(st,at) + αH(π(·|st))]

where: - π: Policy - st: State at time t - at: Action at time t - ρπ: State-action distribution - r: Reward function - α: Temperature parameter - H: Entropy

Key Components

1. Stochastic Policy: Uses a Gaussian policy for continuous action spaces, enabling exploration through probabilistic action selection.

2. Entropy Regularization: Maximizes entropy to encourage exploration while maintaining good performance, controlled by temperature parameter α.

3. Off-Policy Learning: Utilizes experience replay buffer for sample-efficient learning from past experiences.

Environment Configuration

Arena Layout

• 9x9 grid layout (Sudoku-style)
• Start position: Center of upper-left region
• Goal position: Center of lower-right region
• Obstacles: 2 random cubes per grid cell (except start/goal)

State Space

• 20 LIDAR scans (360° coverage)
• Each scan: 18 degrees of freedom
• X-Y plane measurements

Reward Structure

• Positive reward: Decreasing distance to goal
• Negative reward: Collision with obstacles
• Terminal conditions: Goal reached or collision

Implementation Details

Core Components

1. ROS2 Bridge:

   • Enables communication between Isaac Sim and ROS2
   • Handles message passing and synchronization

2. Control System:

   • Steering control publisher
   • Velocity control publisher
   • Real-time state updates

3. Environment Management:

   • Dynamic obstacle placement
   • State reset functionality
   • Model state tracking

4. Collision Detection:

   • Bitwise image processing for collision detection
   • Black/white image logic for obstacle avoidance
   • Real-time collision monitoring

Code Structure

The main script (train_sac_isaac.py) contains: 1. Environment setup and ROS2 bridge initialization 2. SAC implementation with entropy regularization 3. Reward calculation and state management 4. Collision detection using bitwise operations 5. Training loop with experience replay

Training Process

The agent learns through episodic training: 1. Environment reset with random obstacle placement 2. State observation through LIDAR scans 3. Action selection via stochastic policy 4. Reward calculation based on goal proximity and collisions 5. Policy update using SAC algorithm

Dependencies

plaintext isaacsim-extscache-physics==4.2.0.2 isaacsim-extscache-kit==4.2.0.2 isaacsim-extscache-kit-sdk==4.2.0.2 --extra-index-url https://pypi.nvidia.com ```