DualKukaLLM_Bachelor

This repository contains the source code for our bachelor's project in robotics at Aalborg University, and explores the integration of multiple foundation models (LLM Agent, VLM high-level planner, LLM/VLM error correctors + success detectors) to produce a generalist robot cell capable of performing tasks provided by operators. The GitHub contains all the code produced throughout the project period, from February 1st to May 28th

This project is a continuation of the work performed during the 5th semester. The GitHub page for the 5th semester project can be found here: - P5 Project

Quick Start

This quick-start guide assumes a clean installation of Ubuntu 22.04 LTS (Jammy Jellyfish) along with ROS 2 Humble. If either is not yet set up, you can refer to the following resources: - Ubuntu 22.04 LTS - ROS2 - Humble

If your system crashes or you need to switch terminals at any point, ensure you re-run the following command:

source /opt/ros/humble/setup.bash

Before starting the installation of this project, we recommend grabbing a snack and a beverage. The process will take at least 30 minutes.

1. Install ROS 2 development tools

   shell sudo apt install ros-dev-tools

2. Create a workspace, clone, and install dependencies

   shell cd ~ source /opt/ros/humble/setup.bash git clone https://github.com/thor2643/Dual_Kuka_LLM_Bachelor.git cd Dual_Kuka_LLM_Bachelor/p6_ws sudo rosdep init rosdep update rosdep install --from-paths src -i -r -y

3. Installing colcon mixin and pip (Needed for Moveit2)

   shell sudo apt install python3-colcon-common-extensions sudo apt install python3-colcon-mixin sudo apt install python3-pip sudo apt-get install portaudio19-dev python-all-dev colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml colcon mixin update default sudo apt install python3-vcstool

4. Installing Python libraries

   shell cd ~/Dual_Kuka_LLM_Bachelor pip install -r requirements.txt

5. Adding an API key for Janise

shell cd ~/Dual_Kuka_LLM_Bachelor/p6_ws/src/robutler/janise nano API_KEY.json # Added your API Key for GPT following the format # { # "API_KEY": "Insert your API Key here" # } # Remember to write the file, ctrl + s then ctrl + x

1. Colcon build - Re-run if any packages timeout during compile

   shell cd ~/Dual_Kuka_LLM_Bachelor/p6_ws export MAKEFLAGS="-j3" colcon build --mixin release

2. NOTE: If you are running on our setup, remember to set your IP to 172.31.1.148 and Netmask to 255.255.255.0, also ensure that you are correctly connected to the setup (2 USB's 1 Ethernet). (Static IP configuration)

3. Running the system

   shell cd ~/Dual_Kuka_LLM_Bachelor/p6_ws source install/setup.bash ros2 launch robutler_bringup robutler.launch.py

4. Recommendations - Don't do this, if you run multiple ROS Distros

   shell echo "source /opt/ros/humble/setup.bash" >> ~/.bashrc echo "source ~/Dual_Kuka_LLM_Bachelor/p6_ws/install/setup.bash" >> ~/.bashrc

Contributors

This project was developed by Group 660 at Aalborg University, under the supervision of Dimitris Chrysostomou and Chen Li, during their Robotics Bachelor's. Every member of the group can be found listed below, alongside their GitHub pages:

Gustav Bay Baastrup

Thor Iversen

Kasper Lauritsen

Silas Jensen

Signe Mller-Skuldbl