Property-Based Testing: Formalized Robotic Testing for Standard Compliance

   

   

   

   

Copyright and licence

Copyright (C) 2024 by Hochschule Bonn Rhein Sieg

---

Author

Salman Omar Sohail

Co-authors

Nico Hochgeschwender, Paul G. Pl oger, Sven Schneider

Publications

[!IMPORTANT] Please cite the following:

Automated Testing of Standard Conformance for Robots bash @INPROCEEDINGS{10260447, author={Sohail, Salman Omar and Schneider, Sven and Hochgeschwender, Nico}, booktitle={2023 IEEE 19th International Conference on Automation Science and Engineering (CASE)}, title={Automated Testing of Standard Conformance for Robots}, year={2023}, volume={}, number={}, pages={1-8}, abstract={Verifying that robots conform to standards is an essential part of any responsible development process. A common technique to verify whether robots faithfully meet the requirements of standards is testing. Unfortunately, conformance testing is, to a large extent, a manual and therefore costly exercise which needs to be repeated for every robot under test. In this paper, we propose an automated approach to robot standard conformance testing (RSCT). Based on an analysis of conformance clauses in robot standards (e.g., ISO 234821 and ISO 10218) we identify, harmonise and formalise high-level properties representing the assertions to be tested. To verify whether properties hold, we put forward a Robot Test Definition Language (RTDL) which enables developers to specify reusable test scenarios which can be executed in simulation. We evaluate the approach with four heterogeneous robot platforms, identifying previously undiscovered defects in a robot driver. Our approach enables the automated testing of 40% of the conformance clauses defined in ISO 234821.}, keywords={Computer aided software engineering;Automation;ISO Standards;Manuals;Conformance testing;Robots;Standards}, doi={10.1109/CASE56687.2023.10260447}, ISSN={2161-8089}, month={Aug},}

Property-Based Testing in Simulation for Verifying Robot Action Execution in Tabletop Manipulation bash @INPROCEEDINGS{9568837, author={Sohail, Salman Omar and Mitrevski, Alex and Hochgeschwender, Nico and Plger, Paul G.}, booktitle={2021 European Conference on Mobile Robots (ECMR)}, title={Property-Based Testing in Simulation for Verifying Robot Action Execution in Tabletop Manipulation}, year={2021}, volume={}, number={}, pages={1-7}, abstract={An important prerequisite for the reliability and robustness of a service robot is ensuring the robots correct behavior when it performs various tasks of interest. Extensive testing is one established approach for ensuring behavioural correctness; this becomes even more important with the integration of learning-based methods into robot software architectures, as there are often no theoretical guarantees about the performance of such methods in varying scenarios. In this paper, we aim towards evaluating the correctness of robot behaviors in tabletop manipulation through automatic generation of simulated test scenarios in which a robot assesses its performance using property-based testing. In particular, key properties of interest for various robot actions are encoded in an action ontology and are then verified and validated within a simulated environment. We evaluate our framework with a Toyota Human Support Robot (HSR) which is tested in a Gazebo simulation. We show that our framework can correctly and consistently identify various failed actions in a variety of randomised tabletop manipulation scenarios, in addition to providing deeper insights into the type and location of failures for each designed property.}, keywords={Software architecture;Service robots;Reliability theory;Ontologies;Robustness;Software reliability;Mobile robots}, doi={10.1109/ECMR50962.2021.9568837}, ISSN={}, month={Aug},}

Framework Overview

Code Documentation

Docs: Property-Based Testing: Formalized Robotic Testing for Standard Compliance

Running Framework (Hypothesis|ROS|Gazebo|Allure|TextX)

1. Basic setup bash ./atg.sh
2. Run from inside the src/property_based_tester folder, this applies the tests bash python3 -m pytest --alluredir=results tests/pbt_mobile_robot.py -v -W ignore::DeprecationWarning python3 -m pytest --alluredir=results tests/pbt_robotic_arm.py -v -W ignore::DeprecationWarning python3 -m pytest --alluredir=results tests/pbt_random.py -v -W ignore::DeprecationWarning
   • The first test is for any ground vehicle
   • The second test is for robotic manipulators
   • The third test is for randomized testing

NOTE: Examples for applying these tests are available in src/property_based_tester/ robot_test_definition_language/*_test_definitions

1. Run for the navigation tests bash roslaunch jackal_navigation odom_navigation_demo.launch roslaunch husky_navigation move_base_mapless_demo.launch

2. Robot configuration inside the src/property_based_tester/configuration/property_based_tester_params.yaml

Current available robots: bash Robot: robot_urdf_name: husky # URDF name robot_spawner_name: spawn_husky_controller.launch robot_size: [0.580, 0.430, 0.250] # x,y,z in meters

bash Robot: robot_urdf_name: jackal_robot_issac # URDF name robot_spawner_name: spawn_jackal_controller.launch robot_size: [0.990, 0.670, 0.390] # x,y,z in meters

bash Robot: robot_urdf_name: xarm6_gripper robot_spawner_name: spawn_xarm6_controller.launch

bash Robot: robot_urdf_name: rovo # URDF name robot_spawner_name: spawn_rovo_wo_controller.launch robot_controller: rovo_standalone_controller.launch robot_velocity: /rovo_velocity_controller/cmd_vel robot_size: [1.200, 1.230, 0.530] # x,y,z in meters

bash Robot: robot_urdf_name: b1 # URDF name robot_spawner_name: spawn_b1_wo_controller.launch robot_controller: b1_standalone_controller.launch robot_velocity: /b1/cmd_vel/smooth robot_size: [1.10, 0.45, 0.50] # x,y,z in meters

1. Launching their navigation drivers

bash roslaunch husky_navigation move_base_mapless_demo.launch bash roslaunch jackal_navigation odom_navigation_demo.launch bash roslaunch rovo_navi odom_navigation.launch bash roslaunch b1_gazebo qre_gazebo_nav.launch

Result generation (Allure)

• Run from inside the src/property_based_tester folder, this generates the results viewable from chrome

bash ./result_generation.sh

Collision sensor addition in Gazebo

1. Use the default template to add the senor plugin to your robot.

• Link and joint template: ``` <!-- Links and joints templplate if you do not already have -->

<!-- Change param parent link --> <!-- Change param same as link --> ```

• Sensor template: <!-- Contact Sensor --> <gazebo reference="sim_collider"> <!-- Change param same as link --> <sensor name="sim_collider_link_contact_sensor" type="contact"> <always_on>true</always_on> <update_rate>100</update_rate> <contact> <collision>base_link_fixed_joint_lump__sim_collider_collision_2</collision> <!-- Change param same as link --> </contact> <plugin filename="libgazebo_ros_bumper.so" name="gazebo_ros_sim_collider_controller"> <always_on>true</always_on> <update_rate>100</update_rate> <frameName>world</frameName> <!-- Change param --> <bumperTopicName>bumper_contact_state</bumperTopicName> </plugin> </sensor> <material>Gazebo/Orange</material> </gazebo>
• Convert your URDF to SDF as there are parsing issues in detecting collision name. bash gz sdf -p my.urdf > my.sdf
• Incase you have to convert your xacro urdf to URDF

bash rosrun xacro xacro -o model.urdf model.urdf.xacro

1. Search in the .sdf file the collision tag corresponding to your URDF and place it in the contact sensors collision tag (i.e. sim_collider).

Robot Test Definition Language RTDL (TextX)

• Run from inside the src/property_based_tester/robot_test_definition_language folder, generates a DSL

bash ./rtdl.sh

• Test definitions are written in src/property_based_tester/robot_test_definition_language/standard_test_definitions.pblg

• Used for feeding into hypothesis for generating tests

Running Framework (Hypothesis|ROS|Omniverse Nvidia IssacSim|Allure|TextX)

• (In the works!) Generate Universal Scene Description physics based simulation in ISSAC Sim.

bash ./omni.sh

Acknowledgements

Supervised by: - Prof. Dr. Nico Hochgeschwender - Prof. Dr. Paul G. Pl oger - M.Sc. Sven Schneider

Resource Provision:

• MYBOTSHOP

Software Requirements

• Ubuntu 20.04 LTS
• Python 3.6.12 64-bit
• Gazebo 7.16.1
• Catkin-pkg 0.4.22-100
• ROS-Noetic
• numpy 1.11.0
• numpy-stl
• cuda 11.0
• cuddnn 8
• tensorflow 1.4.0
• keras 2.0.8
• pandas 0.17.1
• termcolor 1.1.0
• pytest==4.6.11
• maven
• jdk 8
• allure-pytest==2.6.0
• allure==2.6.0
• torch==1.4.0
• torchvision==0.5.0
• allure-python-commons==2.6.0
• hypothesis--4.57.1
• Nvidia-Issac-Sim
• TextX

Required ROS Packages

• Husky
• Jackal
• xARM6
• Ouster
• Gazebo-pkgs
• MBS ROVO2 (Closed source)
• Quadruped B1 (Closed source)
• Yocs Velocity Smoother
• RoboticsGroup Gazebo Plugins

Hardware Requirements

These constitute the bare minimum requirements to run this package.

• 16 Gb RAM
• AMD Ryzen 5 5600H CPU @ 3.30GHz
• Nvidia GeForce RTX 3060
• 250 Gb SSD

Known Issues

---

The simulator bugging out can be caused by fast physics rendering. Turn the real time factor of the physics simulatior down to get better results.

Movebase not working

The movebase is not running. Make sure the topics are correct for the odom, map, and cmdvel. Also, make sure that simtime is enabled for movebasei.e.: ```bash <param name="usesim_time" value="true" /> ```

Ouster or other Gazebo plugins not working

Add this and place the plugins binaries and shared objects into this folder

bash export GAZEBO_PLUGIN_PATH=/home/sorox23/robotic_ws/master_thesis_ws/src/property_based_tester/plugins/gazebo_plugins_manual_amd64