stars-carla-experiments

This repository analyzes driving data recorded with the Carla Simulator using the STARS framework.

https://github.com/tudo-aqua/stars-carla-experiments

Science Score: 77.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • CITATION.cff file
    Found CITATION.cff file
  • codemeta.json file
    Found codemeta.json file
  • .zenodo.json file
    Found .zenodo.json file
  • DOI references
    Found 2 DOI reference(s) in README
  • Academic publication links
    Links to: zenodo.org
  • Committers with academic emails
    1 of 1 committers (100.0%) from academic institutions
  • Institutional organization owner
  • JOSS paper metadata
  • Scientific vocabulary similarity
    Low similarity (10.6%) to scientific vocabulary

Keywords

autonomous-driving autonomous-vehicles carla-simulator classification stars
Last synced: 4 months ago · JSON representation ·

Repository

This repository analyzes driving data recorded with the Carla Simulator using the STARS framework.

Basic Info
  • Host: GitHub
  • Owner: tudo-aqua
  • License: apache-2.0
  • Language: Kotlin
  • Default Branch: main
  • Homepage:
  • Size: 13.5 MB
Statistics
  • Stars: 5
  • Watchers: 5
  • Forks: 5
  • Open Issues: 5
  • Releases: 7
Topics
autonomous-driving autonomous-vehicles carla-simulator classification stars
Created over 2 years ago · Last pushed 5 months ago
Metadata Files
Readme License Citation

README.md

STARS experiments based on data from CARLA

This repository analyzes driving data recorded with the Carla Simulator using the STARS framework. The Carla data was recorded using the stars-export-carla Repository.

Setup

The analysis requires the recorded data. To receive the data, there are two options: 1. Set DOWNLOAD_EXPERIMENTS_DATA in experimentsConfiguration.kt to true. This will automatically download and unzip the necessary data. 2. Manually download the data. 1. Go to the Zenodo artifact where the experiments data is stored 2. Download the stars-reproduction-source.zip 3. Place the Zip-File into the root folder of this project.

Remark: The downloaded data has a size of approximately 1.3GB. The downloaded zip-file will be extracted during the analysis. Make sure, that you have at least 3GB of free space.

Running the Analysis

This project is a Gradle project with a shipped gradle wrapper. To execute the analysis simply execute:

  • Linux/Mac: ./gradlew run
  • Windows: ./gradlew.bat run

Additional Arguments

The project provides the possibility of adjusting the analysis and its environment. The following arguments are provided:

  • --help: Returns a list of all available arguments.
  • --input (String): Points to the directory of the input files that should be analyzed.
  • --allEgo (Boolean): Sets, whether every vehicle should be treated as ego and therefore increasing the analyzed data by the amount of available vehicles.
  • --minSegmentTicks (Int): Sets the minimum amount of ticks that have to be present for a segment to be analyzed.
  • --sorted (Boolean): Sets, whether the input files should be sorted be the seed in the files name.
  • --dynamicFilter (String): A regex filter that is applied to all dynamic files effectively shrinking the analysis set.
  • --staticFilter (String): A regex filter that is applied to all static files effectively shrinking the analysis set.
  • --ignore (String, separated by ','): A string list filter that is applied to all TSC projections. Make sure to remove all unnecessary white spaces, as they propagate to the final ignore list.

Each argument has a default value to recreate the experiment data from the linked paper in which the results of this repository are used.

Arguments can be passed to the analysis with the following command structure:

  • Linux/Mac: ./gradlew run --args="--allEgo --minSegmentTicks=20 --sorted --dynamicFilter='seed18' --staticFilter='Town01' --ignore='all,static'"
  • Windows: ./gradlew.bat run --args="--allEgo --minSegmentTicks=20 --sorted --dynamicFilter='seed18' --staticFilter='Town01' --ignore='all,static'"

Problem Solving

If you are running into OutOfMemory exceptions, you can add the following statement to the application{} block in your build.gradle.kts file to adjust the available heap size for the application:

applicationDefaultJvmArgs = listOf("-Xmx12g", "-Xms2g")

The example above will set the max heap size to 12GB (-Xmx12g) and the initial heap size to 2GB (-Xms2g).

Analysis Results

After the analysis is finished you can find the results in the analysis-result-logs subfolder which will be created during the analysis.

For each execution of the analysis pipeline a subfolder with the start date and time ist created. In it, each metric of the analysis has its own subfolder. The analysis separates the results of each metric into different categories with different detail levels. - *-severe.txt lists all failed metric results - *-warning.txt lists all warnings that occurred during analysis - *-info.txt contains the summarized result of the metric - *-fine.txt contains a more detailed result of the metric - *-finer.txt contains all possible results of the metric - *-finest.txt contains all possible results of the metric including meta information

IntelliJ Analysis Helper

Intellij can (by default) only handle file up to 2.6MB. As the result file are in many cases larger than this threshold, you can increase the size of files that should be analyzed by IntelliJ. You have to restart IntelliJ for the changes to take effect.

```

custom IntelliJ IDEA properties (expand/override 'bin\idea.properties')

idea.max.intellisense.filesize=999999 ```

(Optional) Git Hooks

If you want to use our proposed Git Hooks you can execute the following command: shell git config --local core.hooksPath .githooks

Owner

  • Name: AQUA - Automated Quality Assurance
  • Login: tudo-aqua
  • Kind: organization

Citation (CITATION.cff)

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: >-
  Tree-Based Scenario Classification - A Formal Framework for Measuring Domain Coverage When Testing Autonomous Systems
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Till
    family-names: Schallau
    email: till.schallau@tu-dortmund.de
    affiliation: TU Dortmund University
    orcid: 'https://orcid.org/0000-0002-1769-3486'
  - given-names: Stefan
    family-names: Naujokat
    email: stefan.naujokat@tu-dortmund.de
    affiliation: TU Dortmund University
    orcid: 'https://orcid.org/0000-0002-6265-6641'
  - given-names: Fiona
    family-names: Kullmann
    email: fiona.kullmann@tu-dortmund.de
    affiliation: TU Dortmund University
    orcid: 'https://orcid.org/0000-0001-5858-0659'
  - given-names: Falk
    family-names: Howar
    email: falk.howar@tu-dortmund.de
    affiliation: 'TU Dortmund University, Fraunhofer ISST'
    orcid: 'https://orcid.org/0000-0002-9524-4459'
repository-code: 'https://github.com/tudo-aqua/stars'
abstract: >-
    Scenario-based testing is envisioned as a key approach for the
    safety assurance of automated driving systems. In scenario-based testing,
    relevant (driving) scenarios are the basis of tests. Many recent works
    focus on specification, variation, generation, and execution of individual
    scenarios. In this work, we address the open challenges of classifying
    sets of recorded test drives into such scenarios and measuring scenario
    coverage in these test drives. Technically, we specify features in logic
    formulas over complex data streams and construct tree-based classifiers
    for scenarios from these feature specifications. For such specifications,
    we introduce CMFTBL, a new logic that extends existing linear-time
    temporal logics with aspects that are essential for concise specifications
    that work on field-recorded data. We demonstrate the expressiveness
    and effectiveness of our approach by defining a family of related scenario
    classifiers for different aspects of urban driving.
keywords:
  - automated driving systems
  - temporal logic
  - scenario classification
  - autonomous-driving
  - carla
license: Apache-2.0
preferred-citation:
  authors:
    - given-names: Till
      family-names: Schallau
      email: till.schallau@tu-dortmund.de
      affiliation: TU Dortmund University
      orcid: 'https://orcid.org/0000-0002-1769-3486'
    - given-names: Stefan
      family-names: Naujokat
      email: stefan.naujokat@tu-dortmund.de
      affiliation: TU Dortmund University
      orcid: 'https://orcid.org/0000-0002-6265-6641'
    - given-names: Fiona
      family-names: Kullmann
      email: fiona.kullmann@tu-dortmund.de
      affiliation: TU Dortmund University
      orcid: 'https://orcid.org/0000-0001-5858-0659'
    - given-names: Falk
      family-names: Howar
      email: falk.howar@tu-dortmund.de
      affiliation: 'TU Dortmund University, Fraunhofer ISST'
      orcid: 'https://orcid.org/0000-0002-9524-4459'
  title: >-
    Tree-Based Scenario Classification - A Formal Framework for Measuring Domain Coverage When Testing Autonomous Systems
  type: conference-paper
  year: 2024
  doi: '10.1007/978-3-031-60698-4_15'
  collection-title: NASA Formal Methods Symposium
  publisher:
    name: Springer Nature Switzerland
  pages: 259-278
  isbn: '978-3-031-60697-7'
  editors:
    - family-names: Nathaniel
      given-names: Benz
    - family-names: Divya
      given-names: Gopinath
    - family-names: Nija
      given-names: Shi

GitHub Events

Total
  • Create event: 13
  • Release event: 1
  • Issues event: 2
  • Watch event: 1
  • Delete event: 7
  • Issue comment event: 2
  • Push event: 34
  • Pull request review event: 4
  • Pull request event: 20
  • Fork event: 4
Last Year
  • Create event: 13
  • Release event: 1
  • Issues event: 2
  • Watch event: 1
  • Delete event: 7
  • Issue comment event: 2
  • Push event: 34
  • Pull request review event: 4
  • Pull request event: 20
  • Fork event: 4

Committers

Last synced: almost 2 years ago

All Time
  • Total Commits: 16
  • Total Committers: 1
  • Avg Commits per committer: 16.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 16
  • Committers: 1
  • Avg Commits per committer: 16.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Till Schallau t****u@t****e 16
Committer Domains (Top 20 + Academic)

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 6
  • Total pull requests: 20
  • Average time to close issues: about 1 month
  • Average time to close pull requests: 11 days
  • Total issue authors: 2
  • Total pull request authors: 3
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.2
  • Merged pull requests: 12
  • Bot issues: 0
  • Bot pull requests: 14
Past Year
  • Issues: 3
  • Pull requests: 13
  • Average time to close issues: N/A
  • Average time to close pull requests: 21 days
  • Issue authors: 2
  • Pull request authors: 2
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.15
  • Merged pull requests: 6
  • Bot issues: 0
  • Bot pull requests: 11
Top Authors
Issue Authors
  • tillschallau (5)
  • dominikmaeckel (1)
Pull Request Authors
  • dependabot[bot] (16)
  • tillschallau (8)
  • dominikmaeckel (4)
Top Labels
Issue Labels
documentation (2) bug (2) enhancement (1)
Pull Request Labels
dependencies (16) java (11) documentation (3) deploy (2) enhancement (2)

Dependencies

build.gradle.kts maven
  • tools.aqua:stars-core $starsVersion implementation
  • tools.aqua:stars-data-av $starsVersion implementation
  • tools.aqua:stars-import-carla $starsVersion implementation
  • tools.aqua:stars-logic-kcmftbl $starsVersion implementation