Data

Tools

Indexes

Applications

Experiments

Open Source Science

cbombench

A testbed for benchmarking Cryptographic Bills of Materials (CBOMs).

https://github.com/seg-unibe/cbombench

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 2 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.3%) to scientific vocabulary

Keywords

benchmark cbom python
Last synced: 6 months ago · JSON representation ·

Repository

A testbed for benchmarking Cryptographic Bills of Materials (CBOMs).

Basic Info
  • Host: GitHub
  • Owner: SEG-UNIBE
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 528 KB
Statistics
  • Stars: 2
  • Watchers: 0
  • Forks: 0
  • Open Issues: 0
  • Releases: 3
Topics
benchmark cbom python
Created 7 months ago · Last pushed 6 months ago
Metadata Files
Readme License Citation

README.md

Changelog License DOI

CBOMbench

classdiagram

CBOMbench is a modular benchmarking framework for evaluating Cryptographic Bill of Materials (CBOM) generation tools. It automates testing of CBOM tools across real-world GitHub repositories and produces analysis based on output quality and performance metrics.

This tool was developed as part of a bachelor’s thesis at the University of Bern and currently supports benchmarking of: - IBM's CBOMkit - CycloneDX's cdxgen - DeepSeek for LLM-based CBOM-generation

Installation & Setup

To run CBOMbench, follow these steps:

1. Install Required Tools

  • Follow the official instructions to install CBOMkit (via Docker or Podman).
  • Install cdxgen using npm: npm install -g @cyclonedx/cdxgen

2. Clone this Repository

bash git clone https://github.com/SEG-UNIBE/cbombench.git cd cbombench

3. Install Python Dependencies

bash pip install -r requirements.txt

4. Set Environment Variables

bash export DEEPSEEK_API_KEY=your_deepseek_key_here export GITHUB_TOKEN=your_github_token_here

Usage

CBOMbench is operated through the command-line: python ./src/cbombench.py [COMMAND]

Available Commands:

  • Repository Discovery and Selection

    Get Github Repositories filtered by main programming language, size constraints in KB and how big the sample should be.

    cbombench.py get-repos --languag java --min-size 1000 --max-size 100000 --sample-size 10

  • Individual Tool Testing

    Individual tool testing for a specified repository and branch (if branch is left empty CBOMbench automatically detects the default branch).

    cbombench.py text cbomkit cdxgen deepseek https://github.com/example/repo --branch main

  • Automated Benchmarking

    Benchmarking the specified tools on a sample of GitHub repositories

    cbombench.py benchmark cbomkit cdxgen deepseek --language java --sample-size 10

  • Statistical Analysis and Reporting

    Generate comparison reports from the collected CBOMs (save flag to save report in files).

    cbombench.py analyze --save

  • Historical Analysis

    Load a past analysis

    cbombench.py load-analysis

  • Data Cleanup

    Delete generated CBOMs

    cbombench.py delete-data

Extending CBOMbench

To add support for a new CBOM generation tool: 1. Create a new adapter 2. Implement a generate_cbom(url, branch) method that returns the CBOM as a JSON and a duration 3. Add the function to cbombench.py

Owner

  • Name: SEG UNIBE
  • Login: SEG-UNIBE
  • Kind: organization

Software Engineering Group, University of Bern, Switzerland

Citation (CITATION.cff) 

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: CBOMbench
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Simon
    family-names: Egli
    email: simon.egli4@students.unibe.ch
    affiliation: University of Bern
  - given-names: Roman
    family-names: Bögli
    email: roman.boegli@unibe.ch
    affiliation: University of Bern
    orcid: "https://orcid.org/0009-0004-8745-7800"
  - given-names: Timo
    family-names: Kehrer
    email: timo.kehrer@unibe.ch
    affiliation: University of Bern
    orcid: "https://orcid.org/0000-0002-2582-5557"
repository-code: "https://github.com/SEG-UNIBE/cbombench"
abstract: A testbed for benchmarking Cryptographic Bills of Materials (CBOMs).
keywords:
  - python
  - benchmark
  - cbom
license: Apache-2.0

GitHub Events

Total
  • Release event: 1
  • Watch event: 1
  • Member event: 1
  • Push event: 10
  • Create event: 1
Last Year
  • Release event: 1
  • Watch event: 1
  • Member event: 1
  • Push event: 10
  • Create event: 1

Dependencies

requirements.txt pypi
  • click *
  • datetime *
  • matplotlib *
  • openai *
  • pandas *
  • podman-compose *
  • requests *
  • seaborn *
  • tabulate *
  • tqdm *
  • websocket-client *