ROSARUM: a novel backdoor detection benchmark

About

The ROSARUM backdoor detection benchmark contains a series of backdoored programs which can be used to evaluate software backdoor detection methods.

Each benchmark comes in three flavors:

• safe: no backdoor exists in the program (to test the detection method's precision)
• backdoored: one or more backdoors exist in the program (to test the detection method's recall)
• ground-truth: the same backdoors exist as in the backdoored version, except that every time they're hit they print something in stderr to identify themselves (such as ***BACKDOOR TRIGGERED***).

The ground-truth versions can be used to perform a precise evaluation of the precision and recall of a given detection method.

The benchmarks are also split into two large categories:

• authentic: real backdoors found in the wild
• synthetic: fake backdoors injected in (hopefully) backdoor-safe software

Benchmark summary

Authentic backdoor benchmarks

| Name | Backdoor description | | ------------------ | -------------------------------------------------------------- | | Belkin | HTTP request with secret URL value leads to web shell | | D-Link | HTTP request with secret field value bypasses authentication | | Linksys | Packet with specific payload enables memory read/write | | Tenda | Packet with specific payload enables command execution | | PHP | HTTP request with secret field value enables command execution | | ProFTPD | Secret FTP command leads to root shell | | vsFTPd | FTP usernames containing ":)" lead to root shell |

Synthetic backdoor benchmarks

| Name | Backdoor description | | ---------------------- | -------------------------------------------------------------------- | | sudo | Hardcoded credentials bypass authentication | | libpng | Secret image metadata values enable command execution | | libsndfile | Secret sound file metadata value triggers home directory encryption | | libtiff | Secret image metadata value enables command execution | | libxml2 | Secret XML node format enables command execution | | Lua | Specific string values in script enable reading from filesystem | | OpenSSL | Secret bignum exponentiation string enables command execution | | PHP | Specific string values in serialized object enable command execution | | Poppler | Secret comment character in PDF enables command execution | | SQLite3 | Secret SQL keyword enables removal of home directory |

Installation

Docker

We highly recommend using ROSARUM in a Docker container, since some backdoors may carry payloads that can affect your machine (e.g., by removing the /home/ directory).

You can simply pull the existing ROSARUM Docker image by running:

console $ docker pull plumtrie/rosarum:latest

Then, you can run a container using that image by running:

console $ docker run -ti --rm plumtrie/rosarum:latest

(Note that this command will start an interactive session within the container, and that exiting the container will trigger its removal.)

Building the Docker image

If you wish to build the Docker image on your machine, you can use the helper build.sh script, which will automatically tag the image with the current version. See the script itself for more information.

Before running the script (or simply docker build ...), make sure that you have cloned all of the submodules used in this repo. You can do this either by cloning the repo with --recurse-submodules, or by running git submodule update --init post-cloning.

Be advised that the build might take some time (it takes ~12 minutes on a laptop with a 20-core 12th Gen Intel(R) Core(TM) i7-12800H CPU).

Once the Docker image is built, the run.sh convenience script may be used to run it. Generally, released versions of the image will be tagged, so you can run git checkout <TAG> and run ./build.sh and ./run.sh to build and run a specific version of the image.

Building from source

WARNING: running the target programs in a native, unprotected environment may endanger the state of your machine. We highly recommend using a Docker container as described above.

You should be able to build all of the target programs on a modern Unix system (the builds have not been tested outside that environment). However, you first need to install a number of dependencies; you can find the full list of dependencies in the Dockerfile.

Once you have installed the dependencies, you should be able to build any target program, with different levels of granularity. To build all variants of all target programs, you can run (from the targets directory):

console $ make

To build all variants of an entire category of target programs (e.g., authentic), you can run (from the targets directory):

console $ make authentic

To build all variants of a specific target program (e.g., Sudo), you can run (from the targets directory):

console $ make sudo-1.9.15p5

To build a specific variant (e.g., ground-truth) of a specific target program (e.g., Sudo), you can run (from the target program's root directory, e.g., targets/synthetic/sudo-1.9.15p5):

console $ make ground-truth

Usage

Reproducing the backdoors

Instructions on how to run all of the variants can be found in the root directory of each backdoor sample.

Generally, for each sample, you'll want to first build it (if it's not built):

console $ make # or `make <type>`, where `<type>` is `safe`, `backdoored` or `ground-truth`

Then, you need to perform any additional setup that may be needed (e.g., copying files to specific directories):

console $ make setup

Once you're done with the target program, to make sure other programs are not affected, you should undo the setup:

console $ make teardown

Evaluating a backdoor detection method on ROSARUM

If you want to evaluate a backdoor detection method, you can run it on the backdoor variants and evaluate the results on the ground-truth variants, by inspecting stderr for the ***BACKDOOR TRIGGERED*** marker.

For instance, let us assume that your backdoor detection tool is used on ./targets/synthetic/sudo-1.9.15p5/backdoored/build/bin/sudo (note the use of the backdoored variant) and produces backdoor-triggering inputs in the sudo-findings/ directory. For example, this simple Bash script goes through the findings (inputs to the target program) and prints the name of the finding file along with the result of the evaluation (true/false positive):

bash for finding in $(ls sudo-findings) do # Note the use of the _ground-truth_ variant here. ./targets/synthetic/sudo-1.9.15p5/ground-truth/build/bin/sudo -Sk -- id 2>&1 \ < sudo-findings/$finding \ | grep "\*\*\*BACKDOOR TRIGGERED\*\*\*" >/dev/null \ && echo "$finding: true positive" \ || echo "$finding: false positive" done

Contributing

Please read CONTRIBUTING.md.

Citing this repo

When citing the associated ICSE'25 paper, use the following snippet:

bibtex @inproceedings{kokkonis-2025-rosa, author = {Kokkonis, Dimitri and Marcozzi, Michaël and Decoux, Emilien and Zacchiroli, Stefano}, booktitle = {2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)}, title = {ROSA: Finding Backdoors with Fuzzing}, year = {2025}, volume = {}, number = {}, pages = {2816-2828}, keywords = {Runtime;Automation;Manuals;Binary codes;Fuzzing;Benchmark testing;Robustness;Software;Performance analysis;Standards;fuzzing;dynamic analysis;metamorphic testing;backdoors;vulnerability detection}, doi = {10.1109/ICSE55347.2025.00183}, }

When citing the actual repository/dataset itself, use CITATION.cff.