IPAL - Datasets

This repository is part of IPAL - an Industrial Protocol Abstraction Layer. IPAL aims to establish an abstract representation of industrial network traffic for subsequent unified and protocol-independent industrial intrusion detection. IPAL consists of a transcriber to automatically translate industrial traffic into the IPAL representation, an IDS Framework implementing various industrial intrusion detection systems (IIDSs), and a collection of evaluation datasets. For details about IPAL, please refer to our publications listed down below.

This repository contains a collection of datasets for evaluating industrial IDS. Therefore, this repository contains scripts to convert (transcribe) existing datasets into IPAL format. It does not contain the raw datasets nor the datasets transcribed into IPAL. We merely use placeholders which can be replaced after obtaining the original datasets at the respective publishers (see link in the table below).

| Dataset | Type | Notes | Link | | ----------------------- | ------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | BATADAL | State | Dataset from the BATtle of the Attack Detection ALgorithms against a Water Distribution Sytem | BATADAL | | ELEGANT | Packet (Modbus) | The ELEGANT dataset consists of a MiTM and a DoS part. Until now we consider only the MiTM dataset and not the DoS dataset. | IEEE Dataport | | Electra | Packet (Modbus, S7) | Not all IPAL features are present, e.g., crc or length are missing. Also the request data/address fields are not always correct. We skip few duplicated packets. | Webseite | | Energy Dataset | Packet (IEC-104) | A short PCAP of the WATTSON simulator from Fraunhofer FKIE. We use the manipulateTraces tool from the DTMC IDS paper to add attacks to the WATTSON PCAP. | Paper, manipulateTraces DTMC Paper | | GeekLounge | Packet (S7) | The dataset does not contain any attacks. We added attacks according to the description of a paper. This results in 6 datasets with 3 attacks types each on requests and responses of S7 packets. | Website , Paper | | HAI | State | Dataset contains three training and five test files. Train and test are not in linear time order and have overlapping time-regions. | Github | | IEC61850SecurityDataset | Packet (Goose) | | Github | | Lemay | Packet (Modbus) | Most attacks are not performed with Modbus and use different protocols not relevant for the transcriber. | Paper Github | | MorrisDS1 | State | There exist different versions of the datset (binary, ternary, or multiclass labels). We use the multi-class dataset. | Website | | MorrisDS4 | Packet (Modbus) | There are minor differences between the Raw and Arff dataset. These differences affect only the attack packets. Default: Use the Arff dataset. | Website | | PowerDuck | Packet (GOOSE) | | Paper | | QUT_DNP3 | Packet (DNP3, GOOSE) | | Git Thesis | | QUT_S7_Myers | Packet (S7). | TODO: Check Rules | Dataset Paper | | QUT_S7comm | Packet (S7) | | Dataset Paper | | Sherlock (v1) | State (and IEC-104 | | Contains three differently sized scenarios of power grids. | Website Paper | | SWaT | State | Attack dataset has a 81s gap which we fill with the previous state. The first 1800s are often skipped in literature. The version 0 of SWaT has a slightly different start of the training data. | iTrust | | TEP-PASAD | State | The dataset consists of 5 different scenarios. Each scenario has its own training and test part combined in one single file. | Github | | WADI | State | WADI has a large gap in the training data of ~73h. Note: we use the row number as index for the timestamp since WADI has a challenging time notation. | iTrust | | WDT | Packet & State (Modbus) | | Paper |

Publications

• Konrad Wolsing, Eric Wagner, Antoine Saillard, and Martin Henze. 2022. IPAL: Breaking up Silos of Protocol-dependent and Domain-specific Industrial Intrusion Detection Systems. In 25th International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2022), October 26–28, 2022, Limassol, Cyprus. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3545948.3545968
• Wolsing, Konrad, Eric Wagner, and Martin Henze. "Poster: Facilitating Protocol-independent Industrial Intrusion Detection Systems." Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. 2020 https://doi.org/10.1145/3372297.3420019

Getting started

If you are new to IPAL and want to learn about the general idea or try out our tutorials, please refer to IPAL's main repository: https://github.com/fkie-cad/ipal.

Prerequisites

Transcribing the datasets requires the ipal-transcriber and tshark to be installed (see IPAL - Transcriber and https://tshark.dev/setup/install/).

On certain operating systems running all available scripts might require additional dependencies. Ensure that the following commands are available: - pv - gzip and gunzip from the gzip project or an alternative implementation with similar features - bash from the Bash project

Install

• After cloning the repository, initialise Git's submodules with git submodule init and git submodule update

• To transcribe a dataset into IPAL, one needs to obtain copy of the original datasets, e.g., from the source listed in table above. This dataset needs to be placed under [dataset-name]/raw/.

• Use the transcribe.sh or transcribe.py scripts to convert the dataset into IPAL. The dataset will be exported to [datset-name]/ipal.

Development

Tooling

The set of tools used for development, code formatting, style checking, and testing can be installed with the following command:

bash python3 -m pip install -r requirements-dev.txt

All tools can be executed manually with the following commands and report errors if encountered:

bash black . flake8 python3 -m pytest

A black and flake8 check of modified files before any commit can also be forced using Git's pre-commit hook functionality:

bash pre-commit install

More information on the black and flake8 setup can be found at https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/

Contributors

• Konrad Wolsing (Fraunhofer FKIE & RWTH Aachen University)
• Sven Zemanek (Fraunhofer FKIE)
• Dominik Kus (RWTH Aachen University)

License

MIT License. See LICENSE for details.