Microsoft Malware Classification

Introduction

In recent years, the malware industry has become a well organized market involving large amounts of money. Well funded, multi-player syndicates invest heavily in technologies and capabilities built to evade traditional protection, requiring anti-malware vendors to develop counter mechanisms for finding and deactivating them. One of the major challenges that anti-malware faces today is the vast amounts of data and files which need to be evaluated for potential malicious intent.

The goal of this project is to train a malware classifier using machine learning that can separate malicious samples into different families with high accuracy and efficiency, such as Virus, Worm, and Trojan.

Dataset

The dataset is from the 2015 Microsoft Malware Classification Challenge. It contains 10868 malware samples representing a mix of nine families.

| Name | # Samples | Type | | :--------------: | :-------: | :----------------: | | Ramnit | 1541 | Worm | | Lollipop | 2478 | Adware | | Kelihos_ver3 | 2942 | Backdoor | | Vundo | 475 | Trojan | | Simda | 42 | Backdoor | | Tracur | 751 | Trojan Downloader | | Kelihos_ver1 | 398 | Backdoor | | Obfuscator.ACY | 1228 | Obfuscated malware | | Gatak | 1013 | Backdoor |

Each sample has two files of different forms: machine code and disassembly script generated by IDA Pro.

10001100 C4 01 74 AC D9 EE D9 C0 DD EA DF E0 F6 C4 44 7A 10001110 0A DD D9 DD 17 DD 1E 8B E5 5D C3 DD D8 E8 74 26 10001120 00 00 DC 0D 78 65 00 10 DC 34 24 E8 60 26 00 00 10001130 DD 44 24 08 D8 C9 DD 1F DC 4C 24 10 DD 1E 8B E5 10001140 5D C3 CC CC CC CC CC CC CC CC CC CC CC CC CC CC

.text:10001106 D9 C0 fld st .text:10001108 DD EA fucomp st(2) .text:1000110A DF E0 fnstsw ax .text:1000110C F6 C4 44 test ah, 44h .text:1000110F 7A 0A jp short loc_1000111B .text:10001111 DD D9 fstp st(1) .text:10001113 DD 17 fst qword ptr [edi]

The data folder only contains several sample files.

Precomputed Features

| Item | Description | | :-------------------: | :----------------------------------------------------------: | | File Size | The sizes of disassembly and machine code files, and their ratios. | | API 4-gram | API sequences. | | Opcode 4-gram | Opcode sequences. | | Import Library | Import libraries in the PE Import Table. | | PE Section Size | The virtual sizes and raw sizes of PE sections, and their ratios. | | PE Section Permission | The total sizes of readable data, writable data and executable code. | | Content Complexity | The original sizes, compressed sizes and compression ratios of disassembly and machine code files. |

Models

These models have been tested in the project:

• Support Vector
• K-Nearest Neighbors
• Random Forest

IDA Pro Classification Plug-in

Unlike other machine learning applications like hand written digit classification, where the shape of numbers is not updated over time, the similarity between previous and future malware will degrade over time due to function updates and polymorphic techniques. Polymorphic techniques can automatically and frequently change identifiable characteristics like encryption types and code distribution to make malware unrecognizable to anti-virus detection.

To solve this, we designed an automatic malware classification workflow to apply and enhance our classifier in practice with IDA Pro's Python development kit.

Getting Started

Copy all the files in the ida-plugin folder except tpot_exported_ida_pipeline.py to IDA Pro's plugins folder.

• generator.py

A generator for generating disassembly and machine code files for an executable sample, relying on IDA Pro's disassembler. These two output files are in the similar format as the files in the dataset.

• generation_cmd.py

A wrapper for generator.py. It can be run under IDA Pro's autonomous mode.

• Generate-Samples.ps1

It uses generation_cmd.py and processes a directory containing executable samples.

• tpot_exported_ida_pipeline.py

The best machine learning pipeline produced by TPOT. It is just a reference.

• tpot_exported_ida_clf.joblib

A fitted machine learning model using the pipeline in tpot_exported_ida_pipeline.py.

• classifier.py

An IDA Pro classification plug-in. When an analyst open a sample with IDA Pro, it can calculate the probability that the sample belongs to each malware family.

console 0.53 -> Ramnit 0.24 -> Lollipop 0.17 -> Obfuscator.ACY 0.05 -> Gatak 0.01 -> Simda 0.01 -> Vundo 0.00 -> Tracur 0.00 -> Kelihos_ver1 0.00 -> Kelihos_ver3

Manual Data Generation

Before the automatic classification, the plug-in can produce disassembly and machine code files by itself. But you can also manually produce them using generation_cmd.py and Generate-Samples.ps1.

generation_cmd.py can be run from the command line with IDA Pro's parameters -A and -S, which launch IDA Pro in autonomous mode and make it run a script.

bash ida -A "-S<full-path-to-generation_cmd.py>" <binary-file>

Note that there is no space between -S and the path.

Or use Generate-Samples.ps1 to process a directory containing executable samples.

powershell Generate-Samples.ps1 -InputDirectory <input-folder> -OutputDirectory <output-folder>

Dependences

• NumPy
• pandas
• SciPy
• scikit-learn
• auto-sklearn
• TPOT
• Matplotlib
• dtreeviz
• seaborn
• joblib
• tqdm

License

Distributed under the MIT License. See LICENSE for more information.

Citing

• arXiv

tex @misc{chen-2021:static-disasm-malware, title = {Malware Classification Using Static Disassembly and Machine Learning}, author = {Zhenshuo Chen and Eoin Brophy and Tomas Ward}, year = 2021, eprint = {2201.07649}, archiveprefix = {arXiv}, primaryclass = {cs.CR} }

• CEUR-WS

```tex @inproceedings{chen-2022:static-disasm-malware, title = {Malware Classification Using Static Disassembly and Machine Learning}, author = {Zhenshuo Chen and Eoin Brophy and Tomas Ward}, pages = {48--59}, url = {http://ceur-ws.org/Vol-3105/paper8.pdf}, crossref = {AICS2021} }

@proceedings{AICS2021, title = {The 29th Irish Conference on Artificial Intelligence and Cognitive Science 2021}, year = 2021, booktitle = {The 29th Irish Conference on Artificial Intelligence and Cognitive Science 2021}, address = {Aachen}, series = {CEUR Workshop Proceedings}, number = 3105, issn = {1613-0073}, url = {http://ceur-ws.org/Vol-3105}, editor = {Arjun Pakrashi and Ellen Rushe and Mehran Bazargani and Mac Namee, Brian}, venue = {Dublin, Republic of Ireland}, eventdate = {2021-12-09} } ```