isolation-forest

Table of contents

• Introduction
• Features
• Getting started
  • Building the library
  • Add an isolation-forest dependency to your project
• Usage examples
  • Model parameters
  • Training and scoring
  • Saving and loading a trained model
• ONNX conversion for portable inference
  • Converting a trained model to ONNX
  • Using the ONNX model for inference (example in Python)
• Performance and benchmarks
• Copyright and license
• Contributing
• References

Introduction

This is a distributed Scala/Spark implementation of the Isolation Forest unsupervised outlier detection algorithm. It features support for ONNX export for easy cross-platform inference. This library was created by James Verbus from the LinkedIn Anti-Abuse AI team.

Features

• Distributed training and scoring: The isolation-forest module supports distributed training and scoring in Scala using Spark data structures. It inherits from the Estimator and Model classes in Spark's ML library in order to take advantage of machinery such as Pipelines. Model persistence on HDFS is supported.
• Broad portability via ONNX: The isolation-forest-onnx module provides Python-based converter to convert a trained model to ONNX format for broad portability across platforms and languages. ONNX is an open format built to represent machine learning models.

Getting started

Building the library

To build using the default of Scala 2.13.14 and Spark 3.5.1, run the following:

bash ./gradlew build This will produce a jar file in the ./isolation-forest/build/libs/ directory.

If you want to use the library with arbitrary Spark and Scala versions, you can specify this when running the build command.

bash ./gradlew build -PsparkVersion=3.5.1 -PscalaVersion=2.13.14

To force a rebuild of the library, you can use: bash ./gradlew clean build --no-build-cache

To just run the tests: bash ./gradlew test

Add an isolation-forest dependency to your project

Please check Maven Central for the latest artifact versions.

Gradle example

The artifacts are available in Maven Central, so you can specify the Maven Central repository in the top-level build.gradle file.

repositories { mavenCentral() }

Add the isolation-forest dependency to the module-level build.gradle file. Here is an example for a recent spark scala version combination.

dependencies { compile 'com.linkedin.isolation-forest:isolation-forest_3.5.1_2.13:3.2.3' }

Maven example

If you are using the Maven Central repository, declare the isolation-forest dependency in your project's pom.xml file. Here is an example for a recent Spark/Scala version combination.

<dependency> <groupId>com.linkedin.isolation-forest</groupId> <artifactId>isolation-forest_3.5.1_2.13</artifactId> <version>3.2.3</version> </dependency>

Usage examples

Model parameters

| Parameter | Default Value | Description | |--------------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | numEstimators | 100 | The number of trees in the ensemble. | | maxSamples | 256 | The number of samples used to train each tree. If this value is between 0.0 and 1.0, then it is treated as a fraction. If it is >1.0, then it is treated as a count. | | contamination | 0.0 | The fraction of outliers in the training data set. If this is set to 0.0, it speeds up the training and all predicted labels will be false. The model and outlier scores are otherwise unaffected by this parameter. | | contaminationError | 0.0 | The error allowed when calculating the threshold required to achieve the specified contamination fraction. The default is 0.0, which forces an exact calculation of the threshold. The exact calculation is slow and can fail for large datasets. If there are issues with the exact calculation, a good choice for this parameter is often 1% of the specified contamination value. | | maxFeatures | 1.0 | The number of features used to train each tree. If this value is between 0.0 and 1.0, then it is treated as a fraction. If it is >1.0, then it is treated as a count. | | bootstrap | false | If true, draw sample for each tree with replacement. If false, do not sample with replacement. | | randomSeed | 1 | The seed used for the random number generator. | | featuresCol | "features" | The feature vector. This column must exist in the input DataFrame for training and scoring. | | predictionCol | "predictedLabel" | The predicted label. This column is appended to the input DataFrame upon scoring. | | scoreCol | "outlierScore" | The outlier score. This column is appended to the input DataFrame upon scoring. |

Training and scoring

Here is an example demonstrating how to import the library, create a new IsolationForest instance, set the model hyperparameters, train the model, and then score the training data. data is a Spark DataFrame with a column named features that contains a org.apache.spark.ml.linalg.Vector of the attributes to use for training. In this example, the DataFrame data also has a labels column; it is not used in the training process, but could be useful for model evaluation.

```scala import com.linkedin.relevance.isolationforest._ import org.apache.spark.ml.feature.VectorAssembler

/** * Load and prepare data */

// Dataset from http://odds.cs.stonybrook.edu/shuttle-dataset/ val rawData = spark.read .format("csv") .option("comment", "#") .option("header", "false") .option("inferSchema", "true") .load("isolation-forest/src/test/resources/shuttle.csv")

val cols = rawData.columns val labelCol = cols.last

val assembler = new VectorAssembler() .setInputCols(cols.slice(0, cols.length - 1)) .setOutputCol("features") val data = assembler .transform(rawData) .select(col("features"), col(labelCol).as("label"))

// scala> data.printSchema // root // |-- features: vector (nullable = true) // |-- label: integer (nullable = true)

/** * Train the model */

val contamination = 0.1 val isolationForest = new IsolationForest() .setNumEstimators(100) .setBootstrap(false) .setMaxSamples(256) .setMaxFeatures(1.0) .setFeaturesCol("features") .setPredictionCol("predictedLabel") .setScoreCol("outlierScore") .setContamination(contamination) .setContaminationError(0.01 * contamination) .setRandomSeed(1)

val isolationForestModel = isolationForest.fit(data)

/** * Score the training data */

val dataWithScores = isolationForestModel.transform(data)

// scala> dataWithScores.printSchema // root // |-- features: vector (nullable = true) // |-- label: integer (nullable = true) // |-- outlierScore: double (nullable = false) // |-- predictedLabel: double (nullable = false) ```

The output DataFrame, dataWithScores, is identical to the input data DataFrame but has two additional result columns appended with their names set via model parameters; in this case, these are named predictedLabel and outlierScore.

Saving and loading a trained model

Once you've trained an isolationForestModel instance as per the instructions above, you can use the following commands to save the model to HDFS and reload it as needed.

```scala val path = "/user/testuser/isolationForestWriteTest"

/** * Persist the trained model on disk */

// You can ensure you don't overwrite an existing model by removing .overwrite from this command isolationForestModel.write.overwrite.save(path)

/** * Load the saved model from disk */

val isolationForestModel2 = IsolationForestModel.load(path) ```

ONNX conversion for portable inference

Converting a trained model to ONNX

The artifacts associated with the isolation-forest-onnx module are available in PyPI.

The ONNX converter can be installed using pip. It is recommended to use the same version of the converter as the version of the isolation-forest library used to train the model.

bash pip install isolation-forest-onnx==3.2.7

You can then import and use the converter in Python.

```python import os from isolationforestonnx.isolationforestconverter import IsolationForestConverter

This is the same path used in the previous example showing how to save the model in Scala above.

path = '/user/testuser/isolationForestWriteTest'

Get model data path

datadirpath = path + '/data' avromodelfile = os.listdir(datadirpath) modelfilepath = datadirpath + '/' + avromodelfile[0]

Get model metadata file path

metadatadirpath = path + '/metadata' metadatafile = os.listdir(path + '/metadata/') metadatafilepath = metadatadirpath + '/' + metadatafile[0]

Convert the model to ONNX format (this will return the ONNX model in memory)

converter = IsolationForestConverter(modelfilepath, metadatafilepath) onnx_model = converter.convert()

Convert and save the model in ONNX format (this will save the ONNX model to disk)

onnxmodelpath = '/user/testuser/isolationForestWriteTest.onnx' converter.convertandsave(onnxmodelpath) ```

Using the ONNX model for inference (example in Python)

```python import numpy as np import onnx from onnxruntime import InferenceSession

onnx_model_path the same path used above in the convert and save operation

onnxmodelpath = '/user/testuser/isolationForestWriteTest.onnx' dataset_path = 'isolation-forest-onnx/test/resources/shuttle.csv'

Load data

inputdata = np.loadtxt(datasetpath, delimiter=',') numfeatures = inputdata.shape[1] - 1 lastcolindex = numfeatures print(f'Number of features for {datasetname}: {num_features}')

The last column is the label column

inputdict = {'features': np.delete(inputdata, lastcolindex, 1).astype(dtype=np.float32)} actuallabels = inputdata[:, lastcolindex]

Load the ONNX model from local disk and do inference

onx = onnx.load(onnxmodelpath) sess = InferenceSession(onx.SerializeToString()) res = sess.run(None, input_dict)

Print scores

actualoutlierscores = res[0] print('ONNX Converter outlier scores:') print(np.transpose(actualoutlierscores[:numexamplesto_print])[0]) ```

Performance and benchmarks

The original 2008 "Isolation forest" paper by Liu et al. published the AUROC results obtained by applying the algorithm to 12 benchmark outlier detection datasets. We applied our implementation of the isolation forest algorithm to the same 12 datasets using the same model parameter values used in the original paper. We used 10 trials per dataset each with a unique random seed and averaged the result. The quoted uncertainty is the one-sigma error on the mean.

| Dataset | Expected mean AUROC (from Liu et al.) | Observed mean AUROC (from this implementation) | |------------------------------------------------------------------------------------|---------------------------------------|------------------------------------------------| | Http (KDDCUP99) | 1.00 | 0.99973 ± 0.00007 | | ForestCover | 0.88 | 0.903 ± 0.005 | | Mulcross | 0.97 | 0.9926 ± 0.0006 | | Smtp (KDDCUP99) | 0.88 | 0.907 ± 0.001 | | Shuttle | 1.00 | 0.9974 ± 0.0014 | | Mammography | 0.86 | 0.8636 ± 0.0015 | | Annthyroid | 0.82 | 0.815 ± 0.006 | | Satellite | 0.71 | 0.709 ± 0.004 | | Pima | 0.67 | 0.651 ± 0.003 | | Breastw | 0.99 | 0.9862 ± 0.0003 | | Arrhythmia | 0.80 | 0.804 ± 0.002 | | Ionosphere | 0.85 | 0.8481 ± 0.0002 |

Our implementation provides AUROC values that are in very good agreement with the results in the original Liu et al. publication. There are a few very small discrepancies that are likely due to the limited precision of the AUROC values reported in Liu et al.

Copyright and license

Copyright 2019 LinkedIn Corporation All Rights Reserved.

Licensed under the BSD 2-Clause License (the "License"). See License in the project root for license information.

Contributing

If you would like to contribute to this project, please review the instructions here.

References

• F. T. Liu, K. M. Ting, and Z.-H. Zhou, Isolation forest, in 2008 Eighth IEEE International Conference on Data Mining, 2008, pp. 413422.
• F. T. Liu, K. M. Ting, and Z.-H. Zhou, Isolation-based anomaly detection, ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 6, no. 1, p. 3, 2012.
• Shebuti Rayana (2016). ODDS Library [http://odds.cs.stonybrook.edu]. Stony Brook, NY: Stony Brook University, Department of Computer Science.