Atlantic - Automated Data Preprocessing Framework for Supervised Machine Learning

Framework Contextualization

The Atlantic project constitutes an comprehensive and objective approach to simplify and automate data processing through the integration and objectively validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows a evaluation structured in tree based models ensembles.

This project aims at providing the following application capabilities:

• General applicability on tabular datasets: The developed preprocessing procedures are applicable on multiple domains associated with Supervised Machine Learning, regardless of the properties or specifications of the data.

• Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated data processing based on the characteristics of the data input columns.

• Robustness and improvement of predictive results: The implementation of the atlantic automated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the data properties.

Main Development Tools

Major frameworks used to built this project:

• H2O.ai
• Scikit-learn
• XGBoost
• Optuna
• Pandas

Framework Architecture

Where to get it

Binary installer for the latest released version is available at the Python Package Index (PyPI).

Installation

To install this package from Pypi repository run the following command:

pip install atlantic

Usage Examples

1. Atlantic - Automated Data Preprocessing Pipeline

In order to be able to apply the automated preprocessing atlantic pipeline you need first to import the package. The following needed step is to load a dataset, split it and define your to be predicted target column name into the variable target. You can customize the fit_processing method by altering the following running pipeline parameters: * splitratio: Division ratio (Train\Validation) in which the preprocessing methods will be evaluated within the loaded Dataset. * relevance: Minimal value of the total sum of relative feature importance percentage selected in the H2O AutoML feature selection step. * h2ofsmodels: Quantity of models generated for competition in step H2O AutoML feature selection to evaluate the relative importance of each feature (only leaderboard model is selected for evaluation). * encodingfs: You can choose if you want to enconde categorical features in order to reduce loading time in H2O AutoML feature selection step. * vif_ratio: This value defines the minimal threshold for Variance Inflation Factor filtering (default value=10).

Once the data fitting process is complete, you can automaticaly optimize preprocessing transformations on all future dataframes with the same properties using the data_processing method.

```py import pandas as pd from sklearn.modelselection import traintest_split from atlantic.pipeline import Atlantic import warnings warnings.filterwarnings("ignore", category=Warning) # -> For a clean console

data = pd.readcsv('csvdirectory_path', encoding='latin', delimiter=',') # Dataframe Loading Example

data["Target Column"] = data["Target Column"].astype(str) # -> If Classification Task

train,test = traintestsplit(data, trainsize = 0.8) test,futuredata = traintestsplit(test, train_size = 0.6)

Resetting Index is Required

train = train.resetindex(drop=True) test = test.resetindex(drop=True) futuredata = futuredata.reset_index(drop=True)

futuredata.drop(columns=["TargetColumn"], inplace=True) # Drop Target

Fit Data Processing

atl = Atlantic(X = train, # X:pd.DataFrame, target:str="Target_Column" target = "Target Column")

atl.fitprocessing(splitratio = 0.75, # splitratio:float=0.75, relevance:float=0.99 [0.5,1] relevance = 0.99, # h2ofsmodels:int [1,100], encodingfs:bool=True\False h2ofsmodels = 7, # vifratio:float=10.0 [3,30] encodingfs = True, vif_ratio = 10.0)

Transform Data Processing

train = atl.dataprocessing(X = train) test = atl.dataprocessing(X = test) futuredata = atl.dataprocessing(X = future_data)

Export Atlantic Preprocessing Metadata

import dill as pickle with open('fit_atl.pkl', 'wb') as output: pickle.dump(atl, output)

```

2. Atlantic - Preprocessing Data

2.1 Encoding Versions

There are multiple preprocessing methods available to direct use. This package provides upgrated encoding LabelEncoder, OneHotEncoder and InverseFrequency (IDF based) methods with an automatic multicolumn application.

```py import pandas as pd from sklearn.modelselection import traintest_split from atlantic.processing.encoders import AutoLabelEncoder, AutoIFrequencyEncoder, AutoOneHotEncoder

train,test = traintestsplit(data, trainsize=0.8) train,test = train.resetindex(drop=True), test.reset_index(drop=True) # Required

target = "Target_Column" # -> target feature name

catcols = [col for col in data.selectdtypes(include=['object']).columns if col != target]

Encoders

Create Label Encoder

encoder = AutoLabelEncoder()

Create InverseFrequency Encoder

encoder = AutoIFrequencyEncoder()

Create One-hot Encoder

encoder = AutoOneHotEncoder()

Fit

encoder.fit(train[cat_cols])

Transform the DataFrame using Label\IF\One-hot Encoding

train = encoder.transform(X = train) test = encoder.transform(X = test)

Perform an inverse transform to convert it back the original categorical columns values

train = encoder.inversetransform(X = train) test = encoder.inversetransform(X = test)

```

2.2 Feature Selection and Null Imputation Methods

Atlantic provides automated feature selection methods (H2O AutoML and VIF-based) and null imputation techniques (Simple, KNN, and Iterative). Check out the for detailed implementations of all preprocessing methods integrated in Atlantic.

Citation

Feel free to cite Atlantic as following:

```

@article{SANTOS2023100532, author = {Luis Santos and Luis Ferreira} title = {Atlantic - Automated data preprocessing framework for supervised machine learning}, journal = {Software Impacts}, volume = {17}, year = {2023}, issn = {2665-9638}, doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532}, url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696} }

```

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Luis Santos - LinkedIn