TabularS3L

Overview | Installation | Available Models with Quick Start Guides | Benchmark | To DO | Contributing

TabularS3L is a PyTorch Lightning-based library designed to facilitate self- and semi-supervised learning with tabular data. While numerous self- and semi-supervised learning tabular models have been proposed, there is a lack of a comprehensive library that addresses the needs of tabular practitioners. This library aims to fill this gap by providing a unified PyTorch Lightning-based framework for exploring and deploying such models.

Installation

We provide a Python package ts3l of TabularS3L for users who want to use semi- and self-supervised learning tabular models.

sh pip install ts3l

Available Models with Quick Start

TabularS3L employs a two-phase learning approach, where the learning strategies differ between phases. Below is an overview of the models available within TabularS3L, highlighting the learning strategies employed in each phase. The abbreviations 'Self-SL', 'Semi-SL', and 'SL' represent self-supervised learning, semi-supervised learning, and supervised learning, respectively.

According to the original implementation and the paper, the encoder of DAE, VIME, and SubTab is frozen during the second phase of learning. However, you can choose to freeze the encoder (i.e. backbone network) or not by setting the freeze_encoder flag in the setsecondphase method.

| Model | First Phase | Second Phase | |:---:|:---:|:---:| | DAE (GitHub)| Self-SL | SL | | VIME (NeurIPS'20) | Self-SL | Semi-SL or SL | | SubTab (NeurIPS'21) | Self-SL | SL | | SCARF (ICLR'22)| Self-SL | SL | | SwitchTab (AAAI'24) | Self-SL | SL |

In addition, TabularS3L employs a modular design, allowing you to freely choose the embedding and backbone modules.

The currently supported modules are:

• Embedding modules:
  • identity
  • feature_tokenizer (from Revisiting Deep Learning Models for Tabular Data)
• Backbone modules:
  • mlp
  • transformer

Note: The transformer backbone requires the feature_tokenizer as its embedding module.

Denoising AutoEncoder (DAE)

DAE processes input data that has been partially corrupted, producing clean data and predicting which features are corrupted during the self-supervised learning. The denoising task enables the model to learn the input distribution and generate latent representations that are robust to corruption. These latent representations can be utilized for a variety of downstream tasks.

VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain

VIME enhances tabular data learning through a dual approach. In its first phase, it utilize a pretext task of estimating mask vectors from corrupted tabular data, alongside a reconstruction pretext task for self-supervised learning. The second phase leverages consistency regularization on unlabeled data.

SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning

SubTab turns the task of learning from tabular data into as a multi-view representation challenge by dividing input features into multiple subsets during its first phase. During the second phase, collaborative inference is used to derive a joint representation by aggregating latent variables across subsets. This approach improves the model's performance in supervised learning tasks.

SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption

SCARF introduces a contrastive learning framework specifically tailored for tabular data. By corrupting random subsets of features, SCARF creates diverse views for self-supervised learning in its first phase. The subsequent phase transitions to supervised learning, utilizing a pretrained encoder to enhance model accuracy and robustness.

SwitchTab: Switched Autoencoders Are Effective Tabular Learners

SwitchTab introduces a novel self-supervised method specifically designed to decuple mutual and salient features among data pair, resulting in more representative embeddings. Moreover, the pre-trained salient embeddings can be utilized as plug-and-play features to enhance the performance of various traditional classification methods.

TabularBinning: Binning as a Pretext Task - Improving Self-Supervised Learning in Tabular Domains

TabularBinning is a novel pretext task based on the classical binning method for auto-encoding-based self-supervised learning (SSL) in tabular domains. The idea is straightforward: reconstructing the bin indices (either orders or classes) rather than the original values. This pretext task provides the encoder with an inductive bias to capture the irregular dependencies, mapping from continuous inputs to discretized bins, and mitigates the feature heterogeneity by setting all features to have category-type targets.

Benchmark

This section provides a simple benchmark comparing TabularS3L with XGBoost. The train-validation-test ratio is 6:2:2, and each model is tuned over 50 trials using Optuna. The results are averaged over five random seeds (0 to 4). The best results are shown in bold. acc, b-acc, and mse stand for Accuracy, Balanced Accuracy, and Mean Squared Error, respectively.

The embedding and backbone modules used for each model are as follows, which align with their original papers or repositories.

| Model | embedding | backbone | |:---:|:---:|:---:| | DAE | identity | mlp | | VIME | identity | mlp | | SubTab | identity | mlp | | SCARF | identity | mlp | | SwitchTab | feature_tokenizer | transformer | | TabularBinning | identity | mlp |

Use this benchmark for reference only, as only a small number of random seeds were used.

10% labeled samples

| Model | diabetes (acc) | cmc (b-acc) | abalone (mse) | |:---:|:---:|:---:|:---:| | XGBoost | 0.7325 | 0.4763 | 5.5739 | | DAE | 0.7208 | 0.4885 | 5.6168 | | VIME | 0.7182 | 0.5087 | 5.6637 | | SubTab | 0.7312 | 0.4930 | 7.2418 | | SCARF | 0.7416 | 0.4710 | 5.8888 | | SwitchTab | 0.7156 | 0.4886 | 5.9907 | | TabularBinning | 0.7450 | 0.5061 | 5.9346 |

100% labeled samples

| Model | diabetes (acc) | cmc (b-acc) | abalone (mse) | |:---:|:---:|:---:|:---:| | XGBoost | 0.7234 | 0.5291 | 4.8377 | | DAE | 0.7390 | 0.5500 | 4.5758 | | VIME | 0.7688 | 0.5477 | 4.5804 | | SubTab | 0.7390 | 0.5432 | 6.3104 | | SCARF | 0.7442 | 0.5521 | 4.4443 | | SwitchTab | 0.7585 | 0.5411 | 4.7489 | | TabularBinning | 0.7597 | 0.5486 | 4.5805 |

Contributing

Contributions to this implementation are highly appreciated. Whether it's suggesting improvements, reporting bugs, or proposing new features, feel free to open an issue or submit a pull request.

Citation

BibTex

@software{Kim_TabularS3L_2024, author = {Kim, Minwook}, doi = {10.5281/zenodo.10776538}, month = mar, title = {{TabularS3L}}, url = {https://github.com/Alcoholrithm/TabularS3L}, version = {0.70}, year = {2025} }

APA

Kim, M. (2024). TabularS3L (Version 0.70) [Computer software]. https://doi.org/10.5281/zenodo.10776538