Large Batch Training for CTR Prediction (CowClip)

LargeBatchCTR aims to train CTR prediction models with large batch (~128k). The framework is based on DeepCTR. You can run the code on a V100 GPU to feel the fast training speed.

Adaptive Column-wise Clipping (CowClip) method from paper "CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU" is implemented in this repo.

Get Started

First, download dataset to the data folder. Use data_utils.py to preprocess the data for training.

sh python data_utils.py --dataset criteo_kaggle --split rand

Then, use train.py to train the network.

```sh

Criteo (baseline)

CUDAVISIBLEDEVICES=0 python train.py --dataset criteo_kaggle --model DeepFM

Avazu (baseline)

CUDAVISIBLEDEVICES=0 python train.py --dataset avazu --model DeepFM ```

For large batch training with CowClip, do as follows:

```sh

Criteo (8K)

CUDAVISIBLEDEVICES=0 python train.py --dataset criteokaggle --model DeepFM --lrembed 1e-4 --warmup 1 --init_stddev 1e-2 --clip 1 --bound 1e-5 --bs 8192 --l2 8e-05 --lr 22.6274e-4

Criteo (128K)

CUDAVISIBLEDEVICES=0 python train.py --dataset criteokaggle --model DeepFM --lrembed 1e-4 --warmup 1 --init_stddev 1e-2 --clip 1 --bound 1e-5 --bs 131072 --l2 128e-05 --lr 90.5096e-4

Avazu (64K)

CUDAVISIBLEDEVICES=0 python train.py --dataset avazu --model DeepFM --lrembed 1e-4 --warmup 1 --initstddev 1e-2 --clip 1 --bound 1e-4 --bs 65536 --l2 64e-05 --lr 8e-4 ```

CowClip Quick Look

Dataset List

• Criteo Kaggle: download train.txt in data/criteo_kaggle/
• Avazu: download train in data/avazu/

Hyperparameters

The meaning of hyperparameters in the command line is as follows:

| params | name | | ------------- | ------------------------------------------- | | --bs | batch size | | --lrembed | learning rate for the embedding layer | | --lr | learning rate for the dense weights | | --l2 | L2-regularization weight λ | | --clip | CowClip coefficient r | | --bound | CowClip bound ζ | | --warmup | number of epochs to warmup on dense weights | | --initstddev | initialization weight standard deviation |

The hyperparameters neet to be scaled are listed as follows. For Criteo dataset:

| bs | lr | l2 | ζ | DeepFM AUC(%) | Time(min) | | :--- | :------- | :----- | :---: | :-----------: | :-------: | | 1K | 8e-4 | 1e-5 | 1e-5 | 80.86 | 768 | | 2K | 11.31e-4 | 2e-5 | 1e-5 | 80.93 | 390 | | 4K | 16e-4 | 4e-5 | 1e-5 | 80.97 | 204 | | 8K | 22.62e-4 | 8e-5 | 1e-5 | 80.97 | 102 | | 16K | 32e-4 | 16e-5 | 1e-5 | 80.94 | 48 | | 32K | 45.25e-4 | 32e-5 | 1e-5 | 80.95 | 27 | | 64K | 64e-4 | 64e-5 | 1e-5 | 80.96 | 15 | | 128K | 90.50e-4 | 128e-5 | 1e-5 | 80.90 | 9 |

For Avazu dataset:

| bs | lr | l2 | ζ | DeepFM AUC(%) | Time(min) | | :--- | :------ | :---- | :---: | :-----------: | :-------: | | 1K | 1e-4 | 1e-5 | 1e-3 | 78.83 | 210 | | 2K | 1.41e-4 | 2e-5 | 1e-3 | 78.82 | 108 | | 4K | 2e-4 | 4e-5 | 1e-4 | 78.90 | 54 | | 8K | 2.83e-4 | 8e-5 | 1e-4 | 79.06 | 30 | | 16K | 4e-4 | 16e-5 | 1e-4 | 79.01 | 17 | | 32K | 5.66e-4 | 32e-5 | 1e-4 | 78.82 | 10 | | 64K | 8e-4 | 64e-5 | 1e-4 | 78.82 | 6.7 | | 128K | 16e-4 | 96e-5 | 1e-4 | 78.80 | 4.8 |

Model List

| Model | Paper | | :------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------- | | Wide & Deep | [DLRS 2016]Wide & Deep Learning for Recommender Systems | | DeepFM | [IJCAI 2017]DeepFM: A Factorization-Machine based Neural Network for CTR Prediction | | Deep & Cross Network | [ADKDD 2017]Deep & Cross Network for Ad Click Predictions | | DCN V2 | [arxiv 2020]DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems |

Requirements

Tensorflow 2.4.0
Tensorflow-Addons

sh pip install -r requirements.txt

Citation

bibtex @article{zheng2022cowclip, title={{CowClip}: Reducing {CTR} Prediction Model Training Time from 12 hours to 10 minutes on 1 {GPU}}, author={Zangwei Zheng, Pengtai Xu, Xuan Zou, Da Tang, Zhen Li, Chenguang Xi, Peng Wu, Leqi Zou, Yijie Zhu, Ming Chen, Xiangzhuo Ding, Fuzhao Xue, Ziheng Qing, Youlong Cheng, Yang You}, journal={arXiv}, volume={abs/2204.06240}, year={2022} }