https://github.com/christophreich1996/smelu
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.0%) to scientific vocabulary
Keywords
Repository
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].
Basic Info
- Host: GitHub
- Owner: ChristophReich1996
- License: mit
- Language: Python
- Default Branch: master
- Homepage: https://arxiv.org/pdf/2202.06499.pdf
- Size: 68.4 KB
Statistics
- Stars: 20
- Watchers: 2
- Forks: 2
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
Smooth ReLU in PyTorch
Unofficial PyTorch reimplementation of the Smooth ReLU (SmeLU) activation function proposed in the paper Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations by Gil I. Shamir and Dong Lin.
This repository includes an easy-to-use pure PyTorch implementation of the Smooth ReLU.
In case you run into performance issues with this implementation, please have a look at my Triton SmeLU implementation.
Installation
The SmeLU can be installed by using pip.
shell script
pip install git+https://github.com/ChristophReich1996/SmeLU
Example Usage
The SmeLU can be simply used as a standard nn.Module:
````python import torch import torch.nn as nn from smelu import SmeLU
network: nn.Module = nn.Sequential( nn.Linear(2, 2), SmeLU(), nn.Linear(2, 2) )
output: torch.Tensor = network(torch.rand(16, 2)) ````
For a more detailed examples on hwo to use this implementation please refer to the example file (requires Matplotlib to be installed).
The SmeLU takes the following parameters.
| Parameter | Description | Type | | ------------- | ------------- | ------------- | | beta | Beta value if the SmeLU activation function. Default 2. | float |
Reference
bibtex
@article{Shamir2022,
title={{Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations}},
author={Shamir, Gil I and Lin, Dong},
journal={{arXiv preprint arXiv:2202.06499}},
year={2022}
}
Owner
- Name: Christoph Reich
- Login: ChristophReich1996
- Kind: user
- Location: Germany
- Company: Technical University of Munich
- Website: christophreich1996.github.io
- Twitter: ChristophR1996
- Repositories: 41
- Profile: https://github.com/ChristophReich1996
ELLIS Ph.D. Student @ Technical University of Munich, Technische Universität Darmstadt & University of Oxford | Prev. NEC Labs
GitHub Events
Total
- Watch event: 1
Last Year
- Watch event: 1
Dependencies
- torch >=1.0.0
- torch >=1.0.0