conformer
Implementation of the convolutional module from the Conformer paper, for use in Transformers
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (5.5%) to scientific vocabulary
Keywords
Repository
Implementation of the convolutional module from the Conformer paper, for use in Transformers
Basic Info
Statistics
- Stars: 420
- Watchers: 9
- Forks: 55
- Open Issues: 6
- Releases: 13
Topics
Metadata Files
README.md
Conformer
Implementation of the convolutional module from the Conformer paper, for improving the local inductive bias in Transformers.
Install
bash
$ pip install conformer
Usage
The Conformer convolutional module, the main novelty of the paper
```python import torch from conformer import ConformerConvModule
layer = ConformerConvModule( dim = 512, causal = False, # auto-regressive or not - 1d conv will be made causal with padding if so expansionfactor = 2, # what multiple of the dimension to expand for the depthwise convolution kernelsize = 31, # kernel size, 17 - 31 was said to be optimal dropout = 0. # dropout at the very end )
x = torch.randn(1, 1024, 512) x = layer(x) + x ```
1 Conformer Block
```python import torch from conformer import ConformerBlock
block = ConformerBlock( dim = 512, dimhead = 64, heads = 8, ffmult = 4, convexpansionfactor = 2, convkernelsize = 31, attndropout = 0., ffdropout = 0., conv_dropout = 0. )
x = torch.randn(1, 1024, 512)
block(x) # (1, 1024, 512) ```
Conformer - just multiple ConformerBlock from above
```python import torch from conformer import Conformer
conformer = Conformer( dim = 512, depth = 12, # 12 blocks dimhead = 64, heads = 8, ffmult = 4, convexpansionfactor = 2, convkernelsize = 31, attndropout = 0., ffdropout = 0., conv_dropout = 0. )
x = torch.randn(1, 1024, 512)
conformer(x) # (1, 1024, 512) ```
Todo
- [ ] switch to a better relative positional encoding. shaw's is dated
- [ ] flash attention with a better RPE
Citations
bibtex
@misc{gulati2020conformer,
title = {Conformer: Convolution-augmented Transformer for Speech Recognition},
author = {Anmol Gulati and James Qin and Chung-Cheng Chiu and Niki Parmar and Yu Zhang and Jiahui Yu and Wei Han and Shibo Wang and Zhengdong Zhang and Yonghui Wu and Ruoming Pang},
year = {2020},
eprint = {2005.08100},
archivePrefix = {arXiv},
primaryClass = {eess.AS}
}
Owner
- Name: Phil Wang
- Login: lucidrains
- Kind: user
- Location: San Francisco
- Website: lucidrains.github.io
- Repositories: 283
- Profile: https://github.com/lucidrains
Working with Attention. It's all we need
GitHub Events
Total
- Watch event: 51
- Fork event: 4
Last Year
- Watch event: 51
- Fork event: 4
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 12
- Total pull requests: 1
- Average time to close issues: 4 days
- Average time to close pull requests: about 1 hour
- Total issue authors: 12
- Total pull request authors: 1
- Average comments per issue: 1.75
- Average comments per pull request: 1.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- chutianxu (1)
- Honghe (1)
- Xujianzhong (1)
- CSLujunyu (1)
- pfeatherstone (1)
- sorobedio (1)
- jinglescode (1)
- JaesungHuh (1)
- pengzhangzhi (1)
- arogozhnikov (1)
- robflynnyh (1)
- gandroz (1)
Pull Request Authors
- enhuiz (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- einops *
- torch *
- actions/checkout v2 composite
- actions/setup-python v2 composite