https://github.com/cloneofsimo/realformer-pytorch
Implementation of RealFormer using pytorch
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.3%) to scientific vocabulary
Repository
Implementation of RealFormer using pytorch
Basic Info
- Host: GitHub
- Owner: cloneofsimo
- Language: Python
- Default Branch: main
- Size: 16.7 MB
Statistics
- Stars: 100
- Watchers: 3
- Forks: 13
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
RealFormer-Pytorch
Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt CIFAR-10 dataset.
Original Paper of the model : https://arxiv.org/abs/2012.11747
So how are RealFormers at vision tasks?
Run the train.py with
python
model = ViR(
image_pix = 32,
patch_pix = 4,
class_cnt = 10,
layer_cnt = 4
)
to Test how RealFormer works on CIFAR-10 dataset compared to just classical ViT, which is
python
model = ViT(
image_pix = 32,
patch_pix = 4,
class_cnt = 10,
layer_cnt = 4
)
... which is of course, much, much smaller version of ViT compared to the origianl ones ().
Results
Model : layers = 4, hiddendim = 128, feedforwarddim = 512, head_cnt = 4
Trained 10 epochs
After 10'th epoch, Realformer achieves 65.45% while Transformer achieves 64.59% RealFormer seems to consistently have about 1% greater accuracy, which seems reasonable (as the papaer suggested simillar result)
Model : layers = 8, hiddendim = 128, feedforwarddim = 512, head_cnt = 4
Having 4 more layers obviously improves in general, and still, RealFormer consistently wins in terms of accuracy (68.3% vs 66.3%). Notice that larger the model, bigger the difference seems to follow here too. (I wonder how much of difference it would make on ViT-Large)
When it comes to computation time, there was almost zero difference. (I guess adding residual attention score is O(L^2) operation, compared to matrix multiplication in softmax which is O(L^2 * D))
Conclusion
Use RealFormer. It benifits with almost zero additional resource!
To make a custom RealFormer for other tasks
Its not a pip package, but you can use the ResEncoderBlock module in the models.py to make a Encoder Only Transformer like the following :
```python
import ResEncoderBlock from models
def RealFormer(nn.Module): ... def init(self, ...): ... self.mains = nn.Sequential(*[ResEncoderBlock(embs = 32, headcnt = 8, dp1 = 0.1, dp2 = 0.1) for _ in range(layer_cnt)]) ... def forward(self, x): ... prev = None for resencoder in self.mains: x, prev = resencoder(x, prev = prev) ... return x ``` If you're not really clear what is going on or what to do, request me to make this a pip package.
Owner
- Name: Simo Ryu
- Login: cloneofsimo
- Kind: user
- Company: Corca AI
- Website: https://fb.com/MLPaperFetchingCat
- Twitter: cloneofsimo
- Repositories: 10
- Profile: https://github.com/cloneofsimo
Cats are Turing machines cloneofsimo@gmail.com
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0