https://github.com/abudubai16/switch-transformer-moe
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (2.9%) to scientific vocabulary
Last synced: 10 months ago
·
JSON representation
Repository
Basic Info
- Host: GitHub
- Owner: abudubai16
- Language: Python
- Default Branch: main
- Size: 16.6 KB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Created about 2 years ago
· Last pushed about 2 years ago
Metadata Files
Readme
README.md
Switch Transformer: Mixture Of Experts
This code is based off of the paper: https://arxiv.org/pdf/2101.03961
```bash
Clone the repository
git clone https://github.com/abudubai16/Switch-Transformer-MoE.git
Navigate to your repository
cd your-repository ``` The code behaves almost exactly like the pytorch implementation of the TransformerDecoderLayer and TransformerDecoder, the only difference is the loss that the module returns to ensure proper distribution of tokens.
Owner
- Login: abudubai16
- Kind: user
- Repositories: 1
- Profile: https://github.com/abudubai16