Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (1.6%) to scientific vocabulary
Last synced: 6 months ago
·
JSON representation
Repository
WIP
Basic Info
Statistics
- Stars: 93
- Watchers: 9
- Forks: 1
- Open Issues: 0
- Releases: 0
Created over 1 year ago
· Last pushed over 1 year ago
Metadata Files
Readme
README.md
Almost Comprehensive Scaling Guide for Transformer Training
WIP
Feature
Activation Tracking:
Weight Update Tracking:
Scope:
- Kaplan vs Chinchilla
- SP vs muP vs layerwise SP
- adamW weight decay
- infinite lr scheduler
- batch_size vs lr (sqrt BS law)
- agd-muP (spectral initialization vs classic muP)
- adam-atan2
- data dependent lr tranfer
- embedding lr transfer
- u-muP
Shoutout
Thanks to Fal.ai for providing compute to run these experiments.
Owner
- Name: Simo Ryu
- Login: cloneofsimo
- Kind: user
- Company: Corca AI
- Website: https://fb.com/MLPaperFetchingCat
- Twitter: cloneofsimo
- Repositories: 10
- Profile: https://github.com/cloneofsimo
Cats are Turing machines cloneofsimo@gmail.com
GitHub Events
Total
- Watch event: 17
Last Year
- Watch event: 17
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0