https://github.com/cheryyunl/megatron-llm
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.7%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: cheryyunl
- License: other
- Language: Python
- Default Branch: main
- Size: 5.5 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Megatron-LLM
This library enables pre-training and fine-tuning of large language models (LLMs) at scale. Our repository is a modification of the original Megatron-LM codebase by Nvidia.
Added key features include: - Llama, Llama 2, Code Llama and Falcon support - support training of large models (70B Llama 2, 65B Llama 1, 34B Code Llama, and 40B Falcon) on commodity hardware on multiple nodes - 3-way parallelism: tensor parallel, pipeline parallel and data parallel training (inherited from Megatron) - full pretraining, finetuning and instruct tuning support - Support for special tokens & tokenizers - grouped-query attention (GQA) and multi-query attention (MQA) - Rotary Position Embeddings (RoPE), RMS layer norm, Lima dropout - RoPE scaling for longer attention context support - FlashAttention 2 - BF16 / FP16 training - WandB integration - Metrics support: Ease to add custom metrics to evaluate on the validation set while training - Conversion to and from Hugging Face hub
Documentation
Take a look at the online documentation.
Alternatively, build the docs from source:
cd docs/
pip install -r requirements.txt
make html
Example models trained with Megatron-LLM
70B Llama 2 1, 40B Falcon 1, 13B Code Llama 1, ... (Let us know about yours!)
Citation
If you use this software please cite it:
@software{epfmgtrn,
author = {Alejandro Hernández Cano and
Matteo Pagliardini and
Andreas Köpf and
Kyle Matoba and
Amirkeivan Mohtashami and
Olivia Simin Fan and
Axel Marmet and
Deniz Bayazit and
Igor Krawczuk and
Zeming Chen and
Francesco Salvi and
Antoine Bosselut and
Martin Jaggi},
title = {epfLLM Megatron-LM},
year = 2023,
url = {https://github.com/epfLLM/Megatron-LLM}
}
Owner
- Name: Cheryl Liang
- Login: cheryyunl
- Kind: user
- Location: Toronto
- Website: https://cheryyunl.github.io/
- Repositories: 1
- Profile: https://github.com/cheryyunl
Reinforcement Learning, Interactive Learning System
GitHub Events
Total
- Push event: 11
- Create event: 2
Last Year
- Push event: 11
- Create event: 2
Dependencies
- sphinx-notes/pages v3 composite
- einops >=0.6.1
- flask >=2.3.0
- flask_restful >=0.3.0
- myst-parser >=2.0.0
- numpy >=1.25
- pillow >=10.0.0
- pydata-sphinx-theme >=0.13.0
- regex >=2023.6.0
- sphinx ==7.1.0
- torch >=2.0.0
- wandb >=0.15.0
- datasets >=2.14.0
- flash-attn >=2.4.2
- nltk >=3.8.0
- sentencepiece >=0.1.0
- torch >=2.0.0
- transformers >=4.36.2