mergekit
Tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models.
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.1%) to scientific vocabulary
Keywords
Repository
Tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models.
Basic Info
- Host: GitHub
- Owner: louisbrulenaudet
- License: apache-2.0
- Language: Jupyter Notebook
- Default Branch: main
- Homepage: https://louisbrulenaudet.com
- Size: 13.7 KB
Statistics
- Stars: 8
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
MergeKit, a tools for merging pretrained LLM and create Mixture of Experts (MoE) from open-source models.
Mergekit uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.
When you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. mergekit generates a README.md for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely ; or use the generated README.md as-is. It is also possible to edit your README.md online once it has been uploaded to the Hub.
Citing this project
If you use this code in your research, please use the following BibTeX entry.
```BibTeX @misc{louisbrulenaudet2023, author = {Louis Brulé Naudet}, title = {MergeKit, a tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models}, howpublished = {\url{https://github.com/louisbrulenaudet/mergeKit}}, year = {2024} }
```
Feedback
If you have any feedback, please reach out at louisbrulenaudet@icloud.com.
Owner
- Name: Louis Brulé Naudet
- Login: louisbrulenaudet
- Kind: user
- Location: Paris
- Company: Université Paris-Dauphine (Paris Sciences et Lettres - PSL)
- Website: https://louisbrulenaudet.com
- Twitter: BruleNaudet
- Repositories: 81
- Profile: https://github.com/louisbrulenaudet
Research in business taxation and development (NLP, LLM, Computer vision...), University Dauphine-PSL 📖 | Backed by the Microsoft for Startups Hub program
Citation (CITATION.cff)
cff-version: 1.2.0 message: "If you use this software, please cite it as below." authors: - family-names: "Brulé Naudet" given-names: "Louis" orcid: "https://orcid.org/0000-0001-9111-4879" title: "MergeKit, a tools for merging pretrained LLM and create Mixture of Experts (MoE) from open-source models" version: 1.0.0 date-released: 2024-02-11
GitHub Events
Total
- Watch event: 2
Last Year
- Watch event: 2
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 1
- Total pull requests: 0
- Average time to close issues: 1 minute
- Average time to close pull requests: N/A
- Total issue authors: 1
- Total pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 1
- Pull requests: 0
- Average time to close issues: 1 minute
- Average time to close pull requests: N/A
- Issue authors: 1
- Pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- louisbrulenaudet (1)