grimoire
Grimoire is All You Need for Enhancing Large Language Models
Science Score: 51.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
✓Committers with academic emails
1 of 3 committers (33.3%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.1%) to scientific vocabulary
Keywords
Repository
Grimoire is All You Need for Enhancing Large Language Models
Basic Info
- Host: GitHub
- Owner: IAAR-Shanghai
- License: apache-2.0
- Language: Python
- Default Branch: main
- Homepage: https://arxiv.org/abs/2401.03385
- Size: 3.61 MB
Statistics
- Stars: 116
- Watchers: 6
- Forks: 13
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
English |
Grimoire is All You Need for Enhancing LLMs
Enhance the capabilities of small language models using grimoires.
Contents
Introduction
In-context learning (ICL) is one of the key methods for enhancing the performance of large language models on specific tasks by providing a set of few-shot question and answer examples. However, the ICL capability of different types of models shows significant variation due to factors such as model architecture, volume of learning data, and the size of parameters. Generally, the larger the model's parameter size and the more extensive the learning data, the stronger its ICL capability. In this paper, we propose a method SLEICL (Strong LLM Enhanced ICL) that involves learning from examples using strong language models and then summarizing and transferring these learned skills to weak language models for inference and application.
This ensures the stability and effectiveness of ICL. Compared to directly enabling weak language models to learn from prompt examples, SLEICL reduces the difficulty of ICL for these models. Our experiments, conducted on up to eight datasets with five language models, demonstrate that weak language models achieve consistent improvement over their own zero-shot or few-shot capabilities using the SLEICL method. Some weak language models even surpass the performance of GPT4-1106-preview (zero-shot) with the aid of SLEICL.
Project Structure
The project is organized into several key directories and modules. Here's an overview of the project structure:
.
archived # Store the grimoire and hard samples used in our experiment.
assets # Store project assets, such as images, diagrams, or any visual elements used to enhance the presentation and understanding of the project.
configs # Store configuration files.
core # Core codebase.
data # Data processing module.
evaluator # Evaluator module.
llm # Load Large Language Models (LLMs) module.
data # Store datasets and data processing scripts.
external # Store the Grimoire Ranking model based on the classifier approach.
outputs # Store experiment output files.
prompts # Store text files used as prompts when interacting with LLMs.
stats # Store experiment statistical results.
tests # Store test code or unit tests.
Get Started
Clone the repository.
git clone https://github.com/IAAR-Shanghai/Grimoire.git && cd Grimoire
Prepare for the conda environment.
conda create -n grimoire python=3.8.18conda activate grimoire
Install Python dependencies and process the data.
chmod +x setup.sh./setup.sh
Configure
- the llms in configs/llm.yaml.
- the experiments in configs/experiment.yaml.
- If it is necessary to reproduce our experiment, you can load the grimoire and hard samples used in this experiment to the current path using the following command:
cp -r ./archived/.cache ./.
Look into experiments.py to see how to run experiments.
Run analyst.py to analyze the results saved in
outputs.
Note: Regarding the deployment of LLMs, we also provide some reference tutorials.
Results
Contact Us
For any questions, feedback, or suggestions, please open a GitHub Issue. You can reach out through GitHub Issues.
TODOs
Click me to show all TODOs
- [x] Write a unified `setup.sh` to implement Python dependencies installation and the implementation of `embed.py` and `compute_similarity.py`; - [x] Provides a simple tutorial on deploying vllm models; - [x] Load large models directly from `huggingface`; - [ ] Add configurable items in `experiment.yaml`; - [ ] The experimental environment and code are packaged based on Docker, which is convenient for researchers to use and deploy quickly;Citation
@article{Grimoire,
title={Grimoire is All You Need for Enhancing Large Language Models},
author={Ding Chen and Shichao Song and Qingchen Yu and Zhiyu Li and Wenjin Wang and Feiyu Xiong and Bo Tang},
journal={arXiv preprint arXiv:2401.03385},
year={2024},
}
Owner
- Name: LLM Group, Institute for Advanced Algorithms Research, Shanghai
- Login: IAAR-Shanghai
- Kind: organization
- Repositories: 1
- Profile: https://github.com/IAAR-Shanghai
Citation (CITATION.bib)
@article{grimoire,
title={Grimoire is All You Need for Enhancing Large Language Models},
author={Ding Chen and Shichao Song and Qingchen Yu and Zhiyu Li and Wenjin Wang and Feiyu Xiong and Bo Tang},
year={2024},
eprint={2401.03385},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
GitHub Events
Total
- Watch event: 11
- Fork event: 3
Last Year
- Watch event: 11
- Fork event: 3
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 1
- Total pull requests: 6
- Average time to close issues: 5 days
- Average time to close pull requests: 1 minute
- Total issue authors: 1
- Total pull request authors: 2
- Average comments per issue: 1.0
- Average comments per pull request: 0.17
- Merged pull requests: 3
- Bot issues: 0
- Bot pull requests: 1
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- Duguce (6)
- dependabot[bot] (2)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- loguru ==0.7.2
- matplotlib *
- numpy *
- openai ==1.3.7
- requests ==2.31.0
- scikit-learn *
- scipy *
- sentence_transformers ==2.2.2
- tenacity ==8.2.3
- torch ==2.1.1
- transformers ==4.35.2