https://github.com/abertsch72/long-context-icl
Data and code for the preprint "In-Context Learning with Long-Context Models: An In-Depth Exploration"
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.8%) to scientific vocabulary
Repository
Data and code for the preprint "In-Context Learning with Long-Context Models: An In-Depth Exploration"
Basic Info
- Host: GitHub
- Owner: abertsch72
- License: apache-2.0
- Language: Jupyter Notebook
- Default Branch: main
- Size: 85.9 MB
Statistics
- Stars: 34
- Watchers: 3
- Forks: 2
- Open Issues: 1
- Releases: 0
Metadata Files
README.md
In-context learning with long-context models
This repository contains the code for reproducing the experiments from the preprint "In-Context Learning with Long-Context Models: An In-Depth Exploration." It is built on the skeleton of the code from the paper Parallel Context Windows for Large Language Models .
Use
To run the experiments from the paper, use the run_evaluation script with appropriate arguments. For instance, to run ICL with several different amounts of data on Llama-80k:
bash
python run_evaluation.py \
--dataset banking77 \
--model yaofu/llama-2-7b-80k \
--subsample-test-set 250 \
--n-runs 10 \
--seed 43 \
--nspw 1,10,25,100,1000,2000
--output-dir $OUTPUT_DIR
Synchronizing data
We ran all final experiments on a 250-example test set sampled with seed=43. To ensure the exact same results, you may wish to use our precomputed test set (rather than re-sampling). The IDs for each test set are provided in the dataset-splits folder and are also visible by looking at the first column of any results file.
Pre-computed outputs
If you do not wish to run inference, we also provide the outputs for all in-context learning experiments in the paper. The folder final-results contains experimental results, divided first by model and then by dataset. We provide the predicted label, true label, IDs of the examples used in the prompt, and label distribution of the prompt examples for reference.
Citation
If you use this code, please cite both In-Context Learning with Long-Context Models and the PCW paper: ``` @misc{bertsch2024incontext, title={In-Context Learning with Long-Context Models: An In-Depth Exploration}, author={Amanda Bertsch and Maor Ivgi and Uri Alon and Jonathan Berant and Matthew R. Gormley and Graham Neubig}, year={2024}, eprint={2405.00200}, archivePrefix={arXiv}, primaryClass={cs.CL} }
@misc{ratner2023parallel, title={Parallel Context Windows for Large Language Models}, author={Nir Ratner and Yoav Levine and Yonatan Belinkov and Ori Ram and Inbal Magar and Omri Abend and Ehud Karpas and Amnon Shashua and Kevin Leyton-Brown and Yoav Shoham}, year={2023}, eprint={2212.10947}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```
Owner
- Name: Amanda Bertsch
- Login: abertsch72
- Kind: user
- Location: Pittsburgh, PA
- Repositories: 39
- Profile: https://github.com/abertsch72
MS student @ CMU LTI
GitHub Events
Total
- Watch event: 11
- Fork event: 2
Last Year
- Watch event: 11
- Fork event: 2
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Amanda Bertsch | 4****2 | 4 |
| Amanda Bertsch | a****2@g****m | 1 |
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 2
- Total pull requests: 0
- Average time to close issues: about 2 months
- Average time to close pull requests: N/A
- Total issue authors: 2
- Total pull request authors: 0
- Average comments per issue: 1.5
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 2
- Pull requests: 0
- Average time to close issues: about 2 months
- Average time to close pull requests: N/A
- Issue authors: 2
- Pull request authors: 0
- Average comments per issue: 1.5
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- csinva (1)
- min-xu-et (1)