memit
Mass-editing thousands of facts into a transformer memory (ICLR 2023)
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.4%) to scientific vocabulary
Keywords
Repository
Mass-editing thousands of facts into a transformer memory (ICLR 2023)
Basic Info
- Host: GitHub
- Owner: kmeng01
- License: mit
- Language: Python
- Default Branch: main
- Homepage: https://memit.baulab.info
- Size: 98.6 KB
Statistics
- Stars: 476
- Watchers: 5
- Forks: 62
- Open Issues: 18
- Releases: 0
Topics
Metadata Files
README.md
MEMIT: Mass-Editing Memory in a Transformer
Editing thousands of facts into a transformer memory at once.
Table of Contents
- Installation
- MEMIT Algorithm Demo
- Running the Full Evaluation Suite
- Generating Scaling Curves
- How to Cite
Installation
We recommend conda for managing Python, CUDA, and PyTorch; pip is for everything else. To get started, simply install conda and run:
bash
CONDA_HOME=$CONDA_HOME ./scripts/setup_conda.sh
$CONDA_HOME should be the path to your conda installation, e.g., ~/miniconda3.
MEMIT Algorithm Demo
notebooks/memit.ipynb demonstrates MEMIT. The API is simple; simply specify a requested rewrite of the following form:
python
request = [
{
"prompt": "{} plays the sport of",
"subject": "LeBron James",
"target_new": {
"str": "football"
}
},
{
"prompt": "{} plays the sport of",
"subject": "Michael Jordan",
"target_new": {
"str": "baseball"
}
},
]
Other similar example(s) are included in the notebook.
Running the Full Evaluation Suite
experiments/evaluate.py can be used to evaluate any method in baselines/.
For example:
python3 -m experiments.evaluate \
--alg_name=MEMIT \
--model_name=EleutherAI/gpt-j-6B \
--hparams_fname=EleutherAI_gpt-j-6B.json \
--num_edits=10000 \
--use_cache
Results from each run are stored at results/<method_name>/run_<run_id> in a specific format:
bash
results/
|__ MEMIT/
|__ run_<run_id>/
|__ params.json
|__ case_0.json
|__ case_1.json
|__ ...
|__ case_10000.json
To summarize the results, you can use experiments/summarize.py:
bash
python3 -m experiments.summarize --dir_name=MEMIT --runs=run_<run1>,run_<run2>
Running python3 -m experiments.evaluate -h or python3 -m experiments.summarize -h provides details about command-line flags.
How to Cite
bibtex
@article{meng2022memit,
title={Mass Editing Memory in a Transformer},
author={Kevin Meng and Sen Sharma, Arnab and Alex Andonian and Yonatan Belinkov and David Bau},
journal={arXiv preprint arXiv:2210.07229},
year={2022}
}
Owner
- Name: Kevin Meng
- Login: kmeng01
- Kind: user
- Location: boston
- Company: @mit, @csail
- Website: mengk.me
- Twitter: mengk20
- Repositories: 3
- Profile: https://github.com/kmeng01
@MIT. interested in language models, compbio, and robotics.
Citation (CITATION.cff)
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
preferred-citation:
type: article
authors:
- family-names: "Meng"
given-names: "Kevin"
- family-names: "Sen Sharma"
given-names: "Arnab"
- family-names: "Andonian"
given-names: "Alex"
- family-names: "Belinkov"
given-names: "Yonatan"
- family-names: "Bau"
given-names: "David"
journal: "arXiv preprint arXiv:2210.07229"
title: "Mass-Editing Memory in a Transformer"
year: 2022
GitHub Events
Total
- Issues event: 2
- Watch event: 71
- Issue comment event: 6
- Fork event: 18
Last Year
- Issues event: 2
- Watch event: 71
- Issue comment event: 6
- Fork event: 18
Dependencies
- allennlp *
- click ==7.1.2
- datasets *
- hydra-core *
- jsonlines *
- numpy *
- spacy *
- torch *
- wandb *