helm

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

https://github.com/stanford-crfm/helm

Science Score: 72.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
✓
Committers with academic emails
32 of 128 committers (25.0%) from academic institutions
✓
Institutional organization owner
Organization stanford-crfm has institutional domain (crfm.stanford.edu)
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (15.1%) to scientific vocabulary

Keywords from Contributors

standardization meshing data-profilers datacleaner pde pinn interpretability pipeline-testing application projections

Last synced: 6 months ago · JSON representation ·

Repository

Basic Info

Host: GitHub
Owner: stanford-crfm
License: apache-2.0
Language: Python
Default Branch: main
Homepage: https://crfm.stanford.edu/helm
Size: 116 MB

Statistics

Stars: 2,433
Watchers: 37
Forks: 327
Open Issues: 153
Releases: 16

Created over 4 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog License Citation

README.md

Holistic Evaluation of Language Models (HELM)

[comment]: <> (When using the img tag, which allows us to specify size, src has to be a URL.)

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. This framework includes the following features:

Datasets and benchmarks in a standardized format (e.g. MMLU-Pro, GPQA, IFEval, WildBench)
Models from various providers accessible through a unified interface (e.g. OpenAI models, Anthropic Claude, Google Gemini)
Metrics for measuring various aspects beyond accuracy (e.g. efficiency, bias, toxicity)
Web UI for inspecting individual prompts and responses
Web leaderboard for comparing results across models and benchmarks

Documentation

Please refer to the documentation on Read the Docs for instructions on how to install and run HELM.

Quick Start

Install the package from PyPI:

sh pip install crfm-helm

Run the following in your shell:

```sh

Run benchmark

helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10

Summarize benchmark results

helm-summarize --suite my-suite

Start a web server to display benchmark results

helm-server --suite my-suite ```

Then go to http://localhost:8000/ in your browser.

Leaderboards

We maintain offical leaderboards with results from evaluating recent models on notable benchmarks using this framework. Our current flagship leaderboards are:

We also maintain leaderboards for a diverse range of domains (e.g. medicine, finance) and aspects (e.g. multi-linguality, world knowledge, regulation compliance). Refer to the HELM website for a full list of leaderboards.

Papers

The HELM framework was used in the following papers for evaluating models.

Holistic Evaluation of Language Models - paper, leaderboard
Holistic Evaluation of Vision-Language Models (VHELM) - paper, leaderboard, documentation
Holistic Evaluation of Text-To-Image Models (HEIM) - paper, leaderboard, documentation
Image2Struct: Benchmarking Structure Extraction for Vision-Language Models - paper
Enterprise Benchmarks for Large Language Model Evaluation - paper, documentation
The Mighty ToRR: A Benchmark for Table Reasoning and Robustness - paper, leaderboard
Reliable and Efficient Amortized Model-based Evaluation - paper, documentation
MedHELM - paper in progress, leaderboard, documentation
Holistic Evaluation of Audio-Language Models - paper, leaderboard

The HELM framework can be used to reproduce the published model evaluation results from these papers. To get started, refer to the documentation links above for the corresponding paper, or the main Reproducing Leaderboards documentation.

Citation

If you use this software in your research, please cite the Holistic Evaluation of Language Models paper as below.

bibtex @article{ liang2023holistic, title={Holistic Evaluation of Language Models}, author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2023}, url={https://openreview.net/forum?id=iO4LZibEqW}, note={Featured Certification, Expert Certification} }

Owner

Name: Stanford Center for Research on Foundation Models
Login: stanford-crfm
Kind: organization
Location: United States of America

Website: https://crfm.stanford.edu/
Repositories: 8
Profile: https://github.com/stanford-crfm

Citation (CITATION.bib)

@article{
liang2023holistic,
title={Holistic Evaluation of Language Models},
author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=iO4LZibEqW},
note={Featured Certification, Expert Certification}
}

Committers

Last synced: 10 months ago

All Time

Total Commits: 4,497
Total Committers: 128
Avg Commits per committer: 35.133
Development Distribution Score (DDS): 0.756

Past Year

Commits: 785
Committers: 46
Avg Commits per committer: 17.065
Development Distribution Score (DDS): 0.345

Top Committers

Name	Email	Commits
Tony Lee	t**e@y**m	1,097
Yifan Mai	y**n@c**u	1,052
Percy Liang	p**g@g**m	313
Dilara Soylu	d****u	265
Yian Zhang	y**g@n**u	165
Dimitris Tsipras	d**0@g**m	149
Rishi Bommasani	r**i@g**m	132
Deepak Narayanan	d**4@g**m	109
Josselin Somerville Roberts	7****s	90
github-actions[bot]	4****]	85
Ryan Chi	r**i@g**m	78
Farzaan Kaiyom	3****k	70
Xuechen Li	1****n	63
Eric Zelikman	e**n@s**u	60
Brian W. Goldman	2****n	50
Ben Newman	n**1@g**m	35
Drew Arad Hudson	d**d@s**u	33
santhnm2	k**2@s**u	32
Frieda Rong	r**f@s**u	31
Ananya Kumar	s**4@g**m	30
Yuhui Zhang	z**u@g**m	29
Nathan Kim	n**n@g**m	29
Haoqin Tu	t**3@g**m	27
Faisal Ladhak	f**l@c**u	22
Huaxiu Yao	y**2@g**m	21
dependabot[bot]	4****]	21
MiguelAFH	m**n@s**u	19
Michihiro Yasunaga	m**u@c**u	19
q-hwang	q**g@s**u	18
Tiiiger	z**x@g**m	17
and 98 more...

Committer Domains (Top 20 + Academic)

stanford.edu: 9 cs.stanford.edu: 7 john0.stanford.edu: 1 jagupard14.stanford.edu: 1 dna808643.sunet: 1 jp.ibm.com: 1 dna80c245.sunet: 1 cs.washington.edu: 1 ccc.ufcg.edu.br: 1 connect.ust.hk: 1 sc.stanford.edu: 1 joelniklaus.ch: 1 mail.mcgill.ca: 1 mkly.io: 1 berkeley.edu: 1 cs.toronto.edu: 1 cs.columbia.edu: 1 nyu.edu: 1 audiomulch.com: 1 vencharging.com: 1 uw.edu: 1 jagupard12.stanford.edu: 1 napoli115.stanford.edu: 1 stats.ox.ac.uk: 1 raiders7.stanford.edu: 1 svl9.stanford.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 428
Total pull requests: 1,444
Average time to close issues: 5 months
Average time to close pull requests: 6 days
Total issue authors: 139
Total pull request authors: 94
Average comments per issue: 1.74
Average comments per pull request: 0.43
Merged pull requests: 1,206
Bot issues: 0
Bot pull requests: 113

Past Year

Issues: 82
Pull requests: 977
Average time to close issues: 19 days
Average time to close pull requests: 3 days
Issue authors: 45
Pull request authors: 44
Average comments per issue: 1.45
Average comments per pull request: 0.22
Merged pull requests: 839
Bot issues: 0
Bot pull requests: 102

View more stats

Top Authors

Issue Authors

yifanmai (174)
zhimin-z (57)
teetone (46)
farzaank (24)
JosselinSomervilleRoberts (19)
percyliang (11)
brianwgoldman (7)
rishibommasani (6)
ogencoglu (5)
sermolin (5)
bryanzhou008 (5)
kapilmayank (5)
msaroufim (5)
shaafsalman (5)
dlwh (5)

Pull Request Authors

yifanmai (1,294)
github-actions[bot] (149)
JosselinSomervilleRoberts (98)
farzaank (98)
teetone (90)
dependabot[bot] (57)
ImKeTT (56)
brianwgoldman (47)
MiguelAFH (46)
liamjxu (33)
andyzorigin (23)
martinakaduc (17)
siyagoel (16)
patelfagun1998 (11)
raileymontalan (10)

Top Labels

Issue Labels

user question (139) p2 (98) models (66) frontend (56) bug (51) p3 (47) enhancement (43) p1 (32) framework (29) additions (24) scenarios (23) good first issue (22) cleanup (19) HEIM (Text2Image) (18) VHELM (16) competition (14) help wanted (12) documentation (11) proxy (10) together (10) methodology (6) devinfra (5) Visualization (5) metrics (5) harms (3) General (3) packaging (3) unit testing (3) results (2) deployment (2)

Pull Request Labels

dependencies (56) python (30) VHELM (23) javascript (22) models (7) scenarios (6) documentation (4) metrics (3) MedHELM (2) Language (1) HEIM (Text2Image) (1) cleanup (1)

Packages

Total packages: 3
Total downloads:
- pypi 2,400 last-month

Total dependent packages: 1
(may contain duplicates)
Total dependent repositories: 2
(may contain duplicates)
Total versions: 34
Total maintainers: 3

pypi.org: crfm-helm

Benchmark for language models

Homepage: https://github.com/stanford-crfm/helm
Documentation: https://crfm-helm.readthedocs.io/
License: Apache License 2.0
Latest release: 0.5.7
published 7 months ago

Versions: 16
Dependent Packages: 1
Dependent Repositories: 2
Downloads: 2,324 Last month
Docker Downloads: 0

Rankings

Stargazers count: 1.9%

Forks count: 3.9%

Docker downloads count: 4.3%

Dependent packages count: 4.8%

Average: 6.1%

Downloads: 10.2%

Dependent repos count: 11.5%

Maintainers (2)

percyliang yifanmai

Last synced: 6 months ago

proxy.golang.org: github.com/stanford-crfm/helm

Documentation: https://pkg.go.dev/github.com/stanford-crfm/helm#section-documentation
License: apache-2.0
Latest release: v0.5.7
published 7 months ago

Versions: 16
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 6.5%

Average: 6.7%

Dependent repos count: 7.0%

Last synced: 6 months ago

pypi.org: nvidia-crfm-helm

NVIDIA: Benchmark for language models - Fork of Stanford CRFM HELM

Homepage: https://github.com/stanford-crfm/helm
Documentation: https://nvidia-crfm-helm.readthedocs.io/
License: Apache License 2.0
Latest release: 25.7.2
published 7 months ago

Versions: 2
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 76 Last month

Rankings

Stargazers count: 2.2%

Forks count: 3.9%

Dependent packages count: 8.7%

Average: 16.0%

Dependent repos count: 49.2%

Maintainers (1)

nvidia

Last synced: 6 months ago

Dependencies

requirements.txt pypi

Mako *
black *
bottle *
dacite *
datasets *
flake8 *
gdown *
google-api-python-client *
gunicorn *
icetk *
importlib-resources *
jsonlines *
mypy *
nltk *
numba *
openai *
pre-commit *
protobuf ==3.20.1
pyext *
pyhocon *
pymongo *
pytest *
pytrec_eval ==0.5
retrying *
rouge-score *
sacrebleu *
scikit-learn *
scipy *
sentencepiece *
spacy *
sqlitedict *
summ-eval *
sympy *
torch *
torchvision *
tqdm *
transformers *
uncertainty-calibration *
websocket-client *
zstandard *

.github/workflows/python-publish.yml actions

actions/checkout v3 composite
actions/setup-python v3 composite
pypa/gh-action-pypi-publish release/v1 composite

.github/workflows/test.yml actions

actions/cache v2 composite
actions/checkout v2 composite
actions/setup-python v4 composite

scripts/data_overlap/Dockerfile docker

python 3.8 build

docs/requirements.txt pypi

mkdocs ==1.4.2
mkdocs-include-markdown-plugin ==4.0.0
mkdocs-macros-plugin ==0.7.0
mkdocstrings ==0.20.0
mkdocstrings-python ==1.3.0

pyproject.toml pypi

scripts/data_overlap/requirements-freeze.txt pypi

apache-beam ==2.47.0
bitarray ==2.7.3
certifi ==2023.5.7
charset-normalizer ==3.1.0
click ==8.1.3
cloudpickle ==2.2.1
crcmod ==1.7
dill ==0.3.1.1
dnspython ==2.3.0
docopt ==0.6.2
fastavro ==1.7.4
fasteners ==0.18
grpcio ==1.54.2
hdfs ==2.7.0
httplib2 ==0.21.0
idna ==3.4
joblib ==1.2.0
nltk ==3.8.1
numpy ==1.24.3
objsize ==0.6.1
orjson ==3.8.13
pandas ==2.0.1
proto-plus ==1.22.2
protobuf ==4.22.5
pyarrow ==11.0.0
pydot ==1.4.2
pymongo ==4.3.3
pyparsing ==3.0.9
python-dateutil ==2.8.2
pytz ==2023.3
regex ==2023.5.5
requests ==2.31.0
six ==1.16.0
tqdm ==4.65.0
typing_extensions ==4.6.2
tzdata ==2023.3
urllib3 ==2.0.2
zstandard ==0.21.0

scripts/data_overlap/requirements.txt pypi

apache-beam ==2.47.0
bitarray ==2.7.3
nltk ==3.8.1
pandas ==2.0.1
pyarrow ==11.0.0
tqdm ==4.65.0