https://github.com/brucewlee/h-test
[ACL 2024] Language Models Don't Learn the Physical Manifestation of Language
Science Score: 33.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
✓Committers with academic emails
1 of 2 committers (50.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.2%) to scientific vocabulary
Keywords
Repository
[ACL 2024] Language Models Don't Learn the Physical Manifestation of Language
Basic Info
- Host: GitHub
- Owner: brucewlee
- License: mit
- Language: Python
- Default Branch: master
- Homepage: https://arxiv.org/abs/2402.11349
- Size: 390 KB
Statistics
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
readme.md
H-TEST: Language Models Don’t Learn the Physical Manifestation of Language
This repository contains the code and data for the paper "Language Models Don't Learn the Physical Manifestation of Language". The paper introduces H-TEST, a series of tasks designed to assess the ability of language models to understand and utilize the visual and auditory properties of language.
Overview
The key components of this repository are:
- data/: Contains scripts for generating test instances for each task in H-TEST.
- generatetraindata.py: Script for generating training data for fine-tuning experiments.
- run_test.py: Main script for running H-TEST on various language models.
- runlettergeometry_test.py: Script for running the Letter Geometry challenge on language models.
- utils.py: Utility functions for interacting with language model APIs and processing responses.
Usage
Running H-TEST
To run H-TEST on a set of language models, modify the modellist and other setups in runtest.py to include the models you wish to test, and then run the script:
python run_test.py
The script will generate test instances, run the tests on the specified models, and print the results.
Running the Letter Geometry Challenge
To run the Letter Geometry challenge on a set of language models, modify the modellist in runlettergeometrytest.py to include the models you wish to test, and then run the script:
python run_letter_geometry_test.py
The script will generate challenge instances, run the tests on the specified models, and print the results.
Generating Training Data
To generate training data for fine-tuning experiments, run the generatetraindata.py script:
python generate_train_data.py
The script will generate training instances for each task in H-TEST and save them in the specified format in the train/ directory.
Citation
If you use this code or the H-TEST benchmark in your research, please cite our paper:
@misc{lee2024language,
title={Language Models Don't Learn the Physical Manifestation of Language},
author={Bruce W. Lee and JaeHyuk Lim},
year={2024},
eprint={2402.11349},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
License
This project is licensed under the MIT License. See the LICENSE file for more information.
Owner
- Name: Bruce W. Lee (이웅성)
- Login: brucewlee
- Kind: user
- Location: Philadelphia, PA
- Company: University of Pennsylvania
- Website: brucewlee.github.io
- Repositories: 3
- Profile: https://github.com/brucewlee
Research Scientist - NLP
GitHub Events
Total
Last Year
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Bruce Lee | w****e@g****m | 2 |
| Bruce Lee | b****s@s****u | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 9 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0