https://github.com/carperai/cheese
Used for adaptive human in the loop evaluation of language and embedding models.
Science Score: 33.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: zenodo.org -
✓Committers with academic emails
2 of 8 committers (25.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.3%) to scientific vocabulary
Keywords from Contributors
Repository
Used for adaptive human in the loop evaluation of language and embedding models.
Basic Info
- Host: GitHub
- Owner: CarperAI
- License: mit
- Language: Python
- Default Branch: main
- Size: 500 KB
Statistics
- Stars: 308
- Watchers: 11
- Forks: 25
- Open Issues: 3
- Releases: 1
Metadata Files
README.md
Coadaptive Harness for Effective Evaluation, Steering, & Enhancement
Used for adaptive human in the loop evaluation of language and embedding models.
Getting Started
First install the required packages:
git clone https://github.com/carperai/cheese
cd cheese
pip install -r requirements.txt
You need rabbitmq for the messaging system in the backend. It can be installed easily with conda as follows:
conda install -c conda-forge rabbitmq-server
Then start the server (should do this on a different screen/window)
rabbitmq-server
Note that because of how rabbitmq works, it is possible several items may remain in the queue after your application using CHEESE has terminated. If the server is not shut down and restarted, then subsequent uses of CHEESE may end up drawing from items that were placed in the queue in a previous run.
As an example, the following script runs an image selection task, where two images from LAION-art dataset are presented and the labeller is asked to choose one.
python -m examples.image_selection
Custom tasks in Gradio
Refer to examples/image_selection.py for a comprehensive example on how to create your own custom task with a Gradio UI.
Owner
- Name: CarperAI
- Login: CarperAI
- Kind: organization
- Repositories: 15
- Profile: https://github.com/CarperAI
GitHub Events
Total
- Watch event: 9
- Fork event: 1
Last Year
- Watch event: 9
- Fork event: 1
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| s | s****1@g****m | 117 |
| Louis Castricato | w****p@g****m | 12 |
| Emerson Havener | e****n@n****u | 10 |
| fugitive-cat | 8****t | 8 |
| Shahbuland Matiana | s****n@u****a | 2 |
| asmith26 | a****6 | 1 |
| Kastan Day | k****y@g****m | 1 |
| Ayush Thakur | m****k@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 16
- Total pull requests: 37
- Average time to close issues: 18 days
- Average time to close pull requests: 10 days
- Total issue authors: 7
- Total pull request authors: 9
- Average comments per issue: 1.75
- Average comments per pull request: 0.16
- Merged pull requests: 34
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- shahbuland (10)
- asmith26 (1)
- gsarti (1)
- aashay96 (1)
- svjack (1)
- KastanDay (1)
- mshuffett (1)
Pull Request Authors
- shahbuland (22)
- ehavener (5)
- jfdoming (2)
- fugitive-cat (2)
- asmith26 (1)
- uSaiPrashanth (1)
- KastanDay (1)
- LouisCastricato (1)
- ayulockin (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- altair *
- b_rabbit *
- datasets *
- gradio *
- joblib *
- sphinx ==4.0.0
- sphinx_rtd_theme *
- webdataset *
- altair *
- b-rabbit *
- datasets *
- gradio *
- jinja2 *
- joblib *
- markupsafe *
- pandas *