https://github.com/cliangyu/med_eval
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (5.7%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: cliangyu
- License: mit
- Language: Python
- Default Branch: main
- Size: 1.73 MB
Statistics
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Medical QA Evaluation
This is the evaluation code for the medical QA tasks.
Features
- Evaluate open-ended and closed-ended questions.
- Evaluate on huggingface datasets.
- Evaluate with vllm-supported LLMs.
Requirements
``` pip install -r requirements.txt
install latest vllm
pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1x8664.whl ```
Datasets
All are huggingface datasets.
Except for uBench, each instance consists of an image-question-answer triplet.
python
{
'image': <PIL.JpegImagePlugin.JpegImageFile image mode=CMYK size=309x272>,
'question': 'where are liver stem cells (oval cells) located?',
'answer': 'in the canals of hering'
}
For uBench, each instance is a dictionary with multiple choice questions.
Usage
```
output the answer to a file and then evaluate the answer file in eval.py
cd .. CUDAVISIBLEDEVICES=0 python -m medeval.inferevalsubcellular --modelidorpath /mnt/disks/persistent-disk/code/swift/output/llava-onevision-qwen2-05b-ov/v12-20241204-034303/checkpoint-123 --maxpixels 50176
evaluate the saved answer file to get the results
python -m medeval.eval --datasetname vqa-rad --predfileparentpath answer-file-qwen2-vl-2b-instruct.jsonl
python -m medeval.eval --datasetname ubench --predfileparentpath answer-file-qwen2-vl-2b-instruct.jsonl
``
The question is closed-ended if the groundtruth answer is one of the options [yes, no].
All predictions and groundtruths are in lowercase for evaluation.
For open-ended questions, the evaluation metric is recall (functioncalculate_f1score`).
For closed-ended questions, the evaluation metric is accuracy (check if 'yes' or 'no' in the response).
Owner
- Name: Liangyu Chen
- Login: cliangyu
- Kind: user
- Location: Singapore
- Company: Nanyang Technological University
- Website: cliangyu.com
- Twitter: cliangyu_
- Repositories: 1
- Profile: https://github.com/cliangyu
GitHub Events
Total
- Watch event: 2
- Push event: 6
- Create event: 2
Last Year
- Watch event: 2
- Push event: 6
- Create event: 2
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- datasets *
- pillow *
- swift *
- torch *
- tqdm *