Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/aehrc/imageclefmedical_caption_23

MedICap: Code for the participation of team CSIRO at the ImageCLEFmedical Caption task of 2023.

https://github.com/aehrc/imageclefmedical_caption_23

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.3%) to scientific vocabulary

Keywords

image-captioning medical-image-captioning medical-imaging multimodal multimodal-learning report-generation
Last synced: 6 months ago · JSON representation

Repository

MedICap: Code for the participation of team CSIRO at the ImageCLEFmedical Caption task of 2023.

Basic Info
Statistics
  • Stars: 3
  • Watchers: 7
  • Forks: 0
  • Open Issues: 3
  • Releases: 0
Topics
image-captioning medical-image-captioning medical-imaging multimodal multimodal-learning report-generation
Created over 2 years ago · Last pushed over 1 year ago
Metadata Files
Readme License

README.md

MedICap: A Concise Model for Medical Image Captioning

MedICap is a medical image captioning model that placed first in the ImageCLEFmedical Caption 2023 challenge: https://www.imageclef.org/2023/medical/caption (team CSIRO). It is available on the Hugging Face Hub: https://huggingface.co/aehrc/medicap. It is presented in the working notes and at CLEF 2023.

Working notes:

https://www.dei.unipd.it/~faggioli/temp/CLEF2023-proceedings/paper-132.pdf

BibTeX:

bibtex @inproceedings{nicolson_aehrc_2021, address = {Thessaloniki, Greece}, title = {A {C}oncise {M}odel for {M}edical {I}mage {C}aptioning}, copyright = {All rights reserved}, language = {en}, booktitle = {Proceedings of the 14th {International} {Conference} of the {CLEF} {Association}}, author = {Nicolson, Aaron and Dowling, Jason and Koopman, Bevan}, month = sep, year = {2023}, }

|| |----| |

Decoder conditioned on the visual features of the image via A) the cross-attention, and B) the self-attention. The visual features are extracted with the encoder. CC BY [Muacevic et al. (2022)]. 𝑁 is the number of Transformer blocks. [BOS] is the beginning-of-sentence special token.

|

Hugging Face model & checkpoint:

The Hugging Face model & checkpoint is available at: https://huggingface.co/aehrc/medicap.

Notebook example:

An example of MedICap generating captions is given in example.ipynb.

Installation:

After cloning the repository, install the required packages in a virtual environment. The required packages are located in requirements.txt: shell script python -m venv --system-site-packages venv source venv/bin/activate python -m pip install --upgrade pip python -m pip install --upgrade -r requirements.txt --no-cache-dir

Test the Hugging Face checkpoints:

To test the Hugging Face model:

shell dlhpcstarter -t imageclefmed_caption_2023_hf -c config/test_huggingface/007_no_ca_scst.yaml --stages_module tools.stages --test

See dlhpcstarter==0.1.4 for more options.

Note: data will be saved in the experiment directory (exp_dir in the configuration file).

Training:

To train with teacher forcing:

dlhpcstarter -t imageclefmed_caption_2023 -c config/train/002_no_ca.yaml --stages_module tools.stages --train

The model can then be tested with the --test flag:

dlhpcstarter -t imageclefmed_caption_2023 -c config/train/002_no_ca.yaml --stages_module tools.stages --test

To then train with Self-Critical Sequence Training (SCST) with the BERTScore reward:

  1. Copy the path to the checkpoint from the exp_dir for the configuration above, then paste it in the configuration for SCST as warm_start_ckpt_path, then:
  2. dlhpcstarter -t mimic_cxr -c config/train/007_no_ca_scst.yaml --stages_module tools.stages --train

See dlhpcstarter==0.1.4 for more options.

Help/Issues:

If you need help, or if there are any issues, please leave an issue and we will get back to you as soon as possible.

Owner

  • Name: The Australian e-Health Research Centre
  • Login: aehrc
  • Kind: organization

The Australian e-Health Research Centre (AEHRC) is CSIRO’s digital health research program.

GitHub Events

Total
  • Issues event: 1
  • Watch event: 2
  • Push event: 1
  • Fork event: 1
Last Year
  • Issues event: 1
  • Watch event: 2
  • Push event: 1
  • Fork event: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 4
  • Total pull requests: 0
  • Average time to close issues: 9 months
  • Average time to close pull requests: N/A
  • Total issue authors: 4
  • Total pull request authors: 0
  • Average comments per issue: 0.5
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 2
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 2
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • flipmeixner (1)
  • Rajagopalhertzian (1)
  • edcand3ball (1)
  • Reckless0 (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels