Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bowang-lab/clinical-camel

https://github.com/bowang-lab/clinical-camel

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.5%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: bowang-lab
  • License: agpl-3.0
  • Language: Python
  • Default Branch: main
  • Size: 188 KB
Statistics
  • Stars: 50
  • Watchers: 9
  • Forks: 5
  • Open Issues: 5
  • Releases: 0
Created about 3 years ago · Last pushed about 3 years ago
Metadata Files
Readme License

README.md

Clinical Camel

⚠️ Upcoming Code Release ⚠️

The codebase related to model inference, training, evaluation, and DBKE is under preparation and will be released soon. Please stay tuned for updates!

Model Description

Clinical Camel model is a transformer-based language model trained on the LLaMA 13B architecture. It is specially designed as a reseach focused medical conversational model.

Access the live demo. This corresponds to an earlier version of the Clinical Camel model.

Model Conversion

Delta weights for the model are provided. The apply_delta script from FastChat can be used to convert LLaMA-13B to Clinical Camel.

The delta weights can be found at this Hugging Face link.

Data

The model was trained on a diverse dataset which includes: - 100,000 synthetic dialogues produced via dialogue-based knowledge encoding (DBKE). - 10,187 USMLE questions which were converted via DBKE. - The ShareGPT dataset was also used, adding further diversity to the training data.

Training

The training code and inference model are based on FastChat. We would like to extend our gratitude to the developers of FastChat for making their code available for use.

The model was trained for 2 epochs. The specific training parameters and configurations used for the Clinical Camel model are detailed below:

| Parameter | Value | |:-------------:|:---------:| | Learning Rate | 2E-5 | | Batch Size | 4 | | Epochs | 2 | | Optimizer | AdamW (Torch) | | Max Gradient Norm | 1 | | Weight Decay | 0 | | Warm-up Steps | 0 | | Warm-up Ratio | 0.03 | | Gradient Accumulation Steps | 8 | | Per Device Training Batch Size | 4 | | Maximum Sequence Length | 2048 | | Learning Rate Scheduler | Cosine |

Model Comparison

The Clinical Camel model was compared to several other popular models in a variety of benchmarks. These include USMLE self-assessment scores, performance on multi-step management problems, and responses to standardized safety questions.

USMLE Self-Assessment Scores

In the USMLE Self-Assessment, Clinical Camel achieved the highest score on Step 1 and Step 3.

| | GPT-3.5 (%) | Clinical Camel (%) | Chat Doctor (%) | PMC-LLaMA | MedAlpaca 13b (%) | Vicuna (%) | |:----:|:----:|:----:|:----:|:----:|:----:|:----:| | Step 1 | 36.1 | 53.2 | 11.7 | 1.1 | 12.2 | 21.3 | | Step 2 | 56.9 | 51.4 | 18.5 | 2.7 | 27.5 | 20.4 | | Step 3 | 55.7 | 58.2 | 14.8 | 3.3 | 26.2 | 30.3 |

Multi-Step Management Problems

Clinical Camel shows competitive performance in multi-step management problems.

| | GPT-3.5 (%) | Clinical Camel (%) | Chat Doctor (%) | PMC-LLaMA | MedAlpaca 13b (%) | Vicuna (%) | |:----:|:----:|:----:|:----:|:----:|:----:|:----:| | CFPC EM | 81.3 | 74.9 | 53.2 | 13.9 | 31.2 | 68.6 | | CFPC FM | 85.0 | 82.1 | 67.0 | 14.3 | 44.7 | 74.8 |

Publication

The work related to the Clinical Camel model has been published in arXiv. Feel free to check out the paper for a more comprehensive understanding of the model and its performance.

Future Updates

We are in the process of preparing the remainder of the code used in this project for publication. Please check back in the near future for additional updates and resources. Your patience is appreciated as we work to provide a comprehensive and usable repository.

License

This project is licensed under AGPL-3.0. Please see the LICENSE file for more details.

Owner

  • Name: WangLab @ U of T
  • Login: bowang-lab
  • Kind: organization
  • Location: 190 Elizabeth St, Toronto, ON M5G 2C4 Canada

BoWang's Lab at University of Toronto

GitHub Events

Total
  • Watch event: 2
  • Fork event: 1
Last Year
  • Watch event: 2
  • Fork event: 1

Committers

Last synced: 12 months ago

All Time
  • Total Commits: 9
  • Total Committers: 1
  • Avg Commits per committer: 9.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Augustin Toma a****a@g****m 9

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 6
  • Total pull requests: 0
  • Average time to close issues: about 2 hours
  • Average time to close pull requests: N/A
  • Total issue authors: 5
  • Total pull request authors: 0
  • Average comments per issue: 0.33
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 2
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • th789 (2)
  • sean0042 (1)
  • delip (1)
  • hyesunyun (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels