Transformers--CLIPSeg

This is the code to train CLIPSeg based on hugging face transformers

For the training file, please go to /examples/pytorch/contrastive-image-text/run_clipseg.py

Also some changes in modeling_Clipseg.py

To run Clipseg, please follow this code:

bash python examples/pytorch/contrastive-image-text/run_clipseg.py \ --output_dir "clipseg.." \ --model_name_or_path "CIDAS/clipseg-rd64-refined" \ --feature_extractor_name "CIDAS/clipseg-rd64-refined"\ --image_column "image_path" \ --caption_column "seg_class_name" \ --label_column "mask_path"\ --train_file "../train_instruments.json" \ --validation_file "../valid_instruments.json"" \ --test_file "../test_instruments.json"" \ --max_seq_length 77 \ --remove_unused_columns=False \ --do_train \ --per_device_train_batch_size 24 \ --per_device_eval_batch_size 24 \ --num_train_epochs 400 \ --learning_rate "5e-4" \ --warmup_steps 0 \ --weight_decay 0.1 \ --overwrite_output_dir \ --report_to none

CLIPSeg training summary

CLIPSeg is another model that we want to try to leverage the text/visual prompts to help with our instruments segmentation task. The CLIPSeg can be served for: 1) Referring Expression Segmentation; 2) Generalized Zero-Shot Segmentation; 3) One-Shot Semantic Segmentation

Experiment 1: Training CLIPSeg for EndoVis2017 with Text prompt only

Training stage input: - Query image (samples in EndoVis2017 training set) - Text prompt (segmentation class name/ segmentation class description) Experiment 1.1: Segmentation class name example: ["Bipolar Forceps"] Experiment 1.2: Segmentation class description example: [“Bipolar forceps with double-action fine curved jaws and horizontal serrations, made by medical grade stainless stell and Surgical grade material, includes a handle and a dark or grey plastic like cylindrical shaft, includes a complex robotic joint for connecting the jaws/handle to the shaft”]

Testing stage:

• Input: sample in EndoVis2017 testing set; Text prompt
• Output example (binary) for experiment 1.1: doesn’t work ☹

• Output example (binary) for experiment 1.2: works but results are very similar to the pre-trained CLIPSeg

• In EndoVis2017 testing set: Experiment 1.2: mean IOU= 79.92%

Experiment 2: Training CLIPSeg for EndoVis2017 with randomly mix text and visual support conditionals

Training stage:

• Input:
• Query image (samples in EndoVis2017 training set)
• Text prompt (segmentation class description) Segmentation class description example is the same as described in experiment 1.2 -Visual prompt Using the visual prompting tips described in the paper, i.e. cropping the image and darkening the background.

Testing stage:

• Input: sample in EndoVis2017 testing set; Text prompt

• Output Example:

• In EndoVis2017 testing set: Experiment 1.2: mean IOU= 81.92% (not much improvement)

Ongoing Experiment: Fine-tuning CLIP as well as training CLIPSeg decoder