Data

Tools

Indexes

Applications

Experiments

Open Source Science

mad

https://github.com/hanqiu-hq/mad

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.6%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Basic Info
  • Host: GitHub
  • Owner: hanqiu-hq
  • License: other
  • Language: Python
  • Default Branch: main
  • Size: 3.07 MB
Statistics
  • Stars: 13
  • Watchers: 3
  • Forks: 0
  • Open Issues: 2
  • Releases: 0
Created over 2 years ago · Last pushed almost 2 years ago
Metadata Files
Readme License Citation

README.md

Masked AutoDecoder

This is the official implementation of the paper "Masked AutoDecoder is Effective Multi-Task Vision Generalist"

Authors: Han Qiu, Jiaxing Huang, Peng Gao, Lewei Lu, Xiaoqin Zhang, Shijian Lu

In this work, we design Masked AutoDecoder(MAD), an effective multi-task vision generalist. MAD consists of two core designs. First, we develop a parallel decoding framework that introduces bi-directional attention to capture contextual dependencies comprehensively and decode vision task sequences in parallel. Second, we design a masked sequence modeling approach that learns rich task contexts by masking and reconstructing task sequences. In this way, MAD handles all the tasks by a single network branch and a simple cross-entropy loss with minimal task-specific designs.


Installation

First Install Detectron2.

Then, bash cd MAD pip install -e .

Please refer to Installation Instructions of Detrex for details of installation.

Data Preparation

```bash

First prepare COCO dataset at "./datasets" as following:

  • datasets
    • coco
    • annotation
      • captions_train2017.json
      • captions_val2017.json
      • instances_train2017.json
      • instances_val2017.json
      • personkeypointstrain2017.json
      • personkeypointsval2017.json
    • train2017
    • val2017

merge the keypoint anno and coco instance anno

python ./project/mad/data/merge_annotations.py ```

Training

bash python tools/train_net.py --num-gpus 8 --dist-url auto --config-file ./project/mad/model/config.py

Evaluation

bash python tools/train_net.py --num-gpus 1 --dist-url auto --config-file ./project/mad/model/config.py --eval-only

Acknowledgement

We build MAD based on detrex.

Citation

If you find our work helpful please cite: BibTex @InProceedings{Qiu_2024_CVPR, author = {Qiu, Han and Huang, Jiaxing and Gao, Peng and Lu, Lewei and Zhang, Xiaoqin and Lu, Shijian}, title = {Masked AutoDecoder is Effective Multi-Task Vision Generalist}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {14152-14161} }

Owner

  • Name: Han Qiu
  • Login: hanqiu-hq
  • Kind: user

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
  - name: "detrex Contributors"
title: "IDEA-CVR Detection-Transformer Toolbox and Benchmark"
date-released: 2022-09-21
url: "https://github.com/IDEA-Research/detrex"
license: Apache-2.0

GitHub Events

Total
  • Issues event: 1
  • Watch event: 4
Last Year
  • Issues event: 1
  • Watch event: 4