Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bytedance/e2str

The official code for the CVPR 2024 paper: Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer

https://github.com/bytedance/e2str

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.2%) to scientific vocabulary

Keywords

research
Last synced: 10 months ago · JSON representation

Repository

The official code for the CVPR 2024 paper: Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer

Basic Info
  • Host: GitHub
  • Owner: bytedance
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 849 KB
Statistics
  • Stars: 53
  • Watchers: 5
  • Forks: 4
  • Open Issues: 0
  • Releases: 0
Topics
research
Created over 2 years ago · Last pushed about 2 years ago
Metadata Files
Readme License

readme.md

E2STR

The official implementation of E2STR: Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer (CVPR-2024) PDF

environment

  1. install mmocr 1.0.0

  2. install requirements.txt

data & model

  1. Download Union14M-L from Union14M-L

  2. Download the MAE pretrained ViT weight from MAERec

  3. Download OPT-125M

  4. Download all the test dataset (listed in Table 1 and Table 2) and modify all the dataroot in configs/textrecog/base_/datasets. We may upload them later.

  5. The 600k training data with character-wise annotations will be available later. But currently, the repository can also run well without this training data (i.e., you can perform in-context training with only Transform Strategy by modifying 'JSON FILE FOR CHARACTER-WISE POSITION INFORMATION' as None). Also refer to Table 4.

train

  1. stage1: vanilla STR training

modify 'MAE PRETRAIN WEIGHT PATH' / 'LM WEIGHT PATH' / 'CHECKPOINT SAVE PATH' / 'SAVENAME' in configs/textrecog/iclocr/stage1.py

sh run_stage1.sh

  1. stage2: in-context training

modify 'STAGE-1 WEIGHT PATH' / 'LM WEIGHT PATH' / 'JSON FILE FOR CHARACTER-WISE POSITION INFORMATION' / 'CHECKPOINT SAVE PATH' / 'SAVENAME' in configs/textrecog/iclocr/stage2.py

sh run_stage2.sh

evaluate

  1. Construct the in-context pool (i.e., a json file) by randomly sample data from any target training set. The json file should be structured as follows:

[ { 'img_path': , 'gt_text': } ]

  1. Modify 'JSON FILE FOR IN-CONTEXT POOL' in configs/textrecog/icl_ocr/stage2.py

  2. Run the following command to evaluate the model.

bash tools/dist_test.sh ./configs/textrecog/icl_ocr/S-stage2.py 'STAGE2-CHECKPOINT-PATH' 8

Citation

If you find our models / code / papers useful in your research, please consider giving stars ⭐ and citations 📝

@article{zhao2023multi, title={Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer}, author={Zhao, Zhen and Huang, Can and Wu, Binghong and Lin, Chunhui and Liu, Hao and Zhang, Zhizhong and Tan, Xin and Tang, Jingqun and Xie, Yuan}, journal={CVPR}, year={2024} }

Owner

  • Name: Bytedance Inc.
  • Login: bytedance
  • Kind: organization
  • Location: Singapore

GitHub Events

Total
  • Watch event: 10
Last Year
  • Watch event: 10

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 4
  • Total Committers: 2
  • Avg Commits per committer: 2.0
  • Development Distribution Score (DDS): 0.25
Past Year
  • Commits: 4
  • Committers: 2
  • Avg Commits per committer: 2.0
  • Development Distribution Score (DDS): 0.25
Top Committers
Name Email Commits
zhaozhen.123 z****3@b****m 3
JohnTang93 2****2@q****m 1
Committer Domains (Top 20 + Academic)
qq.com: 1 bytedance.com: 1

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 4
  • Total pull requests: 0
  • Average time to close issues: about 2 months
  • Average time to close pull requests: N/A
  • Total issue authors: 4
  • Total pull request authors: 0
  • Average comments per issue: 3.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 4
  • Pull requests: 0
  • Average time to close issues: about 2 months
  • Average time to close pull requests: N/A
  • Issue authors: 4
  • Pull request authors: 0
  • Average comments per issue: 3.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • lyb18758 (1)
  • 1193700079 (1)
  • Z-yolo (1)
  • siddiquemu (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

requirements.txt pypi
  • Jinja2 ==3.1.3
  • Levenshtein ==0.23.0
  • Markdown ==3.5.2
  • MarkupSafe ==2.1.4
  • PyWavelets ==1.4.1
  • PyYAML ==6.0.1
  • Pygments ==2.17.2
  • accelerate ==0.26.1
  • addict ==2.4.0
  • albumentations ==1.3.1
  • aliyun-python-sdk-core ==2.14.0
  • aliyun-python-sdk-kms ==2.16.2
  • asynctest ==0.13.0
  • attrs ==23.2.0
  • certifi ==2023.11.17
  • cffi ==1.16.0
  • charset-normalizer ==3.3.2
  • click ==8.1.7
  • cmake ==3.28.1
  • codecov ==2.1.13
  • colorama ==0.4.6
  • contourpy ==1.1.1
  • coverage ==7.4.0
  • crcmod ==1.7
  • cryptography ==42.0.0
  • cycler ==0.12.1
  • einops ==0.7.0
  • einops-exts ==0.0.4
  • exceptiongroup ==1.2.0
  • filelock ==3.13.1
  • flake8 ==7.0.0
  • fonttools ==4.47.2
  • fsspec ==2023.12.2
  • ftfy ==6.1.3
  • huggingface-hub ==0.20.3
  • idna ==3.6
  • imageio ==2.33.1
  • imgaug ==0.4.0
  • importlib-metadata ==7.0.1
  • importlib-resources ==6.1.1
  • iniconfig ==2.0.0
  • interrogate ==1.5.0
  • isort ==5.13.2
  • jmespath ==0.10.0
  • joblib ==1.3.2
  • kiwisolver ==1.4.5
  • kwarray ==0.6.16
  • lanms_neo ==1.0.2
  • lazy_loader ==0.3
  • lit ==17.0.6
  • lmdb ==1.4.1
  • loralib ==0.1.2
  • markdown-it-py ==3.0.0
  • matplotlib ==3.7.4
  • mccabe ==0.7.0
  • mdurl ==0.1.2
  • mmcv ==2.0.1
  • mmdet ==3.0.0
  • mmengine ==0.10.3
  • mmocr ==1.0.0
  • model-index ==0.1.11
  • mpmath ==1.3.0
  • networkx ==3.1
  • numpy ==1.24.4
  • nvidia-cublas-cu11 ==11.10.3.66
  • nvidia-cuda-cupti-cu11 ==11.7.101
  • nvidia-cuda-nvrtc-cu11 ==11.7.99
  • nvidia-cuda-runtime-cu11 ==11.7.99
  • nvidia-cudnn-cu11 ==8.5.0.96
  • nvidia-cufft-cu11 ==10.9.0.58
  • nvidia-curand-cu11 ==10.2.10.91
  • nvidia-cusolver-cu11 ==11.4.0.1
  • nvidia-cusparse-cu11 ==11.7.4.91
  • nvidia-nccl-cu11 ==2.14.3
  • nvidia-nvtx-cu11 ==11.7.91
  • open-clip-torch ==2.24.0
  • open-flamingo ==2.0.1
  • opencv-python ==4.9.0.80
  • opendatalab ==0.0.10
  • openmim ==0.3.9
  • openxlab ==0.0.34
  • ordered-set ==4.1.0
  • oss2 ==2.17.0
  • packaging ==23.2
  • pandas ==2.0.3
  • parameterized ==0.9.0
  • pillow ==10.2.0
  • platformdirs ==4.1.0
  • pluggy ==1.4.0
  • protobuf ==4.25.2
  • psutil ==5.9.8
  • py ==1.11.0
  • pyclipper ==1.3.0.post5
  • pycocotools ==2.0.7
  • pycodestyle ==2.11.1
  • pycparser ==2.21
  • pycryptodome ==3.20.0
  • pyflakes ==3.2.0
  • pyparsing ==3.1.1
  • pytest ==7.4.4
  • pytest-cov ==4.1.0
  • pytest-runner ==6.0.1
  • python-dateutil ==2.8.2
  • pytz ==2023.3.post1
  • qudida ==0.0.4
  • rapidfuzz ==3.6.1
  • regex ==2023.12.25
  • requests ==2.28.2
  • rich ==13.4.2
  • safetensors ==0.4.2
  • scikit-image ==0.21.0
  • scikit-learn ==1.3.2
  • scipy ==1.10.1
  • sentencepiece ==0.1.98
  • shapely ==2.0.2
  • six ==1.16.0
  • sympy ==1.12
  • tabulate ==0.9.0
  • termcolor ==2.4.0
  • terminaltables ==3.1.10
  • threadpoolctl ==3.2.0
  • tifffile ==2023.7.10
  • timm ==0.9.12
  • tokenizers ==0.15.1
  • toml ==0.10.2
  • tomli ==2.0.1
  • torch ==2.0.1
  • torchvision ==0.15.2
  • tqdm ==4.65.2
  • transformers ==4.37.1
  • triton ==2.0.0
  • typing_extensions ==4.9.0
  • tzdata ==2023.4
  • ubelt ==1.3.4
  • urllib3 ==1.26.18
  • wcwidth ==0.2.13
  • xdoctest ==1.1.2
  • yapf ==0.40.2
  • zipp ==3.17.0
tools/requirements.txt pypi
  • Jinja2 ==3.1.3
  • Levenshtein ==0.23.0
  • Markdown ==3.5.2
  • MarkupSafe ==2.1.4
  • PyWavelets ==1.4.1
  • PyYAML ==6.0.1
  • Pygments ==2.17.2
  • accelerate ==0.26.1
  • addict ==2.4.0
  • albumentations ==1.3.1
  • aliyun-python-sdk-core ==2.14.0
  • aliyun-python-sdk-kms ==2.16.2
  • asynctest ==0.13.0
  • attrs ==23.2.0
  • certifi ==2023.11.17
  • cffi ==1.16.0
  • charset-normalizer ==3.3.2
  • click ==8.1.7
  • cmake ==3.28.1
  • codecov ==2.1.13
  • colorama ==0.4.6
  • contourpy ==1.1.1
  • coverage ==7.4.0
  • crcmod ==1.7
  • cryptography ==42.0.0
  • cycler ==0.12.1
  • einops ==0.7.0
  • einops-exts ==0.0.4
  • exceptiongroup ==1.2.0
  • filelock ==3.13.1
  • flake8 ==7.0.0
  • fonttools ==4.47.2
  • fsspec ==2023.12.2
  • ftfy ==6.1.3
  • huggingface-hub ==0.20.3
  • idna ==3.6
  • imageio ==2.33.1
  • imgaug ==0.4.0
  • importlib-metadata ==7.0.1
  • importlib-resources ==6.1.1
  • iniconfig ==2.0.0
  • interrogate ==1.5.0
  • isort ==5.13.2
  • jmespath ==0.10.0
  • joblib ==1.3.2
  • kiwisolver ==1.4.5
  • kwarray ==0.6.16
  • lanms_neo ==1.0.2
  • lazy_loader ==0.3
  • lit ==17.0.6
  • lmdb ==1.4.1
  • loralib ==0.1.2
  • markdown-it-py ==3.0.0
  • matplotlib ==3.7.4
  • mccabe ==0.7.0
  • mdurl ==0.1.2
  • mmcv ==2.0.1
  • mmdet ==3.0.0
  • mmengine ==0.10.3
  • mmocr ==1.0.0
  • model-index ==0.1.11
  • mpmath ==1.3.0
  • networkx ==3.1
  • numpy ==1.24.4
  • nvidia-cublas-cu11 ==11.10.3.66
  • nvidia-cuda-cupti-cu11 ==11.7.101
  • nvidia-cuda-nvrtc-cu11 ==11.7.99
  • nvidia-cuda-runtime-cu11 ==11.7.99
  • nvidia-cudnn-cu11 ==8.5.0.96
  • nvidia-cufft-cu11 ==10.9.0.58
  • nvidia-curand-cu11 ==10.2.10.91
  • nvidia-cusolver-cu11 ==11.4.0.1
  • nvidia-cusparse-cu11 ==11.7.4.91
  • nvidia-nccl-cu11 ==2.14.3
  • nvidia-nvtx-cu11 ==11.7.91
  • open-clip-torch ==2.24.0
  • open-flamingo ==2.0.1
  • opencv-python ==4.9.0.80
  • opendatalab ==0.0.10
  • openmim ==0.3.9
  • openxlab ==0.0.34
  • ordered-set ==4.1.0
  • oss2 ==2.17.0
  • packaging ==23.2
  • pandas ==2.0.3
  • parameterized ==0.9.0
  • pillow ==10.2.0
  • platformdirs ==4.1.0
  • pluggy ==1.4.0
  • protobuf ==4.25.2
  • psutil ==5.9.8
  • py ==1.11.0
  • pyclipper ==1.3.0.post5
  • pycocotools ==2.0.7
  • pycodestyle ==2.11.1
  • pycparser ==2.21
  • pycryptodome ==3.20.0
  • pyflakes ==3.2.0
  • pyparsing ==3.1.1
  • pytest ==7.4.4
  • pytest-cov ==4.1.0
  • pytest-runner ==6.0.1
  • python-dateutil ==2.8.2
  • pytz ==2023.3.post1
  • qudida ==0.0.4
  • rapidfuzz ==3.6.1
  • regex ==2023.12.25
  • requests ==2.28.2
  • rich ==13.4.2
  • safetensors ==0.4.2
  • scikit-image ==0.21.0
  • scikit-learn ==1.3.2
  • scipy ==1.10.1
  • sentencepiece ==0.1.98
  • shapely ==2.0.2
  • six ==1.16.0
  • sympy ==1.12
  • tabulate ==0.9.0
  • termcolor ==2.4.0
  • terminaltables ==3.1.10
  • threadpoolctl ==3.2.0
  • tifffile ==2023.7.10
  • timm ==0.9.12
  • tokenizers ==0.15.1
  • toml ==0.10.2
  • tomli ==2.0.1
  • torch ==2.0.1
  • torchvision ==0.15.2
  • tqdm ==4.65.2
  • transformers ==4.37.1
  • triton ==2.0.0
  • typing_extensions ==4.9.0
  • tzdata ==2023.4
  • ubelt ==1.3.4
  • urllib3 ==1.26.18
  • wcwidth ==0.2.13
  • xdoctest ==1.1.2
  • yapf ==0.40.2
  • zipp ==3.17.0