Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

Dolphin (Document Image Parsing via Heterogeneous Anchor Prompting) is a novel multimodal document image parsing model following an analyze-then-parse paradigm. This repository contains the demo code and pre-trained models for Dolphin.

📑 Overview

Document image parsing is challenging due to its complexly intertwined elements such as text paragraphs, figures, formulas, and tables. Dolphin addresses these challenges through a two-stage approach:

1. 🔍 Stage 1: Comprehensive page-level layout analysis by generating element sequence in natural reading order
2. 🧩 Stage 2: Efficient parallel parsing of document elements using heterogeneous anchors and task-specific prompts

Dolphin achieves promising performance across diverse page-level and element-level parsing tasks while ensuring superior efficiency through its lightweight architecture and parallel parsing mechanism.

🚀 Demo

Try our demo on Demo-Dolphin.

📅 Changelog

• 🔥 2025.07.10 Released the Fox-Page Benchmark, a manually refined subset of the original Fox dataset. Download via: Baidu Yun | Google Drive.
• 🔥 2025.06.30 Added TensorRT-LLM support for accelerated inference！
• 🔥 2025.06.27 Added vLLM support for accelerated inference！
• 🔥 2025.06.13 Added multi-page PDF document parsing capability.
• 🔥 2025.05.21 Our demo is released at link. Check it out!
• 🔥 2025.05.20 The pretrained model and inference code of Dolphin are released.
• 🔥 2025.05.16 Our paper has been accepted by ACL 2025. Paper link: arXiv.

🛠️ Installation

1. Clone the repository: bash git clone https://github.com/ByteDance/Dolphin.git cd Dolphin

2. Install the dependencies: bash pip install -r requirements.txt

3. Download the pre-trained models using one of the following options:

Option A: Original Model Format (config-based)

Download from Baidu Yun or Google Drive and put them in the ./checkpoints folder.

Option B: Hugging Face Model Format

Visit our Huggingface model card, or download model by:

bash # Download the model from Hugging Face Hub git lfs install git clone https://huggingface.co/ByteDance/Dolphin ./hf_model # Or use the Hugging Face CLI pip install huggingface_hub huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model

⚡ Inference

Dolphin provides two inference frameworks with support for two parsing granularities: - Page-level Parsing: Parse the entire document page into a structured JSON and Markdown format - Element-level Parsing: Parse individual document elements (text, table, formula)

📄 Page-level Parsing

Using Original Framework (config-based)

```bash

Process a single document image

python demopage.py --config ./config/Dolphin.yaml --inputpath ./demo/pageimgs/page1.jpeg --save_dir ./results

Process a single document pdf

python demopage.py --config ./config/Dolphin.yaml --inputpath ./demo/pageimgs/page6.pdf --save_dir ./results

Process all documents in a directory

python demopage.py --config ./config/Dolphin.yaml --inputpath ./demo/pageimgs --savedir ./results

Process with custom batch size for parallel element decoding

python demopage.py --config ./config/Dolphin.yaml --inputpath ./demo/pageimgs --savedir ./results --maxbatchsize 8 ```

Using Hugging Face Framework

```bash

Process a single document image

python demopagehf.py --modelpath ./hfmodel --inputpath ./demo/pageimgs/page1.jpeg --savedir ./results

Process a single document pdf

python demopagehf.py --modelpath ./hfmodel --inputpath ./demo/pageimgs/page6.pdf --savedir ./results

Process all documents in a directory

python demopagehf.py --modelpath ./hfmodel --inputpath ./demo/pageimgs --save_dir ./results

Process with custom batch size for parallel element decoding

python demopagehf.py --modelpath ./hfmodel --inputpath ./demo/pageimgs --savedir ./results --maxbatch_size 16 ```

🧩 Element-level Parsing

Using Original Framework (config-based)

```bash

Process a single table image

python demoelement.py --config ./config/Dolphin.yaml --inputpath ./demo/elementimgs/table1.jpeg --element_type table

Process a single formula image

python demoelement.py --config ./config/Dolphin.yaml --inputpath ./demo/elementimgs/lineformula.jpeg --element_type formula

Process a single text paragraph image

python demoelement.py --config ./config/Dolphin.yaml --inputpath ./demo/elementimgs/para1.jpg --element_type text ```

Using Hugging Face Framework

```bash

Process a single table image

python demoelementhf.py --modelpath ./hfmodel --inputpath ./demo/elementimgs/table1.jpeg --elementtype table

Process a single formula image

python demoelementhf.py --modelpath ./hfmodel --inputpath ./demo/elementimgs/lineformula.jpeg --elementtype formula

Process a single text paragraph image

python demoelementhf.py --modelpath ./hfmodel --inputpath ./demo/elementimgs/para1.jpg --elementtype text ```

🌟 Key Features

• 🔄 Two-stage analyze-then-parse approach based on a single VLM
• 📊 Promising performance on document parsing tasks
• 🔍 Natural reading order element sequence generation
• 🧩 Heterogeneous anchor prompting for different document elements
• ⏱️ Efficient parallel parsing mechanism
• 🤗 Support for Hugging Face Transformers for easier integration

📮 Notice

Call for Bad Cases: If you have encountered any cases where the model performs poorly, we would greatly appreciate it if you could share them in the issue. We are continuously working to optimize and improve the model.

💖 Acknowledgement

We would like to acknowledge the following open-source projects that provided inspiration and reference for this work: - Donut - Nougat - GOT - MinerU - Swin - Hugging Face Transformers

📝 Citation

If you find this code useful for your research, please use the following BibTeX entry.

bibtex @article{feng2025dolphin, title={Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting}, author={Feng, Hao and Wei, Shu and Fei, Xiang and Shi, Wei and Han, Yingdong and Liao, Lei and Lu, Jinghui and Wu, Binghong and Liu, Qi and Lin, Chunhui and others}, journal={arXiv preprint arXiv:2505.14059}, year={2025} }

Star History