Vision Document Retrieval (ViDoRe): Benchmarks 👀

---

[Model card] [ViDoRe Leaderboard] [Demo] [Blog Post]

Approach

The Visual Document Retrieval Benchmarks (ViDoRe v1 and v2), is introduced to evaluate the performance of document retrieval systems on visually rich documents across various tasks, domains, languages, and settings. It was used to evaluate the ColPali model, a VLM-powered retriever that efficiently retrieves documents based on their visual content and textual queries using a late-interaction mechanism.

⚠️ Deprecation Warning: Moving from vidore-benchmark to mteb

Since mteb now supports image-text retrieval, we recommend using mteb to evaluate your retriever on the ViDoRe benchmark. We are deprecating vidore-benchmark to facilitate maintenance and have a single source of truth for the ViDoRe benchmark.

If you want your results to appear on the ViDoRe Leaderboard, you should add them to the results Github Project. Check the Submit your model section of the ViDoRe Leaderboard for more information.

New Evaluation Process

Follow the instructions to setup mteb here. Then you have 2 options.

Option 1: CLI

bash mteb run -b "ViDoRe(v1)" -m "vidore/colqwen2.5-v0.2" mteb run -b "ViDoRe(v2)" -m "vidore/colqwen2.5-v0.2"

Option 2: Python Script

```python import mteb from mteb.modelmeta import ModelMeta from mteb.models.colqwenmodels import ColQwen2_5Wrapper

=== Configuration ===

MODEL_NAME = "johndoe/mycolqwen2.5" BENCHMARKS = ["ViDoRe(v1)", "ViDoRe(v2)"]

=== Model Metadata ===

custommodelmeta = ModelMeta( loader=ColQwen25Wrapper, name=MODELNAME, modalities=["image", "text"], framework="Colpali", similarityfnname="max_sim", # Optional metadata (fill in if available else None) ... )

=== Load Model ===

custommodel = custommodelmeta.loadmodel(MODEL_NAME)

=== Load Tasks ===

tasks = mteb.get_benchmarks(names=BENCHMARKS) evaluator = mteb.MTEB(tasks=tasks)

=== Run Evaluation ===

results = evaluator.run(custom_model) ```

For custom models, you should implement your own wrapper. Check the ColPaliEngineWrapper for an example.

[Deprecated] Usage

This packages comes with a Python API and a CLI to evaluate your own retriever on the ViDoRe benchmark. Both are compatible with Python>=3.9.

CLI mode

bash pip install vidore-benchmark

To keep this package lightweight, only the essential packages were installed. Thus, you must specify the dependency groups for models you want to evaluate with CLI (see the list in pyproject.toml). For instance, if you are going to evaluate the ColVision models (e.g. ColPali, ColQwen2, ColSmol, ...), you should run:

bash pip install "vidore-benchmark[colpali-engine]"

[!WARNING] If possible, do not pip install colpali-engine directly in the env dedicated for the CLI.

In particular, make sure not to install both vidore-benchmark[colpali-engine] and colpali-engine[train] simultaneously, as it will lead to a circular depencency conflict.

If you want to install all the dependencies for all the models, you can run:

bash pip install "vidore-benchmark[all-retrievers]"

Note that in order to use BM25Retriever, you will need to download the nltk resources too:

bash pip install "vidore-benchmark[bm25]" python -m nltk.downloader punkt punkt_tab stopwords

Library mode

Install the base package using pip:

bash pip install vidore-benchmark

Command-line usage

Evaluate a retriever on ViDoRE

You can evaluate any off-the-shelf retriever on the ViDoRe benchmark v1. For instance, you can evaluate the ColPali model on the ViDoRe benchmark 1 to reproduce the results from our paper.

bash vidore-benchmark evaluate-retriever \ --model-class colpali \ --model-name vidore/colpali-v1.3 \ --collection-name vidore/vidore-benchmark-667173f98e70a1c0fa4db00d \ --dataset-format qa \ --split test

If you want to evaluate your models on on new collection ViDoRe benchmark 2, a harder version of the previous benchmark you can execute the following command:

bash vidore-benchmark evaluate-retriever \ --model-class colpali \ --model-name vidore/colpali-v1.3 \ --collection-name vidore/vidore-benchmark-v2-67ae03e3924e85b36e7f53b0 \ --dataset-format beir \ --split test

Alternatively, you can evaluate your model on a single dataset. If your retriver uses visual embeddings, you can use any dataset path from the ViDoRe Benchmark v1 collection or the ViDoRe Benchmark v2 (beir format instead of qa), e.g.:

bash vidore-benchmark evaluate-retriever \ --model-class colpali \ --model-name vidore/colpali-v1.3 \ --dataset-name vidore/docvqa_test_subsampled \ --dataset-format qa \ --split test

If you want to evaluate a retriever that relies on pure-text retrieval (no visual embeddings), you should use the datasets from the ViDoRe Chunk OCR (baseline) instead:

bash vidore-benchmark evaluate-retriever \ --model-class bge-m3 \ --model-name BAAI/bge-m3 \ --dataset-name vidore/docvqa_test_subsampled_tesseract \ --dataset-format qa \ --split test

All the above scripts will generate a JSON file in outputs/{model_id}_metrics.json. Follow the instructions on the ViDoRe Leaderboard to learn how to publish your results on the leaderboard too!

[!NOTE] The vidore-benchmark package supports two formats of datasets:

• QA: The dataset is formatted as a question-answering task, where the queries are questions and the passages are the image pages that provide the answers.
• BEIR: Following the BEIR paper, the dataset is formatted in 3 sub-datasets: corpus, queries, and qrels. The corpus contains the documents, the queries contains the queries, and the qrels contains the relevance scores between the queries and the documents.

In the first iteration of the ViDoRe benchmark, we arbitrarily choose to deduplicate the queries for the QA datasets. While this made sense given our data generation process, it wasn't suited for our ViDoRe benchmark v2 which aims at being broader and multilingual. We will release the ViDoRe benchmark v2 soon.

| Dataset | Dataset format | Deduplicate queries | |------------------------------------------------------------------------------------------------------------|----------------|---------------------| | ViDoRe benchmark v1 | QA | ✅ | | ViDoRe benchmark v2 (harder/multilingual) | BEIR | ❌ |

Documentation

To have more control over the evaluation process (e.g. the batch size used at inference), read the CLI documentation using:

bash vidore-benchmark evaluate-retriever --help

In particular, feel free to play with the --batch-query, --batch-passage, --batch-score, and --num-workers inputs to speed up the evaluation process.

Python usage

Quickstart example

While the CLI can be used to evaluate a fixed list of models, you can also use the Python API to evaluate your own retriever. Here is an example of how to evaluate the ColPali model on the ViDoRe benchmark. Note that your processor must implement a process_images and a process_queries methods, similarly to the ColVision processors.

```python import torch from colpaliengine.models import ColIdefics3, ColIdefics3Processor from datasets import loaddataset from tqdm import tqdm

from vidorebenchmark.evaluation.vidoreevaluators import ViDoReEvaluatorQA, ViDoReEvaluatorBEIR from vidorebenchmark.retrievers import VisionRetriever from vidorebenchmark.utils.datautils import getdatasetsfromcollection

modelname = "vidore/colSmol-256M" processor = ColIdefics3Processor.frompretrained(modelname) model = ColIdefics3.frompretrained( modelname, torchdtype=torch.bfloat16, device_map="cuda", ).eval()

Get retriever instance

vision_retriever = VisionRetriever(model=model, processor=processor)

Evaluate on a single BEIR format dataset (e.g one of the ViDoRe benchmark 2 dataset)

vidoreevaluatorbeir = ViDoReEvaluatorBEIR(visionretriever) ds = { "corpus" : loaddataset("vidore/syntheticrserestaurantfilteredv1.0", name="corpus", split="test"), "queries" : loaddataset("vidore/syntheticrserestaurantfilteredv1.0", name="queries", split="test") "qrels" : loaddataset("vidore/syntheticrserestaurantfilteredv1.0", name="qrels", split="test") } metricsdatasetbeir = vidoreevaluatorbeir.evaluatedataset( ds=ds, batchquery=4, batchpassage=4, ) print(metricsdataset_beir)

Evaluate on a single QA format dataset

vidoreevaluatorqa = ViDoReEvaluatorQA(visionretriever) ds = loaddataset("vidore/tabfquadtestsubsampled", split="test") metricsdatasetqa = vidoreevaluatorqa.evaluatedataset( ds=ds, batchquery=4, batchpassage=4, ) print(metricsdataset_qa)

Evaluate on a local directory or a HuggingFace collection

datasetnames = getdatasetsfromcollection("vidore/vidore-benchmark-667173f98e70a1c0fa4db00d") metricscollection = {} for datasetname in tqdm(datasetnames, desc="Evaluating dataset(s)"): metricscollection[datasetname] = vidoreevaluator.evaluatedataset( ds=loaddataset(datasetname, split="test"), batchquery=4, batchpassage=4, ) print(metricscollection) ```

Implement your own retriever

If you want to evaluate your own retriever to use it with the CLI, you should clone the repository and add your own class that inherits from BaseVisionRetriever. You can find the detailed instructions here.

Compare retrievers using the EvalManager

To easily process, visualize and compare the evaluation metrics of multiple retrievers, you can use the EvalManager class. Assume you have a list of previously generated JSON metric files, e.g.:

bash data/metrics/ ├── bisiglip.json └── colpali.json

The data is stored in eval_manager.data as a multi-column DataFrame with the following columns. Use the get_df_for_metric, get_df_for_dataset, and get_df_for_model methods to get the subset of the data you are interested in. For instance:

```python from vidore_benchmark.evaluation import EvalManager

evalmanager = EvalManager.fromdir("data/metrics/") df = evalmanager.getdfformetric("ndcgat5") ```

Citation

ColPali: Efficient Document Retrieval with Vision Language Models

Authors: Manuel Faysse*, Hugues Sibille*, Tony Wu*, Bilel Omrani, Gautier Viaud, Céline Hudelot, Pierre Colombo (* denotes equal contribution)

```latex @misc{faysse2024colpaliefficientdocumentretrieval, title={ColPali: Efficient Document Retrieval with Vision Language Models}, author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo}, year={2024}, eprint={2407.01449}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2407.01449}, }

@misc{macé2025vidorebenchmarkv2raising, title={ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval}, author={Quentin Macé and António Loison and Manuel Faysse}, year={2025}, eprint={2505.17166}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2505.17166}, } ```

If you want to reproduce the results from the ColPali paper, please read the REPRODUCIBILITY.md file for more information.