Image classification using fine-tuned CLIP - for historical document sorting

Goal: solve a task of archive page images sorting (for their further content-based processing)

Scope: Processing of images, training / evaluation of CLIP model,
input file/directory processing, class (category) results of top
N predictions output, predictions summarizing into a tabular format,
HF hub ¹ support for the model, multiplatform (Win/Lin) data
preparation scripts for PDF to PNG conversion

Table of contents

• Versions
• Model description
  • Data
  • Categories
• How to install
• How to run prediction modes
  • Page processing
  • Directory processing
• Results
  • Result tables and their columns
• Data preparation
  • PDF to PNG
  • PNG pages annotation
  • PNG pages sorting for training
• For developers
  • Training
  • Evaluation
• Contacts
• Acknowledgements
• Appendix

Versions

There are currently 4 version of the model available for download, both of them have the same set of categories,
but different data annotations. The latest approved v1.1 is considered to be default and can be found in the main branch
of HF hub ¹

Model description

Fine-tuned model repository: UFAL's clip-historical-page ¹

Base model repository: OpenAI's clip-vit-base-patch16, clip-vit-base-patch32, clip-vit-large-patch14, clip-vit-large-patch14-336 ^{2 3 4 5}

The model was trained on the manually annotated dataset of historical documents, in particular, images of pages
from the archival documents with paper sources that were scanned into digital form.

The images contain various combinations of texts , tables , drawings , and photos -
categories described below were formed based on those archival documents. Page examples can be found in
the category_samples directory.

The key use case of the provided model and data processing pipeline is to classify an input PNG image from PDF scanned
paper source into one of the categories - each responsible for the following content-specific data processing pipeline.

In other words, when several APIs for different OCR subtasks are at your disposal - run this classifier first to
mark the input data as machine-typed (old style fonts) / handwritten / just printed plain text
or structured in tabular format text, as well as to mark the presence of the printed or drawn graphic
materials yet to be extracted from the page images.

Data

The dataset is provided under Public Domain license, and consists of 15855 PNG images of pages from the archival documents.
The source image files and their annotation can be found in the LINDAT repository ⁶ .

Training set of the model: 14267 images

90% of all - proportion in categories tabulated below

Evaluation set: 1583 images

10% of all - same proportion in categories as below and demonstrated in model_EVAL.csv

Manual annotation was performed beforehand and took some time , the categories were formed from
different sources of the archival documents originated in the 1920-2020 years span.

In total, several thousands of separate PDF files were selected and split into PNG pages, ~4k of scanned documents
were one-page long which covered around a third of all data, and ~2k of them were much longer (dozens and hundreds
of pages) covering the rest (more than 60% of all annotated data).

The specific content and language of the
source data is irrelevant considering the model's vision resolution, however, all of the data samples were from archaeological
reports which may somehow affect the drawing detection preferences due to the common form of objects being ceramic pieces,
arrowheads, and rocks formerly drawn by hand and later illustrated with digital tools (examples can be found in
category_samples/DRAW )

Categories

The categories were chosen to sort the pages by the following criteria:

• presence of graphical elements (drawings OR photos )
• type of text (handwritten OR printed OR typed OR mixed )
• presence of tabular layout / forms

The reasons for such distinction are different processing pipelines for different types of pages, which would be
applied after the classification as mentioned above.

Examples of pages sorted by category can be found in the category_samples directory
which is also available as a testing subset of the training data.

How to install

Step-by-step instructions on this program installation are provided here. The easiest way to obtain the model would
be to use the HF hub repository ¹ that can be easily accessed via this project.

cd /local/folder/for/this/project
git init
git clone https://github.com/ufal/atrium-page-classification.git

cd atrium-page-classification
git checkout vit

cd /local/folder/for/this/project/atrium-page-classification
git add <changed_file>
git commit -m 'local changes'

git pull -X theirs

git fetch origin
git checkout vit        
git pull --ff-only origin vit

cd /local/folder/for/this/project/atrium-page-classification

rm *.py
rm *.txt
rm README*
git pull

git reset --hard HEAD
git clean -fd
git fetch origin
git checkout vit
git pull origin vit

git fetch origin
git checkout vit
git reset --hard origin/vit

source <your_venv_dir>/bin/activate

Installation of Python dependencies can be done via:

pip install -r requirements.txt

After the dependencies installation is finished successfully, in the same virtual environment, you can
run the Python program.

To test that everything works okay and see the flag
descriptions call for --help :

python3 run.py -h

You should see a (hopefully) helpful message about all available command line flags. Your next step would be
to pull the model from the HF hub repository ¹ via:

python3 run.py --hf

OR for specific model version (e.g. main, vX.1 or vX.2) use the --revision flag:

python3 run.py --hf -rev v1.1

OR for specific base model version (e.g. ViT-B/16, ViT-B/32, ViT-L/14 or ViT-L/14@336px) use the --base flag (only when the
trained model version demands such base model as described above):

python3 run.py --hf -rev v2.2 -m `ViT-L/14@336`

You should see a message about loading the model from the hub and then saving it locally on
your machine .

Only after you have obtained the trained model files (takes less time than installing dependencies),
you can play with any commands provided below.

After the model is downloaded, you should see a similar file structure:

/local/folder/for/this/project/atrium-page-classification
 models
     <base_code>_rev_<revision> 
         config.json
         model.safetensors
         preprocessor_config.json
 model_checkpoints
     model_<categ_limit>_<base_code>_<lr>.pt
     model_<categ_limit>_<base_code>_<lr>_cp.pt
     ...
 data_scripts
     windows
         move_single.bat
         pdf2png.bat
         sort.bat
     unix
         move_single.sh
         pdf2png.sh
         sort.sh
 result
     plots
         conf_mat_Nn_Cc_<base>_date-time.png
         ...
     tables
         result_date-time_<base>_Nn_Cc.csv
         EVAL_table_Nn_Cc_<base>_date-time.csv
         date-time_<base>_RAW.csv
         ...
 category_samples
     DRAW
         CTX193200994-24.png
         ...
     DRAW_L
     ...
 run.py
 classifier.py
 utils.py
 requirements.txt
 config.txt
 README.md
 ...

Some of the folders may be missing, like mentioned later model_output which is automatically created
only after launching the model.

How to run prediction modes

There are two main ways to run the program:

• Single PNG file classification
• Directory with PNG files classification

To begin with, open config.txt and change folder path in the [INPUT] section, then
optionally change top_N and batch in the [SETUP] section.

The batch variable value depends on your machine memory size

It is safe to use batch size below 12 for a regular office desktop computer, and lower it to 4 if it's an old device.
For training on a High Performance Computing cluster, you may use values above 20 for
the batch variable in the [SETUP] section.

Make sure the virtual environment with all the installed libraries is activated, you are in the project
directory with Python files and only then proceed.

cd /local/folder/for/this/project/
source <your_venv_dir>/bin/activate
cd atrium-page-classification

Page processing

The following prediction should be run using the -f or --file flag with the path argument. Optionally,
you can use the -tn or --topn flag with the number of guesses you want to get, and also the -m or
--model flag with the path to the model folder argument.

python3 run.py -tn 3 -f '/full/path/to/file.png' --model_path '/full/path/to/model/folder' -m '<base_code>'

python3 run.py -f '/full/path/to/file.png'

Directory processing

The following prediction type does NOT require explicit directory path setting with the -d or --directory,
since its default value is set in the config.txt file and awakens when the --dir flag
is used. The same flags for the number of guesses and the model folder path as for the single-page
processing can be used. In addition, a directory-specific flag --raw is available.

Worth mentioning that the directory level processing is performed in batches, therefore you should refer to
the hardware's memory capacity requirements for different batch sizes tabulated above.

python3 run.py -tn 3 -d '/full/path/to/directory' --model_path '/full/path/to/model/folder' -m '<base_code>'

python3 run.py --dir 

python3 run.py -rev v2.2 -m ViT-B/15 --dir

The classification results of PNG pages collected from the directory will be saved to related results
folders defined in [OUTPUT] section of config.txt file.

Naturally, processing of the large amount of PNG pages takes time and progress of this process
is recorded in the console via messages like Processed <BN> images where B
is batch size set in the [SETUP] section of the config.txt file,
and N is an iteration of the current dataloader processing loop.

Only after all images from the input directory are processed, the output table is
saved in the result/tables folder.

Results

There are accuracy performance measurements and plots of confusion matrices for the evaluation
dataset (10% of the provided in [TRAIN]'s folder data). Both graphic plots and tables with
results can be found in the result folder.

v1.1 Evaluation set's accuracy (Top-1): 100.00%

v1.2 Evaluation set's accuracy (Top-1): 100.00%

v2.1 Evaluation set's accuracy (Top-1): 99.94%

v2.2 Evaluation set's accuracy (Top-1): 99.87%

Confusion matrices provided above show the diagonal of matching gold and predicted categories
while their off-diagonal elements show inter-class errors. By those graphs you can judge
what type of mistakes to expect from your model.

By running tests on the evaluation dataset after training you can generate the following output files:

• EVAL_table_Nn_Cc__date-time.csv - (by default) results of the evaluation dataset with TOP-N guesses
• conf_mat_Nn_Cc__date-time.png - (by default) confusion matrix plot for the evaluation dataset also with TOP-N guesses
• date-time__RAW.csv - (by flag --raw) raw probabilities for all classes of the processed directory
• result_date-time__Nn_Cc.csv - (by default) results of the processed directory with TOP-N guesses

Additionally, results of prediction inference run on the directory level without checked results are included.

Result tables and their columns

The reason to use the --raw flag is the possible convenience of results review,
since the rows will be basically sorted by categories, and most ambiguous ones will
have more small probabilities instead of zeros than the most obvious (for the model)
categories .

Data preparation

You can use this section as a guide for creating your own dataset of pages, which will be suitable for
further model processing.

There are useful multiplatform scripts in the data_scripts folder for the whole process of data preparation.

On Windows you must also install the following software before converting PDF documents to PNG images:

• ImageMagick ⁹ - download and install the latest version
• Ghostscript ¹⁰ - download and install the latest version (32 or 64-bit) by AGPL

PDF to PNG

The source set of PDF documents must be converted to page-specific PNG images before processing. The following steps
describe the procedure of converting PDF documents to PNG images suitable for training, evaluation, or prediction inference.

Firstly, copy the PDF-to-PNG converter script to the directory with PDF documents.

move \local\folder\for\this\project\data_scripts\pdf2png.bat \full\path\to\your\folder\with\pdf\files

cp /local/folder/for/this/project/data_scripts/pdf2png.sh /full/path/to/your/folder/with/pdf/files

Now check the content and comments in pdf2png.sh or pdf2png.bat
script, and run it.

cd \full\path\to\your\folder\with\pdf\files
pdf2png.bat

cd /full/path/to/your/folder/with/pdf/files
pdf2png.sh

After the program is done, you will have a directory full of document-specific subdirectories
containing page-specific images with a similar structure:

/full/path/to/your/folder/with/pdf/files
 PdfFile1Name
     PdfFile1Name-001.png
     PdfFile1Name-002.png
     ...
 PdfFile2Name
     PdfFile2Name-01.png
     PDFFile2Name-02.png
     ...
 PdfFile3Name
     PdfFile3Name-1.png 
 PdfFile4Name
 ...

\full\path\to\your\folder\with\pdf\files
 PdfFile1Name
     PdfFile1Name-1.png
     PdfFile1Name-2.png
     ...
 PdfFile2Name
     PdfFile2Name-1.png
     PDFFile2Name-2.png
     ...
 PdfFile3Name
     PdfFile3Name-1.png 
 PdfFile4Name
 ...

Optionally you can use the move_single.sh or move_single.bat script to move
all PNG files from directories with a single PNG file inside to the common directory of one-pagers.

By default, the scripts assume that the onepagers is the back-off directory for PDF document names without a
corresponding separate directory of PNG pages found in the PDF files directory (already converted to
subdirectories of pages).

move \local\folder\for\this\project\atrium-page-classification\data_scripts\move_single.bat \full\path\to\your\folder\with\pdf\files
cd \full\path\to\your\folder\with\pdf\files
move_single.bat

cp /local/folder/for/this//project/atrium-page-classification/data_scripts/move_single.sh /full/path/to/your/folder/with/pdf/files
cd /full/path/to/your/folder/with/pdf/files 
move_single.sh 

The reason for such movement is simply convenience in the following annotation process below.
These changes are cared for in the next sort.sh and sort.bat scripts as well.

PNG pages annotation

The generated PNG images of document pages are used to form the annotated gold data.

Prepare a CSV table with exactly 3 columns:

• FILE - name of the PDF document which was the source of this page
• PAGE - number of the page (NOT padded with 0s)
• CLASS - label of the category

For Windows users, it's NOT recommended to use MS Excel for writing CSV tables, the free
alternative may be Apache's OpenOffice ¹¹ . As for Unix users, the default LibreCalc should be enough to
correctly write a comma-separated CSV table.

FILE,PAGE,CLASS
PdfFile1Name,1,Label1
PdfFile2Name,9,Label1
PdfFile1Name,11,Label3
...

PNG pages sorting for training

Cluster the annotated data into separate folders using the sort.sh or sort.bat
script to copy data from the source folder to the training folder where each category has its own subdirectory.
This division of PNG images will be used as gold data in training and evaluation.

sort.bat

sort.sh

After the program is done, you will have a directory full of label-specific subdirectories
containing document-specific pages with a similar structure:

/full/path/to/your/folder/with/train/pages
 Label1
     PdfFileAName-00N.png
     PdfFileBName-0M.png
     ...
 Label2
 Label3
 Label4
 ...

\full\path\to\your\folder\with\train\pages
 Label1
     PdfFileAName-N.png
     PdfFileBName-M.png
     ...
 Label2
 Label3
 Label4
 ...

The sorting script can help you in moderating mislabeled samples before the training. Accurate data annotation
directly affects the model performance.

Before running the training, make sure to check the config.txt file for the [TRAIN] section variables, where you should
set a path to the data folder. Make sure label directory names do NOT contain special characters like spaces, tabs or paragraph splits.

From this point, you can start model training or evaluation process.

For developers

You can use this project code as a base for your own image classification tasks. The detailed guide on
the key phases of the whole process (settings, training, evaluation) is provided here.

Most of the changeable variables are in the config.txt file, specifically,
in the [TRAIN], [HF], and [SETUP] sections.

In the dev sections of the configuration file, you will find many boolean variables that can be changed from the default False
state to True, yet it's recommended to awaken those variables solely through the specific
command line flags implemented for each of these boolean variables.

For more detailed training process adjustments refer to the related functions in classifier.py
file, where you will find some predefined values not used in the run.py file.

Device requirements for training / evaluation:

• CPU of some kind and memory size
• GPU (for real CUDA ⁷ support - better one of Nvidia's cards)

Worth mentioning that the efficient training is possible only with a CUDA-compatible GPU card.

For test launches on the CPU-only device you should set batch size to lower than 4, and even in this
case, above-average CPU memory capacity is a must-have to avoid a total system crush.

Training

To train the model run:

python3 run.py --train

The training process has an automatic progress logging into console, and should take approximately 5-12h
depending on your machine's CPU / GPU memory size and prepared dataset size.

Above are the default hyperparameters or TrainingArguments ¹² used in the training process that can be partially
(only epoch and log_step) changed in the [TRAIN] section, plus batch in the [SETUP]section,
of the config.txt file. Importantly, avg - average configuration of all texts can be used.

In case your descriptions table contains more than 1 text per category , the --avg flag will be set to True automatically.

descriptions_comparison_graph.png is a graph containing separate and averaged results
of all category descriptions. Using averaged text embeddings of all label description seems to be the most powerful way to
classify our images.

You are free to play with the learning rate right in the training function arguments called in the run.py file,
yet warmup ratio and other hyperparameters are accessible only through the classifier.py file.

Playing with training hyperparameters is
recommended only if training loss (error rate) descends too slow to reach 0.001-0.001
values by the end of the 3rd (last by default) epoch.

In the case evaluation loss starts to steadily going up after the previous descend, this means
you have reached the limit of worthy epochs, and next time you should set epochs to the
number of epoch that has successfully ended before you noticed the evaluation loss growth.

During training image transformations ¹³ are applied sequentially with a 50% chance.

More about selecting the image transformation and the available ones you can read in the PyTorch torchvision docs ¹³.

After training is complete the model will be saved to its separate subdirectory in the model directory, by default,
the naming of the model folder corresponds to the length of its training batch dataloader and the number of epochs -
for example model_<S/B>_E where E is the number of epochs, B is the batch size, and S is the size of your
training dataset (by defaults, 90% of the provided in [TRAIN]'s folder data).

/local/folder/for/this/project/atrium-page-classification
 models
     <base_code>_rev_v<HFrevision1> 
         config.json
         model.safetensors
         preprocessor_config.json
     <base_code>_rev_v<HFrevision2>
     ...
 hf_hub_checkpoints
     models--openai--clip-vit-base-patch16
         blobs
         snapshots
         refs
     .locs
         models--openai--clip-vit-large-patch14
 model_checkpoints
     model_<categ_limit>_<base_code>_<lr>.pt
     model_<categ_limit>_<base_code>_<lr>_cp.pt
     ...
 data_scripts
     windows
     unix
 result
     plots
     tables
 category_samples
     DRAW
     DRAW_L
     ...
 run.py
 classifier.py
 utils.py
 ...

You can slightly change the test_size and / or
the batch variable value in the config.txt file to train a differently named model on the same dataset.
Alternatively, adjust the model naming generation in the classifier.py's training function.

Evaluation

After the fine-tuned model is saved , you can explicitly call for evaluation of the model to get a table of TOP-N classes for
the randomly composed subset (10% in size by default) of the training page folder.

There is an option of setting test_size to 0.8 and use all the sorted by category pages provided
in [TRAIN]'s folder for evaluation, but do NOT launch it on the whole training data you have actually used up
for the evaluated model training.

To do this in the unchanged configuration , automatically create a
confusion matrix plot and additionally get raw class probabilities table run:

python3 run.py --eval

OR when you don't remember the specific [SETUP] and [TRAIN] variables' values for the trained model, you can use:

python3 run.py --eval -model_path 'model_<categ_limit>_<base>_<lr>.pt'

To prove that initial models without finetuning show awful results you can run --zero_shot flag during the evalution.

python3 run.py --eval --zero_shot -m '<base_code>'

Finally, when your model is trained and you are happy with its performance tests, you can uncomment a code line
in the run.py file for HF hub model push. This functionality has already been implemented and can be
accessed through the --hf flag using the values set in the [HF] section for the token and repo_name variables.

In this case, you must rename the trained model folder in respect to the revision value (dots in the naming are skipped, e.g.
revision v1.9.22 turns to model_v1922 model folder), and only then run repo push.

Contacts

For support write to: lutsai.k@gmail.com responsible for this GitHub repository ¹⁴

Information about the authors of this project, including their names and ORCIDs, can
be found in the CITATION.cff file.

Acknowledgements

• Developed by UFAL ¹⁵
• Funded by ATRIUM ¹⁶
• Shared by ATRIUM ¹⁶ & UFAL ¹⁵
• Model type: fine-tuned ViT with a 224x224 ^{2 3 4} or 336x336 ⁵ resolution size

2022 UFAL & ATRIUM

Appendix