pyonb

[!WARNING] This repo is under construction.

pyonb is two things:

• a Python SDK for document extraction via the Hyland OnBase REST API (work in progress)
• a suite of APIs wrapped around open-source Optical Character Recognition (OCR) tools, designed for local deployment, for converting PDFs to structured text including:
  • Marker
  • Sparrow
  • PaddleOCR
  • Docling

Getting Started

Prerequisites

pyonb requires Docker and Docker Compose.

Installation & Usage

1. Rename .env.sample to .env.

2. Edit .env with the correct HOST_DATA_FOLDER location, e.g.:

```sh HOSTDATAFOLDER="/absolute/path/to/documents/folder"

e.g. for unit tests on GAE:

HOSTDATAFOLDER="/gae/pyonb/tests/data/singlesyntheticdoc"

```

1. Set OCR service ports, e.g.:

sh OCR_FORWARDING_API_PORT=8110 MARKER_API_PORT=8112 SPARROW_API_PORT=8001 DOCLING_API_PORT=8115

[!IMPORTANT] For GAE usage, set OCR service ports and UCLH proxy details:

sh http_proxy= https_proxy= HTTPS_PROXY= HTTP_PROXY=

1. Start the OCR API Server (e.g. using marker and docling):

sh docker compose --profile marker --profile docling up -d

1. Open FastAPI Swagger at http://127.0.0.1:8110/docs to view and execute endpoints.

Use the following POST endpoints to execute the chosen OCR tool on a PDFs:

• marker - POST /marker/inference_single
• docling - POST /docling/inference_single

1. View the JSON response:

| | | :-------------------------------------------------------------: | | |

Developer Tips

• Alternatively to Swagger, use Postman to construct, save and make your API requests.

Tests

1. Clone the repo:

sh git clone https://github.com/SAFEHR-data/pyonb.git

1. Create a virtual environment (we suggest using uv) and install dependencies:

sh uv venv --python3.12 source .venv/bin/activate uv sync

1. Copy the tests/ .env file to root directory to use with tox:

sh cp /tests/.env.tests .env

1. Start the Docker services:

sh docker compose --profile marker --profile docling up -d

1. Run tests using tox:

sh tox -e py312

NB: this may take a few minutes to perform the inference tests. Some may fail depending on which OCR tools you choose to raise. For example, with --profile marker --profile docling the Sparrow API will not be raised, so the associated tests will fail.

To run unit tests individually, adapt the following:

sh tox -e py312 -- tests/api/test_routers.py::test_inference_single_file_upload_marker

About

Project Team

• Arman Eshaghi
• Tom Roberts (tom.roberts@ucl.ac.uk)
• Kawsar Noor
• Lawrence Lai
• Stefan Piatek
• Richard Dobson
• Steve Harris
• Sarah Keating

Acknowledgements

This work was funded by the National Institute for Health and Care Research (NIHR, award code NIHR302495).

This project is developed in collaboration with the Centre for Advanced Research Computing, University College London.