Protein Embedding Computational and Benchmarking Tool

Protein Embedding Computational and Benchmarking Tool hosts resources needed to transform mmCIF proteins to 5 state-of-the-art protein vector representations (embeddings), and a benchmarking tool to test how well the produced vectors follow the similarity of original protein files. Protein similarity is measured in TM-score, and the benchmarked metrics of vector space similarity are Euclidean and Cosine distance. The repository is divided in 4 modules.

Convertor Container Image

The convertor_container_image module contains Dockerfile and resources which can be built into a Container image used for transforming mmCIF protein files into embeddings. The code provides embedding implementations used in LMI, DeepFRI, GearNET, GRaSR, and 3D-af-Surfer.

To transform a folder containing mmCIF proteins into embeddings, execute corresponding run() method of selected embedding method. Example how to use converting method in your code:

``` from convertorcontainerimage.3d-af-surfer.cif23dzd import run

run(, .pkl)

: folder with .mmCIF files, you can use our test folder samplemmcifproteins

: pickle file where embeddings will be stored

```

Embedding Computational Pipeline

If you have access to an infrastructure capable of large scale processing of protein data, the embedding_computational_pipeline module serves as an automated pipeline to convert proteins from https://alphafold.ebi.ac.uk/download (~23TB of archives) into a selected embedding type. After deploying contents of the module into a pod in a Kuberneted cluster, one can innitiate the pipeline by executing following command:

python3 job_orchestrator.py --template job_templates/grasr-job.yaml.jinja2 --start_id 0 --jobs_number 25339 --max_concurrent_jobs 50 --namespace <your-kubernetes-cluster>

There are 5 available jinja templates to choose from. The data will be batched and processed in form of Kubernetes jobs.

Open-source results available for download

If you do not have access to an infrastructure to run the conversion code, we open-sourced the results for 5 precomputed embedding types, and hosted them on the Czech National Repository. Each embedding method was computed on AFDB v3, and contains 214m protein embeddings.

|Embedding |Type |Dimensionality |Size |Download link| |--------------------|---------|------------------:|-----:|------------| |3D-af-Surfer |Geometric |120 |161 GiB |10.48700/datst.tbws0-hj147| |GraSR |Neural network |400 |149 GiB |10.48700/datst.br8aq-db495| |PCA-reduced GraSR |PCA reduction |50 |79 GiB |10.48700/datst.rec6m-2sq83| |LMI-10 |Geometric |45 |8 GiB |10.48700/datst.0y0y6-v0783| |LMI-30 |Geometric |435 |67 GiB |10.48700/datst.tbws0-hj147|

Embedding Benchmark Tool

The embedding_benchmark_tool module provides an analysis conducted on a dataset constructed from 100 protein subsets (proteins from all samples form together a dataset of size 143,738), as well as more detailed analysis of one selected protein. It also stores a script to create your own test dataset. The main point of analysis was to assess which embedding corresponds the closest to similarity ranking by TM-score.

Example of the benchmark results. Full results are available under evaluate_dataset.ipynb...

|Rank |Embedding Method |AUC |Corr. coeff. |F1| |-----|--------------------|-----:|------------:|-:| |1. |3d-af-Surfer (C) |0.822 |0.595 |0.743| |2. |3d-af-Surfer (E) |0.815 |0.593 |0.757| |3. |PCA-reduced GraSR (C) |0.777 |0.569 |0.712| |4. |GraSR (C) |0.766 |0.556 |0.712| |5. |GraSR (E) |0.766 |0.556 |0.712| |6. |PCA-reduced GraSR (E) |0.766 |0.556 |0.712| |7. |LMI-30 (C) |0.615 |0.172 |0.549| |8. |LMI-10 (C) |0.669 |0.284 |0.573| |9. |LMI-30 (E) |0.674 |0.254 |0.514| |10. |LMI-10 (E) |0.661 |0.255 |0.516|

```

(C) = Vector distance computed with Cosine distance

(E) = Vector distance computed with Euclidean distance

```