Descriptron

A pipeline and code hub for dark taxa mophology using CNNs/ViTs, classic computer vision, that talks to VLMs (GPT-4o). It helps taxonomists automate taxonomic species descriptions. It can also be used to automate 2D measurements and geometric morphometrics, and experimental 3D measurements. Additionally workflows allow for quantification of color# and color pattern. It also quantifies taxonomic species descriptions using joint image and text embeddings.

Descriptron Version v0.1.0

Descriptron is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.

"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.

1. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.

2. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.

3. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:

   You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.

1. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.

2. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.

3. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.

4. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.

5. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

And for Metric3D

BSD 2-Clause License

Copyright (c) 2024, Wei Yin and Mu Hu

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

So if you use Descriptron you must cite Descriptron and aslo cite SAM2, Detectron2, Metric3D, and EasyOCR, commercial use is prohibited by the above licensces

@software{VanDamDescriptron_2024, author = {Van Dam, Alex R.}, title = {Descriptron}, version = {main}, url = {https://github.com/alexrvandam/Descriptron}, date = {2024-12-31}, note = {Accessed: 2024-12-31} }

@article {Van Dam2025.01.07.631758, author = {Van Dam, Alex and Serbina, Liliya Starhova}, title = {Descriptron: Testing Artificial Intelligence for Automating Taxonomic Species Descriptions with a User-friendly Software Package}, elocation-id = {2025.01.07.631758}, year = {2025}, doi = {10.1101/2025.01.07.631758}, publisher = {Cold Spring Harbor Laboratory}, URL = {https://www.biorxiv.org/content/10.1101/2025.01.07.631758v1}, eprint = {https://www.biorxiv.org/content/early/2025/01/10/2025.01.07.631758v1.full.pdf}, journal = {bioRxiv} }

@misc{Metric3D, author = {Yin, Wei and Hu, Mu}, title = {{Metric3D}: A Toolbox for Zero-shot Metric Depth Estimation}, howpublished = {\url{https://github.com/YvanYin/Metric3D}}, year = {2024} }

@article{hu2024metric3d, title={Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation}, author={Hu, Mu and Yin, Wei and Zhang, Chi and Cai, Zhipeng and Long, Xiaoxiao and Chen, Hao and Wang, Kaixuan and Yu, Gang and Shen, Chunhua and Shen, Shaojie}, journal={arXiv preprint arXiv:2404.15506}, year={2024} }

@article{yin2023metric, title={Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image}, author={Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, Chunhua Shen}, booktitle={ICCV}, year={2023} }

@misc{wu2019detectron2, author = {Yuxin Wu and Alexander Kirillov and Francisco Massa and Wan-Yen Lo and Ross Girshick}, title = {Detectron2}, howpublished = {\url{https://github.com/facebookresearch/detectron2}}, year = {2019} }

@misc{easyocr, author = {JaidedAI}, title = {EasyOCR: Ready-to-use OCR with 80+ supported languages}, howpublished = {\url{https://github.com/JaidedAI/EasyOCR}}, year = {2020}, note = {Accessed: 2025-01-09} }

@article{ravi2024sam2, title={SAM 2: Segment Anything in Images and Videos}, author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and Rädle, Roman and Rolland, Chloe and Gustafson, Laura and others}, journal={arXiv preprint arXiv:2408.00714}, year={2024}, url={https://arxiv.org/abs/2408.00714} }

Introduction

Descriptron v0.1.0 is intended to automate geometric morphometrics of organismal morphological features. It combines the state-of-the-art CNN based instance segmentation of Detectron2 (https://github.com/facebookresearch/detectron2) implemented in PyTorch with classic computer vision implemented in OpenCV. The instance segmentation is via a CNN because that is what CNN's are best at and the precise pixel level data extraction is done using OpenCV as that is what classic computer vision is best at. I aim to use the correct tool for the job at hand. It is all written in python so it is more stable and all the programs play well together.

This work here helps move towards solving automated semi-landmarking, morphometrics, color extraction, and semantic prediction using GPT4-V of 2D data using Psyllidae to automate taxonomic species descriptions. This should be helpful to people interested in ground-truthing measurements for species delimitation using morphological shape and size as data in that delimitation process. This is a first version future versions will continue to expand the capabilities of computer vision and deep-learning for making geometric and standard morphometric analyses more available and in a stable format for the community. We plan to add keypoints very soon and line models as well. We also plan to add stero-vision capabilities for more accurate 3d measurements. For now I implement a forward facing 3d measurements using depth from defocus and depth maps, it is experimental so all 3d measurements all need validation at this stage. The 2d measurements should be very accurate depending on how you set up your pictures, measure your scale bars or utilize a known distance to pixel ratio.

The long term goal of this project is to combine LLM ViTs with instance segmentation and image depth data to produce fine grained descriptions and concomitant semi-supervised and unsupervised clustering for putative novel species binning and accurate automated taxomic descriptions, with taxonomists firmly placed at the helm over this new technology.

Installation Recommended Way (Linux, Mac) Windows coming soon.

```bash

(1) Downlad miniconda or conda if you don't already have it installed

https://docs.anaconda.com/miniconda/install/#

Follow the instructions for your computer (linux, windows, and mac are available).

(2) open terminal or cmd prompt

git clone https://github.com/alexrvandam/Descriptron.git

cd Descriptron

(3) load the environments

make sure you have python activated

which python conda env list

fire up the environments

conda init

close and re-start your terminal or cmd prompt then do

on bash kernal

source ~/.bashrc

on zsh kernal

source ~/.zshrc

next activate your base environment and make sure you knwo what it is, a good idea is to take notes here

conda activate base which python conda env list

a '*' should be next to yoru base environment

this will be the base conda or miniconda you will be adding the environments around

then if you have not already

cd Descriptron

cd environmentstoship

next build the environments

conda env create -f sammenv.yml conda env create -f detectron2envenv.yml conda env create -f measureenvenv.yml conda env create -f metric3denv.yml conda env create -f gpt4_env.yml

then activate each one at a time and install requirements

conda activate samm pip install -r samm_requirements.txt

probably there will be errors, most likely FAIR/META will have moved SAM2 and Detectron2, if that happens

you can get them from the GoolgleDrive folder under Linux Heavy Duty or download them yourself from FAIR/META

conda deactivate conda activate detectron2env pip install -r detectron2env_requirements.txt

again you may have to git install Detectron2 manually

conda deactivate cond activate measureenv pip install -r measureenv_requirements.txt

conda deactivate conda activate metric3d pip install metric3denvrequirements.txt conda deactivate conda activate gpt4 pip install gpt4envrequirements.txt conda deactivate conda activate samm

ok now the samm environment should be activated

now the early August version of sam2 was taken down in the middle of coding this project, so luckily you can download it from the link here

https://drive.google.com/drive/folders/1Cd4b2Sn-0nhSTUJYzQXxFA8i91ejnp?usp=sharing

copy the segment-anything-2 folder to your desktop and one at at time copy the folders to this segment-anything-2 folder,

you will also need to copy the Metric3d files and folders into the metric3d folder

they should all be under one folder Descriptron/segment-anything-2/gui/metric3d/Metric3d/*

once completed you should have

Descriptron/segment-anything-2/gui

or if you run this command from the gui folder

Descriptron/segment-anything-2/gui$

ls -1 -d */

you should see

detectron2/ idefics/ measure/ metric3d/

if that is correct then navigate the to gui folder or run the lauchv4.sh script if you know how to change your conda path and

then simply right click and run as script if you have changed your conda path correctly as well as the script path

OR if you have made it this far it might just be better to

cd Descriptron/segment-anything-2/gui

or what ever relative file path gets you into the gui folder you need to run the script from there

then simply

conda run -n samm python descriptron-label-206.py

that should do it the GUI should run

```

Linux Heavy Duty Way

If you are on a linux machine and you have conda-pack (conda install -c conda-forge conda-pack) you can unpack my environments with all the downloads in place, you will then have to modify the launch_guiv4.sh script to match your conda path. It is about 80GB unpacked

https://drive.google.com/drive/folders/1Cd4b2Sn-0nhSTUJYzQXxFA8i91ejnp?usp=sharing

The Linux Last Resort Hard Way

copy my folder, it is my project folder that is frozen with everyting in it you will need to have a file path that has /home/localuser/ in # order to get it to run otherwise it will not unpack do to symbolic links it is about 140GB

https://drive.google.com/drive/folders/1MqcgrG3fYxsUIJ9ROnFx8jhLNSkp9f4D

Citation for now a Bibtex of this page and BioRxiv Preprint

Preprint link

https://www.biorxiv.org/content/10.1101/2025.01.07.631758v1

@software{VanDamDescriptron_2024, author = {Van Dam, Alex R.}, title = {Descriptron}, version = {main}, url = {https://github.com/alexrvandam/Descriptron}, date = {2024-12-31}, note = {Accessed: 2024-12-31} }

@article {Van Dam2025.01.07.631758, author = {Van Dam, Alex and Serbina, Liliya Starhova}, title = {Descriptron: Testing Artificial Intelligence for Automating Taxonomic Species Descriptions with a User-friendly Software Package}, elocation-id = {2025.01.07.631758}, year = {2025}, doi = {10.1101/2025.01.07.631758}, publisher = {Cold Spring Harbor Laboratory}, abstract = {Recent advances in Computer Vision, Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Large Language Models (LLMs) suggest that it may be possible to approach mimicking the ability to decode knowledge about morphology and taxonomy to describe species in an automated way. Here we test for the first time a current state-of-the-art Vision Language Model (VLM) to approximate taxonomic species descriptions in an automated manner. The test utilizes a new graphical user interface, Descriptron, that collects data about biological images and transmits this highly specialized knowledge to a VLM to decode the taxonomic knowledge encoded in labeled biological images with text. Our results indicate that current state-of-the-art VLM (GPT-4o) can produce automated species descriptions that with error correction approximate taxonomists ability to describe morphological features of species and organize them in taxonomic keys. However, the results are not without significant errors and the VLM requires the input of the taxonomists knowledge to prevent widespread hallucinations by the VLM. We find here that the taxonomist is clearly needed to both teach and closely supervise the VLM. However, the time saved by utilizing Descriptron is also significant. Taxonomists remain essential for teaching and closely supervising the VLM. The time saved by utilizing Descriptron are nevertheless already very significant. The Descriptron program and supporting example prompt files are free to use under an Apache2 License available from: https://github.com/alexrvandam/Descriptron.Competing Interest StatementThe authors have declared no competing interest.}, URL = {https://www.biorxiv.org/content/early/2025/01/10/2025.01.07.631758}, eprint = {https://www.biorxiv.org/content/early/2025/01/10/2025.01.07.631758.full.pdf}, journal = {bioRxiv} }