Semanti🐱

WORK-IN-PROGRESS! (experimental usage)

Semantic@ (or sematicat) is a GUI prototype to annotate and to embed annotations in XML documents in TEI or EAD.

This tool follows a linear workflow: Import the document(s), apply a NER model and correct prediction or annotate manually from-zero and export and/or publish XML with annotations directly inside.

This platform is also designed to adapt generically to the diversity of publishing projects and its components are highly customisable.

:movie_camera: Demo

:battery: Installation

1. Clone the Github repository

bash git clone https://github.com/Lucaterre/semanticat.git

1. Move inside the directory

bash cd ./semanticat

1. Create a virtual environment with virtualenv

bash virtualenv --python=/usr/bin/python3.8 venv

1. Activate the virtual environment

bash source venv/bin/activate

1. Install dependencies

bash pip install -r requirements.txt

:rocket: Run Locally

:fire: This application is intended to be simple and local for the moment. Please note that the application is currently optimized for the Firefox browser.

Use the semantic@ CLI; inside the semanticat/ directory, launch the command :

bash python run.py Application is available on port 3000.

Others arguments :

| Type | Details | |-----------------------|------------------------------------------| | --dev_mode | Launch application in development mode | | --erase_recreate_db | Clean and Restore all database :warning: |

:arrow_forward: Getting started

• Start by creating a project with the button Create a new project and open your project;
• Go to Menu > Manage documents and import your XML, now you can see your documents in Project workflow view (You can mix EAD and TEI);

• In Project workflow view: Apply parse feature on document one by one or apply Parse All on all documents;

• Go to Menu > configuration, two use cases :

1. You don't want to apply a NER model, and you want to manually annotate your data :
2. Define Annotation mapping (see the "Mapping details" section);
3. Add labels with Add new pair to mapping scope;

4. Go to Project workflow > correct named entities and start annotation.

5. You want to use an NER (recommender) model to predict named entities and correct afterwards (see the "NER configuration details" section):

6. First, select checkbox NER Recommenders;

7. Choose the correct language that corresponding to your ressources;

8. Select the model and save;

9. Wait, the pre-mapping appears, you can then adapt it (see the "Mapping details" section);

10. Go to Project workflow > Launch Ner (or Launch Ner on all);

11. When the process is complete, go to correct named entities and correct the predictions or add annotations.

Whatever the chosen scenario, once the correction is finished, you can export your document (see the "Export details" section) !

:dart: Detail sections

Mapping

the mapping table makes it possible to match a NER category with a tag used in the output XML markup:

• Ner Label: The default label use to annotate or use by your model;
• Prefered Index label: The label that will appear in the output;
• Color: label color in annotation view.

You can add new labels to your existing schema via Add new pair to mapping scope.

Be careful if you remove a label from table and your model has already made predictions or if you have started to correct document, all annotations will be destroyed.

NER configuration

Currently, Semantic@ uses the NER SpaCy framework.

When installing the Semantic@, two pre-trained models for French (frcorenewssm) and English (encorewebsm) are already available

For add a new available SpaCy pre-trained model, before starting Semantic@, launch in terminal :

bash python -m spacy download <name-pretrained-model> then restart application.

The new pre-trained model will be directly available in model list from configuration view.

Sometimes, SpaCy’s default in-built pre-trained NER model are too slow and too generic for your data (the model is far from perfect so it doesn't necessarily detect your labels). If you have training a better statistical NER model with SpaCy, you can place your NER model folder under /instance_config/my_features/my_models/

Your model will be directly available in model list from configuration.

Export

There are different XML export solutions :

• annotations inline (based on characters offsets) (TEI specific): This export mode uses standoff-converter and uses the position of annotations in the text to produce an output. It is precise but sometimes it takes a lot of time.
• annotations to controlaccess level (EAD specific): This export mode inserts tags annotations in a level of type .

• annotations inline (based on surface form matching) (TEI & EAD): This export mode uses the surface form of annotated mentions to tag the output. It is fast but sometimes less precise (correct your document before export it).

• annotations in JSON: This export allows to keep track annotations in a JSON format. Import this directly into the annotation view.

:cryingcatface: Bug reports

Feel free to create a new issue (new features, bug reports, documentation etc.).

:computer: Stack

Interface

Main components

:bustinsilhouette: Mainteners

• @Lucaterre

:black_nib: How to cite

Please use the following citation:

@misc{terriel-2022-semanticat,
    title = "Semanticat: Annotation tool (NER) for XML documents",
    author = "Terriel, Lucas",
    year = "2022",
    url = "https://github.com/Lucaterre/semanticat",
}