Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/acdh-oeaw/acdh-search-indexer

https://github.com/acdh-oeaw/acdh-search-indexer

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (7.9%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: acdh-oeaw
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 203 KB
Statistics
  • Stars: 0
  • Watchers: 7
  • Forks: 3
  • Open Issues: 0
  • Releases: 0
Created about 2 years ago · Last pushed about 2 years ago
Metadata Files
Readme License

README.md

ACDH Search Indexer

This repository provides Python scripts to index data from TEI/XML sources into NoSketch Engine TSV Index files. It also contains a Dockerfile to build a NoSketch Engine Docker image with the ACDH Search Indexer scripts. Next to NoSketch it provides Python scripts to create an Index for Typesense Search.

Fetch Data

Update shellscripts/fetch_data.sh to fetch the data you want to index. The script will be executed in the Github Actions Workflow.

Create Noske Index

Update noske/mk_verticals.py constants to match your data. The script will be executed in the Github Actions Workflow. Or manually run:

shell pip install pipenv pipenv install pipenv run python noske/mk_verticals.py

Create Typesense Index

Update typesense/mk_index.py constants to match your data. The script will be executed in the Github Actions Workflow. Or manually run:

shell pip install pipenv pipenv install pipenv run python typesense/mk_index.py

Make sure the returned data type matches the Schema. If not, you can adjust the typesense/mk_index.py script match case statements to your schema. For example, the following code snippet creates a record for the Typesense index:

python def create_index_record(doc: TeiReader, blacklist: list) -> dict: record = {} for key, value in XPATHS.items(): items = extract_structure(doc, value) match key: case "date": # returns an integer of an attribute value record[key] = int("".join(items)) case "places": # returns a list of strings from a list of elements record[key] = [extract_text(place, blacklist) for place in items] case "edition": # returns a list of strings from a list of elements record[key] = [extract_text(edition, blacklist) for edition in items] case _: # returns a string from a list of elements record[key] = " ".join([extract_text(text, blacklist) for text in items]) return record

Noske Docker

The Image uses acdh-oeaw/noske-ubi9 as base image. Build the Docker image with the ACDH Search Indexer scripts:

shell docker build -f nosketchengine/Dockerfile -t acdh/noske .

Run Noske Docker Image:

shell docker run --rm -it -p 8080:8080 acdh/noske:latest

Owner

  • Name: Austrian Centre for Digital Humanities & Cultural Heritage
  • Login: acdh-oeaw
  • Kind: organization
  • Email: acdh@oeaw.ac.at
  • Location: Vienna, Austria

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

.github/workflows/index.yml actions
  • actions/cache v4 composite
  • actions/checkout v4 composite
  • actions/setup-python v5 composite
  • docker/build-push-action v5 composite
  • docker/login-action v3 composite
  • docker/metadata-action v5 composite
  • docker/setup-buildx-action v3 composite
nosketchengine/Dockerfile docker
  • acdhch/noske 5.71.15-2.225.8-open build
Pipfile pypi
  • acdh-cfts-pyutils ==0.2
  • acdh-tei-pyutils *
  • pandas *
  • tqdm *
Pipfile.lock pypi
  • acdh-cfts-pyutils ==0.2
  • acdh-handle-pyutils ==0.4.2
  • acdh-tei-pyutils ==1.3
  • acdh-xml-pyutils ==1.1.1
  • certifi ==2024.6.2
  • charset-normalizer ==3.3.2
  • click ==8.1.7
  • idna ==3.7
  • lxml ==5.2.2
  • numpy ==2.0.0
  • pandas ==2.2.2
  • python-dateutil ==2.9.0.post0
  • python-slugify ==8.0.4
  • pytz ==2024.1
  • requests ==2.32.3
  • six ==1.16.0
  • text-unidecode ==1.3
  • tqdm ==4.66.4
  • typesense ==0.14.0
  • tzdata ==2024.1
  • urllib3 ==2.2.2