https://github.com/centre-for-humanities-computing/friths
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.2%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: centre-for-humanities-computing
- Language: HTML
- Default Branch: main
- Size: 15.6 MB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 8
- Releases: 0
Metadata Files
README.md
Friths
Get started using bash dev_setup.sh
The project is structured following the cookiecutter data science template.
Pipelines
Publications Pipeline
Run using make publications
Input: publication PDFs (data/raw/UTA publications)
Manual work required to keep most data:
- Export scanned files (failed_pdf_paths.txt) with "Embed text" (Sofie)
- Manually convert or fix non-pdf files (non_pdf_paths.txt)
Pipeline:
A) dataset/parse_publications.py
We try to extract information using scipdf, which has a java dependency grobid.
If unsuccessful, the file will get added to failed_pdf_paths.txt & OCR'd later (no metadata avilable then).
B) dataset/ocr_failed_pdf.py
Runs OCR using pytesseract.
Outputs are saved separately from the parsed files.
C1) dataset/metadata_publications.py
Generates metadata & document ids separately for both file types (PARSING and OCR).
Most importatly, tries to reconstruct the publication year.
Otherwise, just takes the metadata found by scipdf (PARSING files only), or adds blank columns (OCR files only)/
C2) dataset/fetch_scopus.py
Fetches metadata from the Scopus API, given author IDs.
C3) dataset/metadata_scopus.py
Merges PARSING metadata with records extracted from Scopus (data/raw/ScopusExport_{author_id}_{date}.csv)
D) dataset/concat_publications.py
Concatenates article sections into a single block of text.
Creates files with the extracted abstracts.
Merges:
- PARSING and OCR files.
- PARSING and OCR metadata files.
- PARSING and OCR abstract files.
E) dataset/quality_checks_publications.py
Adds info about language & text descriptive stats into the metadata.
F) features/run_embeddings.py
Get embeddings from OpenAI
Analysis:
notebooks/experiment_abstracts.ipynb
notebooks/experiment_infodynamics.ipynb
Owner
- Name: Center for Humanities Computing Aarhus
- Login: centre-for-humanities-computing
- Kind: organization
- Email: chcaa@cas.au.dk
- Location: Aarhus, Denmark
- Website: https://chc.au.dk/
- Repositories: 130
- Profile: https://github.com/centre-for-humanities-computing
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 14
- Total pull requests: 12
- Average time to close issues: 8 days
- Average time to close pull requests: about 4 hours
- Total issue authors: 1
- Total pull request authors: 1
- Average comments per issue: 0.64
- Average comments per pull request: 0.33
- Merged pull requests: 12
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- jankounchained (13)
Pull Request Authors
- jankounchained (9)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- PyMuPDF *
- datasets *
- langdetect *
- openai *
- pandas *
- pybliometrics *
- pypdf *
- pytesseract *
- scipdf-parser *
- textdescriptives *
- tiktoken *
- tqdm *
- umap-learn *