https://github.com/acdh-oeaw/arche-curationtools
A collection of scripts and tools for curation (renaming, deleting system files, collecting metadata etc.) of data. Used e.g. for ARCHE
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.8%) to scientific vocabulary
Repository
A collection of scripts and tools for curation (renaming, deleting system files, collecting metadata etc.) of data. Used e.g. for ARCHE
Basic Info
Statistics
- Stars: 1
- Watchers: 6
- Forks: 0
- Open Issues: 1
- Releases: 0
Metadata Files
README.md
arche-curationTools
A collection of scripts and tools for curation of data. Used e.g. for ARCHE A description on what each script does is provided in the sections below.
All tools were mainly developed in an Windows environment. Though interoperability across operating systems has been kept in mind and should be given. If not so, please open an issue.
How to use
You will need Python (3, but 2 might work as well). Save the script on your local machine. Open it with a text editor (or a dedicated Python editor) and change the variables for folder or file locaction. Which variables should be adapted is stated at the top documentation section of each file. Save your changes and then execute the script from your favourite CLI with python scriptname.py.
tif_lzw.sh
Assures all TIFF files in a given directory are LZW-compressed.
Usage on the cluster:
- Login to the arche-ingestion environment on the cluster (see here).
- Run
screen -S myCompressionSessionName - Run
bash bash /ARCHE/curationTools/tif_lzw.sh directoryToProcess 2>&1 | tee pathToLogFile- If you need to process all subdirectories of a given directory, then
bash for i in `find directoryToProcess -type d` ; do bash /ARCHE/curationTools/tif_lzw.sh $i 2>&1 | tee -a pathToLogFile ; donee.g.bash for i in `find /ARCHE/staging/GlaserPhotos_20610/data/ -type d` ; do bash /ARCHE/curationTools/tif_lzw.sh $i 2>&1 | tee -a pathToLogFile ; done
- If you need to process all subdirectories of a given directory, then
Local usage - if you are a hacker, you will figure it out based on the instructions above. If not, do not run it locally. (hint - you need gdal and exiftool binaries installed and in your PATH)
chopttl.py
Cuts up a large ttl file into smaller pieces of a maximum amout of lines. Files might get a bit longer if end of an entity was not reached yet.
deleteSystemFiles.py
A program to delete system files as e.g. thumbs.db or .DS_Store
(This is also automatically done by the repo-filechecker)
extractPLYmetadata.py
A program to extract basic information from PLY files. Currently this information is extracted: number of vertices and number of faces
generateFolders.py
A program to create folders for grouping files with similar names together. For example create a folder 038_FJB_1904-001 to group the files 038_FJB_1904-001a.tif and 038_FJB_1904-001b.tif
The rule(s) to determine similarity of names is specified within a dedicated function and can be adapted to any user's needs.
metadatacrawler
See readme inside of it.
(This script has been deprecated in favor of the arche-metadata-crawler)
metahead.py
A script sanitizing the metahead.ttl.
Currently it automatically generates missing acdh:hasTitle for resources of type acdh:Person.
renameFiles.py
A program to rename files and folders in a given folder by applying a set of renaming rules. The set of rules can be expanded, either with Python string replace operations or by using regular expressions.
When executed as is Umlauts and ß will be replaced with ae, ss etc. all other special characters are normalised with the normalize function from unicodedata. Thus e.g. é will become e. Currently the mode 'NFKD' is used. What the mode means and what other modes are available is nicely described in this article on towards data science
Finally all characters that are not alphanumeric or not an underscore (_) or a hyphen (-) are removed.
This set of rules ensures that file names comply with the requirements of the digital archive ARCHE
Owner
- Name: Austrian Centre for Digital Humanities & Cultural Heritage
- Login: acdh-oeaw
- Kind: organization
- Email: acdh@oeaw.ac.at
- Location: Vienna, Austria
- Website: https://www.oeaw.ac.at/acdh
- Repositories: 476
- Profile: https://github.com/acdh-oeaw
GitHub Events
Total
- Push event: 2
Last Year
- Push event: 2
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| bellerophons-pegasus | b****s@y****e | 18 |
| Mateusz Żółtak | z****k@z****g | 4 |
| Massimiliano Carloni | 7****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 1
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 1.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- carlonim (1)