https://github.com/centre-for-humanities-computing/hope_dataprep
Scripts for ongoing data preparation for the HOPE project
https://github.com/centre-for-humanities-computing/hope_dataprep
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.3%) to scientific vocabulary
Repository
Scripts for ongoing data preparation for the HOPE project
Basic Info
- Host: GitHub
- Owner: centre-for-humanities-computing
- License: mit
- Language: Python
- Default Branch: master
- Size: 10.7 KB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
HOPE Data Preparation
This repository contains up-to-date scripts for Twitter data preparation for the HOPE project. At this point the main processes here are: * Daily data preparation: checkng for newly scraped data, which is then reformatted from .tsv to .ndjson (new line json, a more 'streamable' format) and filtered based on language (using Twitter's language tag) into folders for Danish, Swedish, and Norwegian tweets. * Sentiment extraction: applying Vader on raw tweet text from .ndjson files, and attaching the scores to every line in the file.
Previously there have also been scripts that would conduct basic preprocessing like tokenisation, lemmatization, and pos-tagging for different nordic languages, but they have been discontinued as end-users always used their own preprocessing pipeline for the particular analysis they had in mind. If you are interested in these legacy scripts, explore the 'Preprocessing' folder in covid19_rbkh repo on Grudntvig (beware that it is quite messy in some places).
Project Developers
|Key | Value | | --- |:---:| | name | Anita Kurm | | email |anitakurm@gmail.com |
Project Description
You can read more about the project here
Data Assessment
| Source | risk | Storage | Comment| | --- |:---:|---|---| |Twitter|GDPR|/data on the computer called Grundtvig| |
Data Identification
| Item | Answers | | --- | --- | | Description | Twitter data | | Collection | Ongoing scraping using nordic stop words | | Volume | 60+ GB| | Formats | Raw data is in tsv format, processed – in .ndjson format| | Software | - |
Data Management Comments
License
This software is MIT licensed.
Owner
- Name: Center for Humanities Computing Aarhus
- Login: centre-for-humanities-computing
- Kind: organization
- Email: chcaa@cas.au.dk
- Location: Aarhus, Denmark
- Website: https://chc.au.dk/
- Repositories: 130
- Profile: https://github.com/centre-for-humanities-computing
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0