https://github.com/centre-for-humanities-computing/twitter_hope_data_structure
https://github.com/centre-for-humanities-computing/twitter_hope_data_structure
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.3%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: centre-for-humanities-computing
- Default Branch: main
- Size: 2.93 KB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
readme.md
Twitter HOPE Data Structure
A brief introduction to the data structure for the HOPE project
Folder Structure
Located on Grundtvig on: /data
twitter_hope_001
├── raw
│ ├── rehydration DATE
│ │ ├── readme.md
│ │ └── ...
│ ├── rehydration DATE
│ ├── ...
│ ├── subdatafolders
│ │ ├── readme.md
│ │ └── ...
└── preproccessed
│ ├── da
│ │ ├── tweet_id.txt
│ │ ├── td_DATE_da_tlpdsn_001.ndjson
│ │ └── ...
│ ├── en
│ │ └── ...
│ ├── no
│ │ └── ...
│ ├── ...
...
001 denote the project ID.
Ideally, all folders should use lowercase and be split by _.
Raw Data
Raw data can be found in the raw folder and includes folders directly from the scrape. These include two categories, initial scrape (including historic data) and rehydration. Each folder is intended to include data from the scrape as well as a markdown (readme.md) on how the data was collected.
Preprocessed Data
The preprocessed data can be found in preprocessed and contains a folder for each language. Each of these folders contains a .txt file containing all the tweets ids (tweet_id_001.txt) and preprocessed Twitter data. The Twitter data is in the following format:
td_{DATE}_{LANG}_{PREPROCESSING FLAGS}_{ID}.ndjson
Where;
- td stand for Twitter Data
- DATE is the date the tweets in the file is from
- PREPROCESSING FLAGS is denotes what preprocessing have been applied to the data
- LANG is the language the tweets are believed to contain (or at least should be processed as)
- h: Historic/purchased data tag
- t: Tokenization
- l: Lemmatization
- p: POS-tags
- d: Dependency Parsing
- s: Sentiment Analysis
- n: Named Entity Recognition
- ID: project id, 001 for HOPE
Intended pipeline
- Data comes into the raw folder with a
readme.mddescribing the scrape method (or is added to its respective scrape) - Raw data is sorted into preprocessed by:
- sorting into specific languages (currently using twitters flag)
- standardizing to .ndjson files (e.g. .tsv's from the TCAT scrape is transformed to .ndjson)
- Preprocessing tags is added one or multiple at a times and the replacing the file with its respective tag.
Owner
- Name: Center for Humanities Computing Aarhus
- Login: centre-for-humanities-computing
- Kind: organization
- Email: chcaa@cas.au.dk
- Location: Aarhus, Denmark
- Website: https://chc.au.dk/
- Repositories: 130
- Profile: https://github.com/centre-for-humanities-computing
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0