https://github.com/alfredtruong/cc-athena
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (2.2%) to scientific vocabulary
Last synced: 10 months ago
·
JSON representation
Repository
Basic Info
- Host: GitHub
- Owner: alfredtruong
- Language: Python
- Default Branch: main
- Size: 13.7 KB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Created almost 2 years ago
· Last pushed almost 2 years ago
Metadata Files
Readme
README.md
WHAT
- query common crawl (columnar) index that sits natively on S3
- send SQL queries to AWS Athena directly from Python
APPROACHES
- use sqlalchemy + pyathena
- use boto3
FEATURES
- queries and results are automatically cached locally
- cached results are used if sql query is the same
OUTPUT
- generates .pkl of required warc record locations of interest
Owner
- Login: alfredtruong
- Kind: user
- Repositories: 9
- Profile: https://github.com/alfredtruong