https://github.com/commoncrawl/ccf-eot-seeds-2024
Common Crawl's contribution of seeds to the End of Term Archive 2024
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (4.3%) to scientific vocabulary
Repository
Common Crawl's contribution of seeds to the End of Term Archive 2024
Basic Info
- Host: GitHub
- Owner: commoncrawl
- License: apache-2.0
- Language: Makefile
- Default Branch: main
- Size: 6.84 KB
Statistics
- Stars: 1
- Watchers: 6
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
ccf-eot-seeds-2024
Code used to generate some of the "seed lists" used for the End of Term Web Archive 2024 crawl.
Install
pip install -r requirements.txt
2024 recipes
make get-csvs
Downloads 2 csvs from get.gov, listing all of the federal and non-federal domains registered in the .gov tld.
make get-webgraph
Download web graph summaries from CCF, as tab-separated values (tsv).
make make-subsets
Given web graph domain and host ranks, grep out the .mil and .gov domains therein. Output is still the web graph table tsv format.
make hosts-to-seed
Take current-federal.csv plus the hosts webgraph, and output all .gov hosts whose domains are in current-federal.csv. For .mil hosts, output all hosts. This output is what is checked into eot2024/seed-lists.
- ccf-gov-federal-web-graph-2024-jun-jul-aug.txt
- ccf-mil-web-graph-2024-jun-jul-aug.txt
Owner
- Name: Common Crawl Foundation
- Login: commoncrawl
- Kind: organization
- Email: info@commoncrawl.org
- Website: https://commoncrawl.org
- Twitter: commoncrawl
- Repositories: 50
- Profile: https://github.com/commoncrawl
Common Crawl provides an archive of webpages going back to 2007.
GitHub Events
Total
- Watch event: 1
Last Year
- Watch event: 1
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Greg Lindahl | g****g@c****g | 2 |
| Greg Lindahl | l****l@p****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- duckdb *
- flake8 *
- jsonlines *
- numpy *
- pandas *