https://github.com/commoncrawl/cc-monitoring
Code that monitors Common Crawl infrastructure
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (4.6%) to scientific vocabulary
Repository
Code that monitors Common Crawl infrastructure
Basic Info
- Host: GitHub
- Owner: commoncrawl
- Language: Python
- Default Branch: main
- Size: 41 KB
Statistics
- Stars: 2
- Watchers: 9
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
cc-monitoring
This code monitors the latency of the Common Crawl CDX index server, and the AWS Open Data bucket. Time series points are stored in an sqlite3 database, suitable for display in Grafana and other dashboard tools.
Installation
git clone ...
pip install .
Running
ccm -v initdb
nohup ccm -v collect
This code uses sqlite3's WAL (write ahead log) feature. Due to that, the directory that the database is in must be writeable by the ccm process.
Deployment and documentation
- Grafana server (to be named later)
- User-oriented documentation
Exporting dashboards
- Click the share icon in the upper left
- Export
- turn on Export for sharing externally
- Save to file
Owner
- Name: Common Crawl Foundation
- Login: commoncrawl
- Kind: organization
- Email: info@commoncrawl.org
- Website: https://commoncrawl.org
- Twitter: commoncrawl
- Repositories: 50
- Profile: https://github.com/commoncrawl
Common Crawl provides an archive of webpages going back to 2007.
GitHub Events
Total
Last Year
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Greg Lindahl | g****g@c****g | 22 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0