Data

Tools

Indexes

Applications

Experiments

Open Source Science

crates-index-survey

Tool to OCCASIONALLY survey the crates.io registry index for Binstall stats

https://github.com/cargo-bins/crates-index-survey

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.8%) to scientific vocabulary
Last synced: 6 months ago · JSON representation ·

Repository

Tool to OCCASIONALLY survey the crates.io registry index for Binstall stats

Basic Info
  • Host: GitHub
  • Owner: cargo-bins
  • License: gpl-3.0
  • Language: Rust
  • Default Branch: main
  • Homepage:
  • Size: 155 KB
Statistics
  • Stars: 1
  • Watchers: 3
  • Forks: 0
  • Open Issues: 5
  • Releases: 0
Created over 3 years ago · Last pushed almost 3 years ago
Metadata Files
Readme License Citation

README.md

Tools to survey the crates.io registry

/!\ While these tools hit the CDN directly, you should still refrain from using them except than "occasionally", so as to not disrupt crates.io operations. /!\

These tools were written specifically for investigating Cargo Binstall and make informed decisions about planned features. They are not maintained and may not suit your usecase. Caveat emptor.

Building

These crates require building with RUSTFLAGS="--cfg tokio_unstable", and it's highly recommended to build in release mode.

fetch-crates

Updates the local cargo index, traverses it to get a list of all crates at their latest non-prerelease versions, and then downloads-and-unpacks the crates in the ./crates folder, one cratename-version folder per.

It's highly recommended to have ample free space, a fast internet connection, and filesystem compression enabled. At writing the tool downloads about 11 gigabytes of data, and writes roughly 45 gigabytes to the filesystem. With Btrfs ZSTD compression, that works out to about 15 gigabytes of disk usage.

On decent fibre broadband and an SSD it should take less than half an hour.

load-crates

Uses the DATABASE_URL to open a connection to a postgres database, which must have this table created:

sql CREATE TABLE crates ( name varchar(200) not null, version varchar(100) not null, manifest jsonb not null );

It then traverses the ./crates folder as prepared by fetch-crates, resolves the full Cargo manifest from the Cargo.toml and the src folder for each crate, converts that structure to JSON, and finally loads it into the database.

It is then possible to use standard PostgreSQL queries to interrogate the registry.

Examples

Count all crates

sql select count(*) from crates;

Count of binary crates

sql select count(*) from crates where manifest->'bin' is not null;

List crates with a pijul repository and their authors

sql select name, version, manifest->'package'->'authors' from crates where (manifest->'package'->'repository')::text like '%pijul%';

Summarise crate counts by the hostname of their repository field

sql select * from ( select substring((manifest->'package'->'repository')::text from '://[^/]+') hostname, count(*) count from crates where true and manifest->'package'->'metadata'->'binstall' is null and manifest->'package'->'repository' != 'null'::jsonb group by substring((manifest->'package'->'repository')::text from '://[^/]+') ) n order by count desc;

Show non-Binstall binary crates with a repository field which uses plain HTTP

sql select name, manifest->'package'->'repository' from crates where true and manifest->'package'->'metadata'->'binstall' is null and manifest->'package'->'repository' != 'null'::jsonb and substring( (manifest->'package'->'repository')::text from '^[^:]+:' ) = '"http:' and manifest->'bin' is not null;

Owner

  • Name: cargo-bins
  • Login: cargo-bins
  • Kind: organization

Cargo (rust) binary tools

Citation (CITATION.cff) 

# YAML 1.2
---
cff-version: 1.2.0
message: |
  If you use this software, please cite it using these metadata.

title: Crates Index Survey
version: 0.1.0
date-released: 2022-08-30

repository-code: https://github.com/cargo-bins/crates-index-survey
license: GPL-3.0

authors:
  - family-names: Saparelli
    given-names: Félix
    orcid: https://orcid.org/0000-0002-2010-630X

references:

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 0
  • Total pull requests: 15
  • Average time to close issues: N/A
  • Average time to close pull requests: about 2 months
  • Total issue authors: 0
  • Total pull request authors: 2
  • Average comments per issue: 0
  • Average comments per pull request: 0.6
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 14
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • dependabot[bot] (14)
  • NobodyXu (1)
Top Labels
Issue Labels
Pull Request Labels
dependencies (14) rust (14)

Dependencies

.github/workflows/checks.yml actions
  • actions/cache v3 composite
  • actions/checkout v3 composite
Cargo.lock cargo
  • 249 dependencies
Cargo.toml cargo