Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/althonos/cksfv.rs

A 10x faster drop-in reimplementation of cksfv using Rust and the crc32fast crate.

https://github.com/althonos/cksfv.rs

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 5 committers (20.0%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.0%) to scientific vocabulary

Keywords from Contributors

genomics metagenomics bioinformatics annotation progress-bar archive
Last synced: 10 months ago · JSON representation

Repository

A 10x faster drop-in reimplementation of cksfv using Rust and the crc32fast crate.

Basic Info
  • Host: GitHub
  • Owner: althonos
  • License: mit
  • Language: Rust
  • Default Branch: master
  • Homepage:
  • Size: 97.7 KB
Statistics
  • Stars: 12
  • Watchers: 2
  • Forks: 3
  • Open Issues: 5
  • Releases: 4
Created over 6 years ago · Last pushed almost 3 years ago
Metadata Files
Readme Changelog License

README.md

cksfv.rs Star me

A 10x faster drop-in reimplementation of cksfv using Rust and the crc32fast crate.

Actions Codecov License Source Crate Documentation Changelog GitHub issues

🗺️ Overview

cksfv is a FOSS tool developed by Bryan Call and maintained by Heikki Orsila to validate integrity of files using the CRC32 checksum. It can be used to check a list of files against a .sfv signature list, and to generate new lists.

This repository contains code for an implementation written from scratch in Rust, that provides the same CLI as the original program but much better performance thanks to a CRC32 implementation that takes advantage of modern CPUs.

🛠️ Features

Features from the original binary:

  • [x] Exact same Command-Line Interface for scripting compatibility
  • [x] Exact same behaviour with respect to argument parsing
  • [x] SFV listing generation compatible with the original cksfv
  • [x] SFV listing validation with -f, -g or -r
  • [ ] SFV filtering in validation mode using files given as arguments
  • [ ] Specific options:
    • [x] -b flag to only print base filenames
    • [x] -c flag to print everything to STDOUT
    • [x] -C flag to change directory when processing files
    • [x] -i flag to ignore case on filenames
    • [x] -L flag to follow symlinks
    • [x] -q flag to only print error messages
    • [ ] -s flag to replace backslashes

Additional features:

  • [x] Support for mmap syscall to avoid reading the file directly
  • [ ] Multithreading for several files

⏱️ Benchmarks

Benchmark where conducted using a 1.6GiB file using the cksfv binary distributed with ArchLinux, or this program compiled with either stable or nightly Rust, running hyperfine, on an Intel i7-8550U CPU:

``` Benchmark #1: cksfv test.mkv Time (mean ± σ): 4.376 s ± 0.120 s [User: 4.117 s, System: 0.251 s] Range (min … max): 4.237 s … 4.555 s 10 runs

Benchmark #2: cargo +stable run --release -- test.mkv Time (mean ± σ): 387.9 ms ± 9.3 ms [User: 158.3 ms, System: 224.9 ms] Range (min … max): 380.0 ms … 402.3 ms 10 runs

Benchmark #3: cargo +nightly run --release -- test.mkv Time (mean ± σ): 387.9 ms ± 11.7 ms [User: 160.9 ms, System: 226.6 ms] Range (min … max): 373.8 ms … 414.9 ms 10 runs

Benchmark #4: cargo +nightly run --all-features --release test.mkv Time (mean ± σ): 347.6 ms ± 12.3 ms [User: 208.8 ms, System: 136.1 ms] Range (min … max): 330.7 ms … 368.3 ms 10 runs d

Summary 'cargo +nightly run --all-features --release test.mkv' ran 1.12 ± 0.05 times faster than 'cargo +stable run --release -- test.mkv' 1.12 ± 0.05 times faster than 'cargo +nightly run --release -- test.mkv' 12.59 ± 0.56 times faster than 'cksfv test.mkv' ```

Using the generated SFV listing to check the same file, we get the following results:

``` Benchmark #1: cksfv -f test.sfv Time (mean ± σ): 4.672 s ± 0.102 s [User: 4.380 s, System: 0.269 s] Range (min … max): 4.526 s … 4.845 s 10 runs

Benchmark #2: cargo +stable run --release -- -f test.sfv Time (mean ± σ): 344.8 ms ± 12.0 ms [User: 145.9 ms, System: 194.1 ms] Range (min … max): 325.0 ms … 355.7 ms 10 runs

Benchmark #3: cargo +nightly run --release -- -f test.sfv Time (mean ± σ): 356.5 ms ± 7.8 ms [User: 144.4 ms, System: 210.2 ms] Range (min … max): 349.3 ms … 374.3 ms 10 runs

Benchmark #4: cargo +nightly run --all-features --release -- -f test.sfv Time (mean ± σ): 300.4 ms ± 12.4 ms [User: 197.3 ms, System: 99.8 ms] Range (min … max): 290.7 ms … 324.6 ms 10 runs

Summary 'cargo +nightly run --all-features --release -- -f test.sfv' ran 1.15 ± 0.06 times faster than 'cargo +stable run --release -- -f test.sfv' 1.19 ± 0.06 times faster than 'cargo +nightly run --release -- -f test.sfv' 15.55 ± 0.72 times faster than 'cksfv -f test.sfv' ```

📜 License

This tool is provided under the open-source MIT license.

Owner

  • Name: Martin Larralde
  • Login: althonos
  • Kind: user
  • Location: Heidelberg, Germany
  • Company: EMBL / LUMC, @zellerlab

PhD candidate in Bioinformatics, passionate about programming, SIMD-enthusiast, Pythonista, Rustacean. I write poems, and sometimes they are executable.

GitHub Events

Total
Last Year

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 51
  • Total Committers: 5
  • Avg Commits per committer: 10.2
  • Development Distribution Score (DDS): 0.471
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Martin Larralde m****e@e****r 27
dependabot-preview[bot] 2****] 12
Martin Larralde m****e@e****e 10
Ola Stensson s****n@g****m 1
Martin Lindhe m****e 1
Committer Domains (Top 20 + Academic)
embl.de: 1 ens-paris-saclay.fr: 1

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 1
  • Total pull requests: 100
  • Average time to close issues: 8 months
  • Average time to close pull requests: about 2 months
  • Total issue authors: 1
  • Total pull request authors: 4
  • Average comments per issue: 5.0
  • Average comments per pull request: 0.9
  • Merged pull requests: 4
  • Bot issues: 0
  • Bot pull requests: 98
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • zkxs (1)
Pull Request Authors
  • dependabot[bot] (90)
  • dependabot-preview[bot] (8)
  • stensson (1)
  • martinlindhe (1)
Top Labels
Issue Labels
Pull Request Labels
dependencies (98) enhancement (1)

Packages

  • Total packages: 1
  • Total downloads:
    • cargo 5,321 total
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 4
  • Total maintainers: 1
crates.io: cksfv

A 10x faster cksfv reimplementation using Rust and the crc32fast crate.

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 5,321 Total
Rankings
Forks count: 24.8%
Dependent repos count: 29.3%
Stargazers count: 29.5%
Dependent packages count: 33.8%
Average: 34.5%
Downloads: 55.1%
Maintainers (1)
althonos
Last synced: 11 months ago

Dependencies

Cargo.lock cargo
  • ansi_term 0.11.0
  • assert_cli 0.6.3
  • atty 0.2.14
  • autocfg 1.0.0
  • backtrace 0.3.46
  • backtrace-sys 0.1.35
  • bitflags 1.2.1
  • cc 1.0.50
  • cfg-if 0.1.10
  • chrono 0.4.13
  • clap 2.33.1
  • colored 1.9.3
  • crc32fast 1.2.0
  • difference 2.0.0
  • environment 0.1.1
  • failure 0.1.7
  • failure_derive 0.1.7
  • getrandom 0.1.14
  • getset 0.1.1
  • hermit-abi 0.1.10
  • itoa 0.4.5
  • lazy_static 1.4.0
  • libc 0.2.71
  • memmap 0.7.0
  • num-integer 0.1.42
  • num-traits 0.2.11
  • ppv-lite86 0.2.6
  • proc-macro-error 1.0.2
  • proc-macro-error-attr 1.0.2
  • proc-macro-hack 0.5.15
  • proc-macro2 1.0.10
  • quote 1.0.3
  • rand 0.7.3
  • rand_chacha 0.2.2
  • rand_core 0.5.1
  • rand_hc 0.2.0
  • redox_syscall 0.1.56
  • remove_dir_all 0.5.2
  • rustc-demangle 0.1.16
  • ryu 1.0.3
  • same-file 1.0.6
  • serde 1.0.106
  • serde_json 1.0.51
  • smawk 0.3.1
  • strsim 0.8.0
  • syn 1.0.17
  • syn-mid 0.5.0
  • synstructure 0.12.3
  • tempfile 3.1.0
  • textwrap 0.11.0
  • textwrap 0.12.1
  • textwrap 0.13.2
  • textwrap-macros 0.2.5
  • textwrap-macros-impl 0.2.5
  • time 0.1.43
  • unicode-width 0.1.7
  • unicode-xid 0.2.0
  • vec_map 0.8.1
  • version_check 0.9.1
  • walkdir 2.3.1
  • wasi 0.9.0+wasi-snapshot-preview1
  • winapi 0.3.8
  • winapi-i686-pc-windows-gnu 0.4.0
  • winapi-util 0.1.4
  • winapi-x86_64-pc-windows-gnu 0.4.0
Cargo.toml cargo
  • chrono 0.4.13
  • clap 2.33.1
  • crc32fast 1.2.0
  • getset 0.1.1
  • memmap 0.7.0
  • walkdir 2.3.1
.github/workflows/test.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/cache v2 composite
  • actions/checkout v1 composite
  • actions/checkout v3 composite
  • rasmus-saks/release-a-changelog-action v1.0.1 composite
  • taiki-e/upload-rust-binary-action v1 composite