Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/cahya-wirawan/json2bin

A fast jsonl to binidx converter in Rust

https://github.com/cahya-wirawan/json2bin

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (7.1%) to scientific vocabulary
Last synced: 11 months ago · JSON representation

Repository

A fast jsonl to binidx converter in Rust

Basic Info
  • Host: GitHub
  • Owner: cahya-wirawan
  • License: apache-2.0
  • Language: Rust
  • Default Branch: main
  • Size: 311 KB
Statistics
  • Stars: 7
  • Watchers: 2
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Created about 2 years ago · Last pushed over 1 year ago
Metadata Files
Readme License

README.md

Json2bin

Crates.io Version Crates.io Downloads License: Apache 2.0

A fast multithreading Jsonl converter to RWKV binidx files written in Rust.

performance-multithreading

Installation

$ cargo install json2bin

Usage

``` $ json2bin -h Json converter to RWKV binidx file format Usage: json2bin [OPTIONS] --input

Options: -i, --input Jsonlines file to read -o, --output-dir Output directory for binidx files [default: -] -t, --thread Number of threads [default: 8] -v, --verbose Verbosity -c, --context-length Context Length [default: 4096] -h, --help Print help -V, --version Print version Following command will convert the jsonl file src/sample.jsonl into src/sample.bin and src/sample.idx files. $ json2bin -i src/sample.jsonl The output directory can be set with the argument "--output-dir <OUTPUT_DIR>" or "-o <OUTPUT_DIR>" $ json2bin -i src/sample.jsonl -o output The default threads number is 8, it can be changed with the argument "--thread" or "-t" $ json2bin -i src/sample.jsonl -t 4 ```

Performance comparison

We converted a 19GB English Wikipedia (20231101.en) in jsonl format to binidx format in M2 Apple machine. The Rust json2bin run with 7 threads, and it was 70 times faster than the Python json2binidx: - The Python json2binidx: 1:01:45 or 5.13MB/s - This Rust json2bin: 52.64s or 360.86MB/s

Owner

  • Name: Cahya Wirawan
  • Login: cahya-wirawan
  • Kind: user
  • Location: Vienna, Austria

System engineer, currently working on NLP, CV and Speech Recognition for fun and curiosity

GitHub Events

Total
  • Watch event: 6
  • Push event: 4
  • Create event: 1
Last Year
  • Watch event: 6
  • Push event: 4
  • Create event: 1

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 33
  • Total Committers: 1
  • Avg Commits per committer: 33.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 33
  • Committers: 1
  • Avg Commits per committer: 33.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Cahya Wirawan c****n@g****m 33

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

Cargo.lock cargo
  • aho-corasick 1.1.3
  • anstream 0.6.14
  • anstyle 1.0.7
  • anstyle-parse 0.2.4
  • anstyle-query 1.1.0
  • anstyle-wincon 3.0.3
  • anyhow 1.0.86
  • autocfg 1.3.0
  • bitflags 1.3.2
  • bitflags 2.6.0
  • bytemuck 1.16.1
  • cfg-if 1.0.0
  • clap 4.5.9
  • clap_builder 4.5.9
  • clap_derive 4.5.8
  • clap_lex 0.7.1
  • colorchoice 1.0.1
  • crossterm 0.25.0
  • crossterm_winapi 0.9.1
  • heck 0.5.0
  • is_terminal_polyfill 1.70.0
  • itoa 1.0.11
  • libc 0.2.155
  • lock_api 0.4.12
  • log 0.4.22
  • memchr 2.7.4
  • mio 0.8.11
  • once_cell 1.19.0
  • parking_lot 0.12.3
  • parking_lot_core 0.9.10
  • proc-macro2 1.0.86
  • quote 1.0.36
  • redox_syscall 0.5.2
  • regex 1.10.5
  • regex-automata 0.4.7
  • regex-syntax 0.8.4
  • rwkv-tokenizer 0.8.5
  • ryu 1.0.18
  • scopeguard 1.2.0
  • serde 1.0.204
  • serde_derive 1.0.204
  • serde_json 1.0.120
  • signal-hook 0.3.17
  • signal-hook-mio 0.2.3
  • signal-hook-registry 1.4.2
  • smallvec 1.13.2
  • strsim 0.11.1
  • syn 2.0.70
  • tqdm 0.7.0
  • unescape 0.1.0
  • unicode-ident 1.0.12
  • utf8parse 0.2.2
  • wasi 0.11.0+wasi-snapshot-preview1
  • winapi 0.3.9
  • winapi-i686-pc-windows-gnu 0.4.0
  • winapi-x86_64-pc-windows-gnu 0.4.0
  • windows-sys 0.48.0
  • windows-sys 0.52.0
  • windows-targets 0.48.5
  • windows-targets 0.52.6
  • windows_aarch64_gnullvm 0.48.5
  • windows_aarch64_gnullvm 0.52.6
  • windows_aarch64_msvc 0.48.5
  • windows_aarch64_msvc 0.52.6
  • windows_i686_gnu 0.48.5
  • windows_i686_gnu 0.52.6
  • windows_i686_gnullvm 0.52.6
  • windows_i686_msvc 0.48.5
  • windows_i686_msvc 0.52.6
  • windows_x86_64_gnu 0.48.5
  • windows_x86_64_gnu 0.52.6
  • windows_x86_64_gnullvm 0.48.5
  • windows_x86_64_gnullvm 0.52.6
  • windows_x86_64_msvc 0.48.5
  • windows_x86_64_msvc 0.52.6
Cargo.toml cargo