https://github.com/mdrokz/rust-llama.cpp

LLama.cpp rust bindings

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
1 of 12 committers (8.3%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (8.2%) to scientific vocabulary

Keywords

api-bindings cpp crates-io ffi llama llama-cpp machine-learning model rust

Last synced: 5 months ago · JSON representation

Repository

LLama.cpp rust bindings

Basic Info

Host: GitHub
Owner: mdrokz
License: mit
Language: Rust
Default Branch: master
Homepage: https://crates.io/crates/llama_cpp_rs/
Size: 62.5 KB

Statistics

Stars: 399
Watchers: 8
Forks: 48
Open Issues: 15
Releases: 0

Topics

api-bindings cpp crates-io ffi llama llama-cpp machine-learning model rust

Created over 2 years ago · Last pushed over 1 year ago

Metadata Files

Readme License

README.md

rust_llama.cpp

LLama.cpp rust bindings.

The rust bindings are mostly based on https://github.com/go-skynet/go-llama.cpp/

Building Locally

Note: This repository uses git submodules to keep track of LLama.cpp.

Clone the repository locally:

bash git clone --recurse-submodules https://github.com/mdrokz/rust-llama.cpp

bash cargo build

Usage

toml [dependencies] llama_cpp_rs = "0.3.0"

```rs use llamacpprs::{ options::{ModelOptions, PredictOptions}, LLama, };

fn main() { let model_options = ModelOptions::default();

let llama = LLama::new(
    "../wizard-vicuna-13B.ggmlv3.q4_0.bin".into(),
    &model_options,
)
.unwrap();

let predict_options = PredictOptions {
    token_callback: Some(Box::new(|token| {
        println!("token1: {}", token);

        true
    })),
    ..Default::default()
};

llama
    .predict(
        "what are the national animals of india".into(),
         predict_options,
    )
    .unwrap();

}

```

Examples

The examples contain dockerfiles to run them

see examples

TODO

[x] Implement support for cublas,openBLAS & OpenCL #7
[x] Implement support for GPU (Metal)
[ ] Add some test cases
[ ] Support for fetching models through http & S3
[x] Sync with latest master & support GGUF
[x] Add some proper examples https://github.com/mdrokz/rust-llama.cpp/pull/7

LICENSE

MIT

Owner

Name: mdrokz
Login: mdrokz
Kind: user
Location: India, Madhya Pradesh
Company: @MakerStudio-io

Repositories: 168
Profile: https://github.com/mdrokz

Tech prodigy since age 15, mastering languages and tech. Secured job at 18, showcasing exceptional skills. Gamer, seeker of knowledge, and versatile programmer.

GitHub Events

Total

Issues event: 1
Watch event: 69
Fork event: 6

Last Year

Issues event: 1
Watch event: 69
Fork event: 6

Committers

Last synced: 6 months ago

All Time

Total Commits: 95
Total Committers: 12
Avg Commits per committer: 7.917
Development Distribution Score (DDS): 0.274

Past Year

Commits: 0
Committers: 0
Avg Commits per committer: 0.0
Development Distribution Score (DDS): 0.0

Top Committers

Name	Email	Commits
mdrokz	m**i@g**m	69
Gabin Lefranc	g**c@g**m	13
Mark CDA	a**e@g**m	3
Jeff McDonald	m**2@o**u	2
kouta	d**m@v**y	1
Tommy van der Vorst	t**y@p**l	1
Nick Gregory	c**7@g**m	1
Giovanni Merlino	g**o@u**t	1
Dmitry Savintsev	d****s	1
Benjamin Massey	b**y@g**m	1
André Hoffmann	a**6@g**m	1
bedwards	b**s@j**m	1

Committer Domains (Top 20 + Academic)

jalopymusic.com: 1 unime.it: 1 pixelspark.nl: 1 vera.com.uy: 1 osu.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 23
Total pull requests: 23
Average time to close issues: 20 days
Average time to close pull requests: 3 days
Total issue authors: 20
Total pull request authors: 15
Average comments per issue: 2.78
Average comments per pull request: 0.57
Merged pull requests: 19
Bot issues: 0
Bot pull requests: 1

Past Year

Issues: 2
Pull requests: 0
Average time to close issues: about 5 hours
Average time to close pull requests: N/A
Issue authors: 2
Pull request authors: 0
Average comments per issue: 0.0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

jasonw247 (2)
Scaletta (2)
alescdb (1)
JewishLewish (1)
GIOVANNI-75 (1)
Kuinox (1)
RussellCanfield (1)
RodrigoSdeCarvalho (1)
benbot (1)
philschmid (1)
vnermolaev (1)
Philipp-Sc (1)
tc-wolf (1)
pixelspark (1)
werruww (1)

Pull Request Authors

mdrokz (5)
tc-wolf (2)
BenjaminMassey (2)
gmerlino (2)
glcraft (2)
casualjim (2)
markcda (2)
kallsyms (2)
bedwards (2)
jmickeyd (1)
kouta-kun (1)
dependabot[bot] (1)
janeisklar (1)
pixelspark (1)
dmitris (1)

Top Labels

Issue Labels

bug (5) question (1)

Pull Request Labels

enhancement (10) bugfix (5) documentation (3) dependencies (1)

Dependencies

Cargo.lock cargo

bindgen 0.66.1
bitflags 2.3.3
cc 1.0.79
cexpr 0.6.0
cfg-if 1.0.0
clang-sys 1.6.1
either 1.8.1
glob 0.3.1
lazy_static 1.4.0
lazycell 1.3.0
libc 0.2.147
libloading 0.7.4
log 0.4.19
memchr 2.5.0
minimal-lexical 0.2.1
nom 7.1.3
once_cell 1.18.0
peeking_take_while 0.1.2
prettyplease 0.2.9
proc-macro2 1.0.63
quote 1.0.29
regex 1.8.4
regex-syntax 0.7.2
rustc-hash 1.1.0
shlex 1.1.0
syn 2.0.22
unicode-ident 1.0.9
which 4.4.0
winapi 0.3.9
winapi-i686-pc-windows-gnu 0.4.0
winapi-x86_64-pc-windows-gnu 0.4.0

Cargo.toml cargo

examples/basic/Cargo.toml cargo

examples/cuda/Cargo.toml cargo

examples/opencl/Cargo.toml cargo

examples/cuda/Dockerfile docker

nvidia/cuda 12.2.0-devel-ubuntu20.04 build

examples/opencl/Dockerfile docker

cebxan/amdgpu-opencl latest build