Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (16.5%) to scientific vocabulary
Last synced: 11 months ago
·
JSON representation
Repository
Tokenizers from HuggingFace
Basic Info
Statistics
- Stars: 43
- Watchers: 2
- Forks: 3
- Open Issues: 1
- Releases: 5
Created over 3 years ago
· Last pushed 11 months ago
Metadata Files
Readme
Changelog
License
README.Rmd
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# tok
[](https://github.com/mlverse/tok/actions)
[](https://CRAN.R-project.org/package=tok)
[](https://cran.r-project.org/package=tok)
[](https://extendr.github.io/extendr/extendr_api/)
tok provides bindings to the [🤗tokenizers](https://huggingface.co/docs/tokenizers/v0.13.3/en/index) library. It uses the same Rust libraries that powers the Python implementation.
We still don't provide the full API of tokenizers. Please open a issue if there's
a feature you are missing.
## Installation
You can install tok from CRAN using:
```
install.packages("tok")
```
Installing tok from source requires working Rust toolchain. We recommend using [rustup.](https://rustup.rs/)
On Windows, you'll also have to add the `i686-pc-windows-gnu` and `x86_64-pc-windows-gnu` targets:
rustup target add x86_64-pc-windows-gnu
rustup target add i686-pc-windows-gnu
Once Rust is working, you can install this package via:
``` r
remotes::install_github("dfalbel/tok")
```
## Features
We still don't have complete support for the 🤗tokenizers API. Please open an issue
if you need a feature that is currently not implemented.
## Loading tokenizers
`tok` can be used to load and use tokenizers that have been previously serialized.
For example, HuggingFace model weights are usually accompanied by a 'tokenizer.json'
file that can be loaded with this library.
To load a pre-trained tokenizer from a json file, use:
```{r}
path <- testthat::test_path("assets/tokenizer.json")
tok <- tok::tokenizer$from_file(path)
```
Use the `encode` method to tokenize sentendes and `decode` to transform them back.
```{r}
enc <- tok$encode("hello world")
tok$decode(enc$ids)
```
## Using pre-trained tokenizers
You can also load any tokenizer available in HuggingFace hub by using the `from_pretrained`
static method. For example, let's load the GPT2 tokenizer with:
```{r}
tok <- tok::tokenizer$from_pretrained("gpt2")
enc <- tok$encode("hello world")
tok$decode(enc$ids)
```
Owner
- Name: mlverse
- Login: mlverse
- Kind: organization
- Repositories: 27
- Profile: https://github.com/mlverse
Open source libraries to scale Data Science
GitHub Events
Total
- Issues event: 3
- Watch event: 3
- Push event: 10
- Pull request event: 2
- Create event: 2
Last Year
- Issues event: 3
- Watch event: 3
- Push event: 10
- Pull request event: 2
- Create event: 2
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Daniel Falbel | d****l@g****m | 83 |
| cregouby | c****y | 1 |
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 12
- Total pull requests: 24
- Average time to close issues: 9 days
- Average time to close pull requests: 2 days
- Total issue authors: 2
- Total pull request authors: 2
- Average comments per issue: 0.25
- Average comments per pull request: 0.08
- Merged pull requests: 20
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 8
- Pull requests: 16
- Average time to close issues: 5 days
- Average time to close pull requests: 3 days
- Issue authors: 1
- Pull request authors: 2
- Average comments per issue: 0.0
- Average comments per pull request: 0.13
- Merged pull requests: 12
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- dfalbel (7)
- eitsupi (1)
- cregouby (1)
Pull Request Authors
- dfalbel (19)
- cregouby (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- cran 1,312 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 6
- Total maintainers: 1
cran.r-project.org: tok
Fast Text Tokenization
- Homepage: https://github.com/mlverse/tok
- Documentation: http://cran.r-project.org/web/packages/tok/tok.pdf
- License: MIT + file LICENSE
-
Latest release: 0.2.0
published 11 months ago
Rankings
Stargazers count: 11.2%
Forks count: 17.1%
Dependent repos count: 24.0%
Dependent packages count: 28.8%
Average: 32.5%
Downloads: 81.6%
Maintainers (1)
Last synced:
11 months ago
Dependencies
.github/workflows/R-CMD-check.yaml
actions
- actions-rs/toolchain v1 composite
- actions/checkout v2 composite
- r-lib/actions/check-r-package v2 composite
- r-lib/actions/setup-pandoc v2 composite
- r-lib/actions/setup-r v2 composite
- r-lib/actions/setup-r-dependencies v2 composite
src/rust/Cargo.lock
cargo
- extendr-api 0.3.1
- extendr-engine 0.3.1
- extendr-macros 0.3.1
- lazy_static 1.4.0
- libR-sys 0.3.0
- paste 1.0.11
- proc-macro2 1.0.49
- quote 1.0.23
- syn 1.0.107
- unicode-ident 1.0.6
- winapi 0.3.9
- winapi-i686-pc-windows-gnu 0.4.0
- winapi-x86_64-pc-windows-gnu 0.4.0
DESCRIPTION
cran
- R >= 4.2.0 depends
- R6 * imports
- cli * imports
- hfhub * suggests
- rmarkdown * suggests
- testthat >= 3.0.0 suggests
src/rust/Cargo.toml
cargo