tok - v0.1.4

• Updated libR-sys to fix mac oldrel notes. (#18)
• Report rust version during compilation.

- R
Published by dfalbel almost 2 years ago

tok - v0.1.3

- R
Published by dfalbel about 2 years ago

tok - v0.1.2

• Added support for authentication. (#7)
• Added support for training tokenizers from scratch. (#8)
• Added support for serializing tokenizers. (#8)
• Implemented support for configuring padding and truncation. (#9)
• Added support for decoders, post_processors and normalizers. (#10)
• Added support for get_vocab_size(). (#10)
• Updated extendr version to avoid non-API function calls. (#11)

- R
Published by dfalbel about 2 years ago

tok - v0.1.1

• Dependencies are now vendored to avoid versioning problems on CRAN (#5)

- R
Published by dfalbel almost 3 years ago

tok - v0.1.0

• Initial CRAN submission.

- R
Published by dfalbel about 3 years ago