The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

tok 0.2.0

• Updated upstream tokenizers to 0.20.3
• Update extendr-api to 0.8.1

tok 0.1.5

• add message translation in FR (#19, @cregouby)

tok 0.1.4

• Updated libR-sys to fix mac oldrel notes. (#18)
• Report rust version during compilation.

tok 0.1.3

tok 0.1.2

• Added support for authentication. (#7)
• Added support for training tokenizers from scratch. (#8)
• Added support for serializing tokenizers. (#8)
• Implemented support for configuring padding and truncation. (#9)
• Added support for decoders, post_processors and normalizers. (#10)
• Added support for get_vocab_size(). (#10)
• Updated extendr version to avoid non-API function calls. (#11)

tok 0.1.1

• Dependencies are now vendored to avoid versioning problems on CRAN. (#5)

tok 0.1.0

• Initial CRAN submission.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.