BBPE16 Tokenizer

Why BBPE16 (UTF-16 Byte-level Byte-Pair Encoding)?

BBPE16 (UTF-16 Byte-level Byte-Pair Encoding) represents a different approach from traditional Byte-Level BPE (BBPE), particularly in how it handles character encoding internally:

• UTF-16 vs UTF-8 Processing: While standard BBPE processes text at the UTF-8 byte level, BBPE16 operates on UTF-16 byte sequences internally. This fundamental difference affects how characters are segmented and merged during the BPE algorithm.

• Seamless Integration: When encoding, BBPE16 automatically converts UTF-8 strings to UTF-16 internally for processing, and during decoding, it processes tokens based on UTF-16 before returning standard UTF-8 strings. This makes it transparent to users—no manual encoding or decoding steps are required.

• Different Character Representation: UTF-16 uses 16-bit code units, representing most common characters in a single unit while using surrogate pairs for characters beyond the Basic Multilingual Plane. UTF-8, in contrast, uses variable-length encoding with 1-4 bytes per character. This encoding difference affects how the BPE algorithm identifies and merges byte patterns.

• CJK Language Characteristics: CJK characters typically require 3 bytes in UTF-8 encoding but fit within a single 16-bit unit in UTF-16 (for characters in the Basic Multilingual Plane). This difference in representation affects the byte-level patterns that BPE algorithms learn and apply.

• Same Alphabet Foundation: BBPE16 maintains compatibility with the standard 256-character byte-level alphabet used in traditional BBPE, ensuring consistent vocabulary mapping strategies.

• Automatic Model Detection: The tokenizer can automatically distinguish between BBPE and BBPE16 models when loading, providing seamless integration without requiring manual configuration.

UTF16ByteLevelBPETokenizer

A specialized tokenizer that processes text at the UTF-16 byte level while maintaining compatibility with the standard ByteLevel alphabet, offering an alternative approach to traditional UTF-8 byte-level processing.

Key Characteristics:

• Processes text using UTF-16 internal representation
• Maintains 100% accuracy across all languages
• Compatible byte-level alphabet (256 characters, same mapping strategy as ByteLevel)
• Transparent UTF-8 to UTF-16 conversion during processing

Usage:

```python from tokenizers.implementations import UTF16ByteLevelBPETokenizer

Initialize tokenizer

tokenizer = UTF16ByteLevelBPETokenizer()

Train on your dataset

tokenizer.train(files=["trainingdata.txt"], vocabsize=1000)

Encode text with UTF-16 internal processing

output = tokenizer.encode("안녕하세요 你好 こんにちは") print(f"Tokens: {output.tokens}") ```

When to Consider:

• Working with applications that benefit from UTF-16 internal processing
• Exploring alternative byte-level tokenization approaches
• Building systems where UTF-16 representation aligns with other components
• Research comparing different encoding approaches in tokenization

Technical Differences:

The core distinction lies in the internal character representation during BPE processing. While UTF-8 BBPE works with variable-length byte sequences (1-4 bytes per character), UTF-16 BBPE processes 16-bit code units, which affects how byte patterns are identified and merged during vocabulary construction.

Citation:

If you use UTF16ByteLevelBPETokenizer in your research or applications, please cite:

bibtex @misc{kim2025utf16bytelevel, title={UTF16ByteLevelBPETokenizer: Alternative UTF-16 Based Tokenization}, author={Hyunsik Kim}, year={2025}, month={May}, note={Implementation based on HuggingFace Tokenizers library}, email={avantkim@gmail.com}, url={https://github.com/peblove/tokenizers} }

Technical References:

• Base Implementation: HuggingFace Tokenizers - https://github.com/huggingface/tokenizers
• GPT-2 Byte-Level BPE: Radford et al. (2019) - https://github.com/openai/gpt-2
• UTF-16 Encoding Standard: Unicode Consortium - https://unicode.org/standard/standard.html

Performance Benchmarks

Performance characteristics can vary depending on hardware configuration. Running the ~/bindings/python/benches/test_tiktoken.py benchmark provides comparative analysis on various systems:

Bindings

We provide bindings to the following languages (more to come!): - Rust (Original implementation) - Python - Node.js - Ruby (Contributed by @ankane, external repo)

Installation

You can install from source using: bash pip install git+https://github.com/peblove/tokenizers.git#subdirectory=bindings/python

or install the released versions with:

bash pip install tokenizers

Quick Example Using Python

Choose your model between Byte-Pair Encoding, WordPiece or Unigram and instantiate a tokenizer:

```python from tokenizers import Tokenizer from tokenizers.models import BPE

tokenizer = Tokenizer(BPE()) ```

You can customize how pre-tokenization (e.g., splitting into words) is done:

```python from tokenizers.pre_tokenizers import Whitespace

tokenizer.pre_tokenizer = Whitespace() ```

Then training your tokenizer on a set of files just takes two lines of code:

```python from tokenizers.trainers import BpeTrainer

trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) tokenizer.train(files=["wiki.train.raw", "wiki.valid.raw", "wiki.test.raw"], trainer=trainer) ```

Once your tokenizer is trained, encode any text with just one line: ```python output = tokenizer.encode("Hello, y'all! How are you 😁 ?") print(output.tokens)

["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]

```

Check the documentation or the quicktour to learn more!