strings.bpe_tokenizer

Byte-Pair Encoding: Subword-based tokenization algorithm used by state-of-the-art language models.

Wikipedia: https://en.wikipedia.org/wiki/Byte_pair_encoding

Classes

Tokenizer

Tokenize a string using the byte-pair encoding algorithm

Functions

get_byte_pair_counts(→ dict)	Count consecutive byte-pairs of an encoded string.
merge(→ list[int])	Replace most occurring byte pair with new byte that is not used

Module Contents

class strings.bpe_tokenizer.Tokenizer(num_merges: int = 20, verbose: bool = False)

Tokenize a string using the byte-pair encoding algorithm

decode(ids: list[int]) → str

Convert a list of tokens to the original string

>>> t = Tokenizer()
>>> ids = [73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]
>>> t.decode(ids)
'I am JonSnow.'

>>> t = Tokenizer()
>>> ids = []
>>> t.decode(ids)
''

encode(text: str) → list[int]

Convert a string to tokens (bytes)

>>> t = Tokenizer()
>>> text = "I am JonSnow."
>>> t.encode(text)
[73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]

>>> t = Tokenizer()
>>> text = ""
>>> t.encode(text)
[]

merges: dict

num_merges = 20

verbose = False

strings.bpe_tokenizer.get_byte_pair_counts(ids: list[int]) → dict

Count consecutive byte-pairs of an encoded string.

>>> ids = [73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]
>>> get_byte_pair_counts(ids)
{(73, 32): 1, (32, 97): 1, (97, 109): 1, (109, 32): 1, (32, 74): 1, (74, 111): 1, (111, 110): 1, (110, 83): 1, (83, 110): 1, (110, 111): 1, (111, 119): 1, (119, 46): 1}
>>> ids = [2, 3, 6, 2, 3, 6, 2, 5]
>>> get_byte_pair_counts(ids)
{(2, 3): 2, (3, 6): 2, (6, 2): 2, (2, 5): 1}

strings.bpe_tokenizer.merge(ids: list[int], pair: tuple, idx: int) → list[int]

Replace most occurring byte pair with new byte that is not used in the data. For utf-8 encoding, we start with 256 as the new byte

>>> ids = [2, 3, 6, 2, 3, 6, 2, 5]
>>> pair = (2, 3)
>>> idx = 256
>>> merge(ids, pair, idx)
[256, 6, 256, 6, 2, 5]