strings.bpe_tokenizer
=====================

.. py:module:: strings.bpe_tokenizer

.. autoapi-nested-parse::

   Byte-Pair Encoding: Subword-based tokenization algorithm used by
   state-of-the-art language models.

   Wikipedia: https://en.wikipedia.org/wiki/Byte_pair_encoding



Classes
-------

.. autoapisummary::

   strings.bpe_tokenizer.Tokenizer


Functions
---------

.. autoapisummary::

   strings.bpe_tokenizer.get_byte_pair_counts
   strings.bpe_tokenizer.merge


Module Contents
---------------

.. py:class:: Tokenizer(num_merges: int = 20, verbose: bool = False)

   Tokenize a string using the byte-pair encoding algorithm


   .. py:method:: decode(ids: list[int]) -> str

      Convert a list of tokens to the original string

      >>> t = Tokenizer()
      >>> ids = [73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]
      >>> t.decode(ids)
      'I am JonSnow.'

      >>> t = Tokenizer()
      >>> ids = []
      >>> t.decode(ids)
      ''



   .. py:method:: encode(text: str) -> list[int]

      Convert a string to tokens (bytes)

      >>> t = Tokenizer()
      >>> text = "I am JonSnow."
      >>> t.encode(text)
      [73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]

      >>> t = Tokenizer()
      >>> text = ""
      >>> t.encode(text)
      []



   .. py:attribute:: merges
      :type:  dict


   .. py:attribute:: num_merges
      :value: 20



   .. py:attribute:: verbose
      :value: False



.. py:function:: get_byte_pair_counts(ids: list[int]) -> dict

   Count consecutive byte-pairs of an encoded string.

   >>> ids = [73, 32, 97, 109, 32, 74, 111, 110, 83, 110, 111, 119, 46]
   >>> get_byte_pair_counts(ids)
   {(73, 32): 1, (32, 97): 1, (97, 109): 1, (109, 32): 1, (32, 74): 1, (74, 111): 1, (111, 110): 1, (110, 83): 1, (83, 110): 1, (110, 111): 1, (111, 119): 1, (119, 46): 1}
   >>> ids = [2, 3, 6, 2, 3, 6, 2, 5]
   >>> get_byte_pair_counts(ids)
   {(2, 3): 2, (3, 6): 2, (6, 2): 2, (2, 5): 1}


.. py:function:: merge(ids: list[int], pair: tuple, idx: int) -> list[int]

   Replace most occurring byte pair with new byte that is not used
   in the data. For utf-8 encoding, we start with 256 as the new byte

   >>> ids = [2, 3, 6, 2, 3, 6, 2, 5]
   >>> pair = (2, 3)
   >>> idx = 256
   >>> merge(ids, pair, idx)
   [256, 6, 256, 6, 2, 5]