machine_learning.mab
====================

.. py:module:: machine_learning.mab

.. autoapi-nested-parse::

   Multi-Armed Bandit (MAB) is a problem in reinforcement learning where an agent must
   learn to choose the best action from a set of actions to maximize its reward.

   learn more here: https://en.wikipedia.org/wiki/Multi-armed_bandit


   The MAB problem can be described as follows:
   - There are N arms, each with a different probability of giving a reward.
   - The agent must learn to choose the best arm to pull in order to maximize its reward.

   Here 3 optimising strategies have been implemented:
   - Epsilon-Greedy
   - Upper Confidence Bound (UCB)
   - Thompson Sampling

   There are two other strategies implemented to show the performance of
   the optimising strategies:
   - Random strategy (full exploration)
   - Greedy strategy (full exploitation)

   The performance of the strategies is evaluated by the cumulative reward
   over a number of rounds.



Classes
-------

.. autoapisummary::

   machine_learning.mab.Bandit
   machine_learning.mab.EpsilonGreedy
   machine_learning.mab.GreedyStrategy
   machine_learning.mab.RandomStrategy
   machine_learning.mab.Strategy
   machine_learning.mab.ThompsonSampling
   machine_learning.mab.UCB


Functions
---------

.. autoapisummary::

   machine_learning.mab.demo_mab_strategies
   machine_learning.mab.test_mab_strategies


Module Contents
---------------

.. py:class:: Bandit(probabilities: list[float])

   A class to represent a multi-armed bandit.


   .. py:method:: pull(arm_index: int) -> int

      Pull an arm of the bandit.

      Args:
          arm_index: The arm to pull.

      Returns:
          The reward for the arm.

      Example:
          >>> bandit = Bandit([0.1, 0.5, 0.9])
          >>> isinstance(bandit.pull(0), int)
          True



   .. py:attribute:: num_arms


   .. py:attribute:: probabilities


.. py:class:: EpsilonGreedy(epsilon: float, num_arms: int)

   Bases: :py:obj:`Strategy`


   A class for a simple implementation of the Epsilon-Greedy strategy.
   Follow this link to learn more:
   https://medium.com/analytics-vidhya/the-epsilon-greedy-algorithm-for-reinforcement-learning-5fe6f96dc870


   .. py:method:: select_arm() -> int

      Select an arm to pull.

      Returns:
          The index of the arm to pull.

      Example:
          >>> strategy = EpsilonGreedy(epsilon=0.1, num_arms=3)
          >>> 0 <= strategy.select_arm() < 3
          True



   .. py:method:: update(arm_index: int, reward: int) -> None

      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.

      Example:
          >>> strategy = EpsilonGreedy(epsilon=0.1, num_arms=3)
          >>> strategy.update(0, 1)
          >>> strategy.counts[0] == 1
          np.True_



   .. py:attribute:: counts


   .. py:attribute:: epsilon


   .. py:attribute:: num_arms


   .. py:attribute:: values


.. py:class:: GreedyStrategy(num_arms: int)

   Bases: :py:obj:`Strategy`


   A class for the Greedy strategy to show how full exploitation can be
   detrimental to the performance of the strategy.


   .. py:method:: select_arm() -> int

      Select an arm to pull.

      Returns:
          The index of the arm to pull.

      Example:
          >>> strategy = GreedyStrategy(num_arms=3)
          >>> 0 <= strategy.select_arm() < 3
          True



   .. py:method:: update(arm_index: int, reward: int) -> None

      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.

      Example:
          >>> strategy = GreedyStrategy(num_arms=3)
          >>> strategy.update(0, 1)
          >>> strategy.counts[0] == 1
          np.True_



   .. py:attribute:: counts


   .. py:attribute:: num_arms


   .. py:attribute:: values


.. py:class:: RandomStrategy(num_arms: int)

   Bases: :py:obj:`Strategy`


   A class for choosing an arm uniformly at random at each round to give
   a better comparison with the other optimised strategies.


   .. py:method:: select_arm() -> int

      Select an arm to pull.

      Returns:
          The index of the arm to pull.

      Example:
          >>> strategy = RandomStrategy(num_arms=3)
          >>> 0 <= strategy.select_arm() < 3
          True



   .. py:method:: update(arm_index: int, reward: int) -> None

      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.

      Example:
          >>> strategy = RandomStrategy(num_arms=3)
          >>> strategy.update(0, 1)



   .. py:attribute:: num_arms


.. py:class:: Strategy

   Bases: :py:obj:`abc.ABC`


   Base class for all strategies.


   .. py:method:: select_arm() -> int
      :abstractmethod:


      Select an arm to pull.

      Returns:
          The index of the arm to pull.



   .. py:method:: update(arm_index: int, reward: int) -> None
      :abstractmethod:


      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.



.. py:class:: ThompsonSampling(num_arms: int)

   Bases: :py:obj:`Strategy`


   A class for the Thompson Sampling strategy.
   Follow this link to learn more:
   https://en.wikipedia.org/wiki/Thompson_sampling


   .. py:method:: select_arm() -> int

      Select an arm to pull.

      Returns:
          The index of the arm to pull based on the Thompson Sampling strategy
          which relies on the Beta distribution.

      Example:
          >>> strategy = ThompsonSampling(num_arms=3)
          >>> 0 <= strategy.select_arm() < 3
          True



   .. py:method:: update(arm_index: int, reward: int) -> None

      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.

      Example:
          >>> strategy = ThompsonSampling(num_arms=3)
          >>> strategy.update(0, 1)
          >>> strategy.successes[0] == 1
          np.True_



   .. py:attribute:: failures


   .. py:attribute:: num_arms


   .. py:attribute:: successes


.. py:class:: UCB(num_arms: int)

   Bases: :py:obj:`Strategy`


   A class for the Upper Confidence Bound (UCB) strategy.
   Follow this link to learn more:
   https://people.maths.bris.ac.uk/~maajg/teaching/stochopt/ucb.pdf


   .. py:method:: select_arm() -> int

      Select an arm to pull.

      Returns:
          The index of the arm to pull.

      Example:
          >>> strategy = UCB(num_arms=3)
          >>> 0 <= strategy.select_arm() < 3
          True



   .. py:method:: update(arm_index: int, reward: int) -> None

      Update the strategy.

      Args:
          arm_index: The index of the arm to pull.
          reward: The reward for the arm.

      Example:
          >>> strategy = UCB(num_arms=3)
          >>> strategy.update(0, 1)
          >>> strategy.counts[0] == 1
          np.True_



   .. py:attribute:: counts


   .. py:attribute:: num_arms


   .. py:attribute:: total_counts
      :value: 0



   .. py:attribute:: values


.. py:function:: demo_mab_strategies() -> None

   Run a stochastic simulation of the MAB strategies and plot their
   cumulative reward over time for visual comparison.


.. py:function:: test_mab_strategies() -> None

   Deterministic behavioural tests for the MAB strategies.

   These checks feed each strategy a fixed sequence of rewards and assert
   on the resulting internal state and arm selection, so a regression in
   the update/select logic will fail the suite instead of only being
   visible in the (stochastic) plotted demo.