machine_learning.mab

Multi-Armed Bandit (MAB) is a problem in reinforcement learning where an agent must learn to choose the best action from a set of actions to maximize its reward.

learn more here: https://en.wikipedia.org/wiki/Multi-armed_bandit

The MAB problem can be described as follows: - There are N arms, each with a different probability of giving a reward. - The agent must learn to choose the best arm to pull in order to maximize its reward.

Here 3 optimising strategies have been implemented: - Epsilon-Greedy - Upper Confidence Bound (UCB) - Thompson Sampling

There are two other strategies implemented to show the performance of the optimising strategies: - Random strategy (full exploration) - Greedy strategy (full exploitation)

The performance of the strategies is evaluated by the cumulative reward over a number of rounds.

Classes

Bandit	A class to represent a multi-armed bandit.
EpsilonGreedy	A class for a simple implementation of the Epsilon-Greedy strategy.
GreedyStrategy	A class for the Greedy strategy to show how full exploitation can be
RandomStrategy	A class for choosing an arm uniformly at random at each round to give
Strategy	Base class for all strategies.
ThompsonSampling	A class for the Thompson Sampling strategy.
UCB	A class for the Upper Confidence Bound (UCB) strategy.

Functions

demo_mab_strategies(→ None)	Run a stochastic simulation of the MAB strategies and plot their
test_mab_strategies(→ None)	Deterministic behavioural tests for the MAB strategies.

Module Contents

class machine_learning.mab.Bandit(probabilities: list[float])

A class to represent a multi-armed bandit.

pull(arm_index: int) → int

Pull an arm of the bandit.

Args:

arm_index: The arm to pull.

Returns:

The reward for the arm.

Example:

>>> bandit = Bandit([0.1, 0.5, 0.9])
>>> isinstance(bandit.pull(0), int)
True

num_arms

probabilities

class machine_learning.mab.EpsilonGreedy(epsilon: float, num_arms: int)

Bases: Strategy

A class for a simple implementation of the Epsilon-Greedy strategy. Follow this link to learn more: https://medium.com/analytics-vidhya/the-epsilon-greedy-algorithm-for-reinforcement-learning-5fe6f96dc870

select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull.

Example:

>>> strategy = EpsilonGreedy(epsilon=0.1, num_arms=3)
>>> 0 <= strategy.select_arm() < 3
True

update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

Example:

>>> strategy = EpsilonGreedy(epsilon=0.1, num_arms=3)
>>> strategy.update(0, 1)
>>> strategy.counts[0] == 1
np.True_

counts

epsilon

num_arms

values

class machine_learning.mab.GreedyStrategy(num_arms: int)

Bases: Strategy

A class for the Greedy strategy to show how full exploitation can be detrimental to the performance of the strategy.

select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull.

Example:

>>> strategy = GreedyStrategy(num_arms=3)
>>> 0 <= strategy.select_arm() < 3
True

update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

Example:

>>> strategy = GreedyStrategy(num_arms=3)
>>> strategy.update(0, 1)
>>> strategy.counts[0] == 1
np.True_

counts

num_arms

values

class machine_learning.mab.RandomStrategy(num_arms: int)

Bases: Strategy

A class for choosing an arm uniformly at random at each round to give a better comparison with the other optimised strategies.

select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull.

Example:

>>> strategy = RandomStrategy(num_arms=3)
>>> 0 <= strategy.select_arm() < 3
True

update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

Example:

>>> strategy = RandomStrategy(num_arms=3)
>>> strategy.update(0, 1)

num_arms

class machine_learning.mab.Strategy

Bases: abc.ABC

Base class for all strategies.

abstractmethod select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull.

abstractmethod update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

class machine_learning.mab.ThompsonSampling(num_arms: int)

Bases: Strategy

A class for the Thompson Sampling strategy. Follow this link to learn more: https://en.wikipedia.org/wiki/Thompson_sampling

select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull based on the Thompson Sampling strategy which relies on the Beta distribution.

Example:

>>> strategy = ThompsonSampling(num_arms=3)
>>> 0 <= strategy.select_arm() < 3
True

update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

Example:

>>> strategy = ThompsonSampling(num_arms=3)
>>> strategy.update(0, 1)
>>> strategy.successes[0] == 1
np.True_

failures

num_arms

successes

class machine_learning.mab.UCB(num_arms: int)

Bases: Strategy

A class for the Upper Confidence Bound (UCB) strategy. Follow this link to learn more: https://people.maths.bris.ac.uk/~maajg/teaching/stochopt/ucb.pdf

select_arm() → int

Select an arm to pull.

Returns:

The index of the arm to pull.

Example:

>>> strategy = UCB(num_arms=3)
>>> 0 <= strategy.select_arm() < 3
True

update(arm_index: int, reward: int) → None

Update the strategy.

Args:

arm_index: The index of the arm to pull. reward: The reward for the arm.

Example:

>>> strategy = UCB(num_arms=3)
>>> strategy.update(0, 1)
>>> strategy.counts[0] == 1
np.True_

counts

num_arms

total_counts = 0

values

machine_learning.mab.demo_mab_strategies() → None

Run a stochastic simulation of the MAB strategies and plot their cumulative reward over time for visual comparison.

machine_learning.mab.test_mab_strategies() → None

Deterministic behavioural tests for the MAB strategies.

These checks feed each strategy a fixed sequence of rewards and assert on the resulting internal state and arm selection, so a regression in the update/select logic will fail the suite instead of only being visible in the (stochastic) plotted demo.