machine_learning.q_learning

Q-Learning is a widely-used model-free algorithm in reinforcement learning that learns the optimal action-value function Q(s, a), which tells an agent the expected utility of taking action a in state s and then following the optimal policy after. It is able to find the best policy for any given finite Markov decision process (MDP) without requiring a model of the environment.

See: [https://en.wikipedia.org/wiki/Q-learning](https://en.wikipedia.org/wiki/Q-learning)

Attributes

DISCOUNT_FACTOR
EPSILON
EPSILON_DECAY
EPSILON_MIN
GOAL
LEARNING_RATE
SIZE
State
current_state
q_table

Functions

choose_action(→ int)	Choose action using epsilon-greedy policy.
get_available_actions_env(→ list[int])	Get available actions in the current environment state.
get_best_action(→ int)	Get the action with maximum Q-value in the given state.
get_policy(→ dict[State, int])	Extract a deterministic policy from the Q-table.
get_q_value(→ float)	Get Q-value for a given state-action pair.
reset_env(→ State)	Reset the environment to initial state.
run_q_learning(→ None)	Run Q-Learning on the simple grid world environment.
step_env(→ tuple[State, float, bool])	Take a step in the environment with the given action.
update(→ None)	Perform Q-value update for a transition using the Q-learning rule.

Module Contents

machine_learning.q_learning.choose_action(state: State, available_actions: list[int]) → int

Choose action using epsilon-greedy policy.

>>> q_table.clear()
>>> old_epsilon = EPSILON
>>> EPSILON = 0.0
>>> q_table[(0, 0)][1] = 1.0
>>> q_table[(0, 0)][2] = 0.5
>>> result = choose_action((0, 0), [1, 2])
>>> EPSILON = old_epsilon  # Restore
>>> result
1

machine_learning.q_learning.get_available_actions_env() → list[int]

Get available actions in the current environment state.

machine_learning.q_learning.get_best_action(state: State, available_actions: list[int]) → int

Get the action with maximum Q-value in the given state.

>>> q_table.clear()
>>> q_table[(0, 0)][1] = 0.7
>>> q_table[(0, 0)][2] = 0.7
>>> q_table[(0, 0)][3] = 0.5
>>> get_best_action((0, 0), [1, 2, 3]) in [1, 2]
True

machine_learning.q_learning.get_policy() → dict[State, int]

Extract a deterministic policy from the Q-table.

>>> q_table.clear()
>>> q_table[(1, 2)][1] = 2.0
>>> q_table[(1, 2)][2] = 1.0
>>> get_policy()[(1, 2)]
1

machine_learning.q_learning.get_q_value(state: State, action: int) → float

Get Q-value for a given state-action pair.

>>> q_table.clear()
>>> get_q_value((0, 0), 2)
0.0

machine_learning.q_learning.reset_env() → State

Reset the environment to initial state.

>>> old_state = current_state
>>> current_state = (1, 1)  # Simulate non-initial state
>>> result = reset_env()
>>> current_state = old_state  # Restore for other tests
>>> result
(0, 0)

machine_learning.q_learning.run_q_learning() → None

Run Q-Learning on the simple grid world environment.

machine_learning.q_learning.step_env(action: int) → tuple[State, float, bool]

Take a step in the environment with the given action.

machine_learning.q_learning.update(state: State, action: int, reward: float, next_state: State, next_available_actions: list[int], done: bool = False, alpha: float | None = None, gamma: float | None = None) → None

Perform Q-value update for a transition using the Q-learning rule.

Q(s,a) <- Q(s,a) + alpha * (r + gamma * max_a’ Q(s’,a’) - Q(s,a))

>>> q_table.clear()
>>> update((0, 0), 1, 1.0, (0, 1), [1, 2], done=True, alpha=0.5, gamma=0.9)
>>> get_q_value((0, 0), 1)
0.5

machine_learning.q_learning.DISCOUNT_FACTOR = 0.97

machine_learning.q_learning.EPSILON = 0.2

machine_learning.q_learning.EPSILON_DECAY = 0.995

machine_learning.q_learning.EPSILON_MIN = 0.01

machine_learning.q_learning.GOAL

machine_learning.q_learning.LEARNING_RATE = 0.1

machine_learning.q_learning.SIZE = 4

type machine_learning.q_learning.State = tuple[int, int]

machine_learning.q_learning.current_state = (0, 0)

machine_learning.q_learning.q_table: dict[State, dict[int, float]]