machine_learning.q_learning
===========================

.. py:module:: machine_learning.q_learning

.. autoapi-nested-parse::

   Q-Learning is a widely-used model-free algorithm in reinforcement learning that
   learns the optimal action-value function Q(s, a), which tells an agent the expected
   utility of taking action a in state s and then following the optimal policy after.
   It is able to find the best policy for any given finite Markov decision process (MDP)
   without requiring a model of the environment.

   See: [https://en.wikipedia.org/wiki/Q-learning](https://en.wikipedia.org/wiki/Q-learning)



Attributes
----------

.. autoapisummary::

   machine_learning.q_learning.DISCOUNT_FACTOR
   machine_learning.q_learning.EPSILON
   machine_learning.q_learning.EPSILON_DECAY
   machine_learning.q_learning.EPSILON_MIN
   machine_learning.q_learning.GOAL
   machine_learning.q_learning.LEARNING_RATE
   machine_learning.q_learning.SIZE
   machine_learning.q_learning.State
   machine_learning.q_learning.current_state
   machine_learning.q_learning.q_table


Functions
---------

.. autoapisummary::

   machine_learning.q_learning.choose_action
   machine_learning.q_learning.get_available_actions_env
   machine_learning.q_learning.get_best_action
   machine_learning.q_learning.get_policy
   machine_learning.q_learning.get_q_value
   machine_learning.q_learning.reset_env
   machine_learning.q_learning.run_q_learning
   machine_learning.q_learning.step_env
   machine_learning.q_learning.update


Module Contents
---------------

.. py:function:: choose_action(state: State, available_actions: list[int]) -> int

   Choose action using epsilon-greedy policy.

   >>> q_table.clear()
   >>> old_epsilon = EPSILON
   >>> EPSILON = 0.0
   >>> q_table[(0, 0)][1] = 1.0
   >>> q_table[(0, 0)][2] = 0.5
   >>> result = choose_action((0, 0), [1, 2])
   >>> EPSILON = old_epsilon  # Restore
   >>> result
   1


.. py:function:: get_available_actions_env() -> list[int]

   Get available actions in the current environment state.


.. py:function:: get_best_action(state: State, available_actions: list[int]) -> int

   Get the action with maximum Q-value in the given state.

   >>> q_table.clear()
   >>> q_table[(0, 0)][1] = 0.7
   >>> q_table[(0, 0)][2] = 0.7
   >>> q_table[(0, 0)][3] = 0.5
   >>> get_best_action((0, 0), [1, 2, 3]) in [1, 2]
   True


.. py:function:: get_policy() -> dict[State, int]

   Extract a deterministic policy from the Q-table.


   >>> q_table.clear()
   >>> q_table[(1, 2)][1] = 2.0
   >>> q_table[(1, 2)][2] = 1.0
   >>> get_policy()[(1, 2)]
   1


.. py:function:: get_q_value(state: State, action: int) -> float

   Get Q-value for a given state-action pair.

   >>> q_table.clear()
   >>> get_q_value((0, 0), 2)
   0.0


.. py:function:: reset_env() -> State

   Reset the environment to initial state.

   >>> old_state = current_state
   >>> current_state = (1, 1)  # Simulate non-initial state
   >>> result = reset_env()
   >>> current_state = old_state  # Restore for other tests
   >>> result
   (0, 0)


.. py:function:: run_q_learning() -> None

   Run Q-Learning on the simple grid world environment.


.. py:function:: step_env(action: int) -> tuple[State, float, bool]

   Take a step in the environment with the given action.


.. py:function:: update(state: State, action: int, reward: float, next_state: State, next_available_actions: list[int], done: bool = False, alpha: float | None = None, gamma: float | None = None) -> None

   Perform Q-value update for a transition using the Q-learning rule.

   Q(s,a) <- Q(s,a) + alpha * (r + gamma * max_a' Q(s',a') - Q(s,a))

   >>> q_table.clear()
   >>> update((0, 0), 1, 1.0, (0, 1), [1, 2], done=True, alpha=0.5, gamma=0.9)
   >>> get_q_value((0, 0), 1)
   0.5


.. py:data:: DISCOUNT_FACTOR
   :value: 0.97


.. py:data:: EPSILON
   :value: 0.2


.. py:data:: EPSILON_DECAY
   :value: 0.995


.. py:data:: EPSILON_MIN
   :value: 0.01


.. py:data:: GOAL

.. py:data:: LEARNING_RATE
   :value: 0.1


.. py:data:: SIZE
   :value: 4


.. py:type:: State
   :canonical: tuple[int, int]


.. py:data:: current_state
   :value: (0, 0)


.. py:data:: q_table
   :type:  dict[State, dict[int, float]]