machine_learning.rmsprop

RMSprop (Root Mean Square Propagation) optimizer implementation.

RMSprop is an adaptive learning rate optimizer that maintains a moving average of squared gradients to normalize the gradient. It was proposed by Geoffrey Hinton in his Coursera course on Neural Networks.

Key idea: Instead of using a fixed learning rate, RMSprop adapts the learning rate for each parameter by dividing by a running average of recent gradient magnitudes.

Update rules:

v(t) = rho * v(t-1) + (1 - rho) * gradient^2 param = param - (learning_rate / sqrt(v(t) + epsilon)) * gradient

Where:

v(t) = moving average of squared gradients rho = decay factor (typically 0.9) learning_rate = step size epsilon = small value to avoid division by zero

Reference: https://en.wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp

>>> rmsprop([0.0], [0.1], 0.01)
[-0.031...]
>>> rmsprop([1.0, -1.0], [0.5, -0.5], 0.01)
[0.968..., -0.968...]

Attributes

param

Functions

rmsprop(→ list[float])

Perform one step of the RMSprop optimization algorithm.

Module Contents

machine_learning.rmsprop.rmsprop(params: list[float], gradients: list[float], learning_rate: float, rho: float = 0.9, epsilon: float = 1e-08, moving_avg: list[float] | None = None) → list[float]

Perform one step of the RMSprop optimization algorithm.

Parameters:

  • params – Current parameter values to be updated.

  • gradients – Gradients of the loss with respect to each parameter.

  • learning_rate – Step size for the update (must be positive).

  • rho – Decay factor for the moving average (default 0.9).

  • epsilon – Small constant to avoid division by zero (default 1e-8).

  • moving_avg – Running average of squared gradients. Updated in place each call. Initialized to zeros if not provided.

Returns:

Updated parameter values after one RMSprop step.

Raises:

  • ValueError – If params and gradients have different lengths.

  • ValueError – If learning_rate, rho, or epsilon are out of range.

>>> rmsprop([0.0], [0.0], 0.01)
[0.0]
>>> rmsprop([1.0], [0.0], 0.01)
[1.0]
>>> len(rmsprop([1.0, 2.0, 3.0], [0.1, 0.2, 0.3], 0.01)) == 3
True
>>> rmsprop([1.0], [0.5], learning_rate=0.01)
[0.968...]
machine_learning.rmsprop.param = [5.0]