neural_network.lstm

A simple implementation of Long Short-Term Memory (LSTM) networks in Python.

Classes

LongShortTermMemory

Functions

test_with_sample_data(→ None)

Module Contents

class neural_network.lstm.LongShortTermMemory(input_data: str, hidden_layer_size: int = 25, training_epochs: int = 100, learning_rate: float = 0.05)
backward_pass(errors: list[numpy.ndarray], inputs: list[numpy.ndarray]) → None

Perform the backward pass for the LSTM model, adjusting weights and biases.

Parameters:

  • errors – A list of errors computed from the output layer.

  • inputs – A list of input one-hot encoded vectors.

Example: >>> lstm = LongShortTermMemory(“abcde” * 50, hidden_layer_size=10) >>> inputs = [lstm.one_hot_encode(char) for char in lstm.input_sequence] >>> predictions = lstm.forward_pass(inputs) >>> errors = [-lstm.softmax(predictions[t]) for t in range(len(predictions))] >>> for t in range(len(predictions)): … errors[t][lstm.char_to_index[lstm.target_sequence[t]]] += 1 >>> lstm.backward_pass(errors, inputs) # Should run without any errors

forward_pass(inputs: list[numpy.ndarray]) → list[numpy.ndarray]

Perform a forward pass through the LSTM network for the given inputs.

Parameters:

inputs – A list of input arrays (sequences).

Returns:

A list of network outputs.

Example: >>> lstm = LongShortTermMemory(“abcde” * 50, hidden_layer_size=10) >>> inputs = [np.random.rand(5, 1) for _ in range(5)] >>> outputs = lstm.forward_pass(inputs) >>> len(outputs) == len(inputs) True

init_weights(input_dim: int, output_dim: int) → numpy.ndarray

Initialize weights with random values.

Parameters:

  • input_dim – The input dimension.

  • output_dim – The output dimension.

Returns:

A matrix of initialized weights.

Example: >>> lstm = LongShortTermMemory(“abcde” * 50, hidden_layer_size=10) >>> weights = lstm.init_weights(5, 10) >>> isinstance(weights, np.ndarray) True >>> weights.shape (10, 5)

initialize_weights() → None

Initialize the weights and biases for the LSTM network.

This method initializes the forget gate, input gate, cell candidate, and output gate weights and biases, as well as the output layer weights and biases. It ensures that the weights and biases have the correct shapes.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)

# Check the shapes of the weights and biases after initialization >>> lstm.initialize_weights()

# Forget gate weights and bias >>> lstm.forget_gate_weights.shape (10, 15) >>> lstm.forget_gate_bias.shape (10, 1)

# Input gate weights and bias >>> lstm.input_gate_weights.shape (10, 15) >>> lstm.input_gate_bias.shape (10, 1)

# Cell candidate weights and bias >>> lstm.cell_candidate_weights.shape (10, 15) >>> lstm.cell_candidate_bias.shape (10, 1)

# Output gate weights and bias >>> lstm.output_gate_weights.shape (10, 15) >>> lstm.output_gate_bias.shape (10, 1)

# Output layer weights and bias >>> lstm.output_layer_weights.shape (5, 10) >>> lstm.output_layer_bias.shape (5, 1)

one_hot_encode(char: str) → numpy.ndarray

One-hot encode a character.

Parameters:

char – The character to encode.

Returns:

A one-hot encoded vector.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)
>>> output = lstm.one_hot_encode('a')
>>> isinstance(output, np.ndarray)
True
>>> output.shape
(5, 1)
>>> output = lstm.one_hot_encode('c')
>>> isinstance(output, np.ndarray)
True
>>> output.shape
(5, 1)
reset_network_state() → None

Reset the LSTM network states.

Resets the internal states of the LSTM network, including the combined inputs, hidden states, cell states, gate activations, and network outputs.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)
>>> lstm.reset_network_state()
>>> lstm.hidden_states[-1].shape == (10, 1)
True
>>> lstm.cell_states[-1].shape == (10, 1)
True
>>> lstm.combined_inputs == {}
True
>>> lstm.network_outputs == {}
True
sigmoid(input_array: numpy.ndarray, derivative: bool = False) → numpy.ndarray

Sigmoid activation function.

Parameters:

  • input_array – The input array.

  • derivative – Whether to compute the derivative.

Returns:

The sigmoid activation or its derivative.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)
>>> output = lstm.sigmoid(input_array=np.array([[1, 2, 3]]))
>>> isinstance(output, np.ndarray)
True
>>> np.round(output, 3)
array([[0.731, 0.881, 0.953]])
>>> derivative_output = lstm.sigmoid(input_array=output, derivative=True)
>>> np.round(derivative_output, 3)
array([[0.197, 0.105, 0.045]])
softmax(input_array: numpy.ndarray) → numpy.ndarray

Softmax activation function.

Parameters:

input_array – The input array.

Returns:

The softmax activation.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)
>>> output = lstm.softmax(input_array=np.array([1, 2, 3]))
>>> isinstance(output, np.ndarray)
True
>>> np.round(output, 3)
array([0.09 , 0.245, 0.665])
tanh(input_array: numpy.ndarray, derivative: bool = False) → numpy.ndarray

Tanh activation function.

Parameters:

  • input_array – The input array.

  • derivative – Whether to compute the derivative.

Returns:

The tanh activation or its derivative.

>>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10)
>>> output = lstm.tanh(input_array=np.array([[1, 2, 3]]))
>>> isinstance(output, np.ndarray)
True
>>> np.round(output, 3)
array([[0.762, 0.964, 0.995]])
>>> derivative_output = lstm.tanh(input_array=output, derivative=True)
>>> np.round(derivative_output, 3)
array([[0.42 , 0.071, 0.01 ]])
test() → str

Test the LSTM model.

Returns:

str: The output predictions.

Example: >>> lstm = LongShortTermMemory(“abcde” * 50, hidden_layer_size=10) >>> output = lstm.test() >>> isinstance(output, str) True >>> len(output) == len(lstm.input_sequence) True

train() → None

Train the LSTM model.

Example: >>> lstm = LongShortTermMemory(“abcde” * 50, hidden_layer_size=10) >>> lstm.train()

cell_state_candidates: dict[int, numpy.ndarray]
cell_states: dict[int, numpy.ndarray]
char_to_index: dict[str, int]
combined_inputs: dict[int, numpy.ndarray]
data_length: int
forget_gate_activations: dict[int, numpy.ndarray]
hidden_layer_size: int = 25
hidden_states: dict[int, numpy.ndarray]
index_to_char: dict[int, str]
input_data: str
input_gate_activations: dict[int, numpy.ndarray]
input_sequence: str
learning_rate: float = 0.05
network_outputs: dict[int, numpy.ndarray]
output_gate_activations: dict[int, numpy.ndarray]
random_generator: numpy.random.Generator
target_sequence: str
training_epochs: int = 100
unique_chars: set
vocabulary_size: int
neural_network.lstm.test_with_sample_data() → None
>>> test_with_sample_data()