neural_network.lstm =================== .. py:module:: neural_network.lstm .. autoapi-nested-parse:: A simple implementation of Long Short-Term Memory (LSTM) networks in Python. Classes ------- .. autoapisummary:: neural_network.lstm.LongShortTermMemory Functions --------- .. autoapisummary:: neural_network.lstm.test_with_sample_data Module Contents --------------- .. py:class:: LongShortTermMemory(input_data: str, hidden_layer_size: int = 25, training_epochs: int = 100, learning_rate: float = 0.05) .. py:method:: backward_pass(errors: list[numpy.ndarray], inputs: list[numpy.ndarray]) -> None Perform the backward pass for the LSTM model, adjusting weights and biases. :param errors: A list of errors computed from the output layer. :param inputs: A list of input one-hot encoded vectors. Example: >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> inputs = [lstm.one_hot_encode(char) for char in lstm.input_sequence] >>> predictions = lstm.forward_pass(inputs) >>> errors = [-lstm.softmax(predictions[t]) for t in range(len(predictions))] >>> for t in range(len(predictions)): ... errors[t][lstm.char_to_index[lstm.target_sequence[t]]] += 1 >>> lstm.backward_pass(errors, inputs) # Should run without any errors .. py:method:: forward_pass(inputs: list[numpy.ndarray]) -> list[numpy.ndarray] Perform a forward pass through the LSTM network for the given inputs. :param inputs: A list of input arrays (sequences). :return: A list of network outputs. Example: >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> inputs = [np.random.rand(5, 1) for _ in range(5)] >>> outputs = lstm.forward_pass(inputs) >>> len(outputs) == len(inputs) True .. py:method:: init_weights(input_dim: int, output_dim: int) -> numpy.ndarray Initialize weights with random values. :param input_dim: The input dimension. :param output_dim: The output dimension. :return: A matrix of initialized weights. Example: >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> weights = lstm.init_weights(5, 10) >>> isinstance(weights, np.ndarray) True >>> weights.shape (10, 5) .. py:method:: initialize_weights() -> None Initialize the weights and biases for the LSTM network. This method initializes the forget gate, input gate, cell candidate, and output gate weights and biases, as well as the output layer weights and biases. It ensures that the weights and biases have the correct shapes. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) # Check the shapes of the weights and biases after initialization >>> lstm.initialize_weights() # Forget gate weights and bias >>> lstm.forget_gate_weights.shape (10, 15) >>> lstm.forget_gate_bias.shape (10, 1) # Input gate weights and bias >>> lstm.input_gate_weights.shape (10, 15) >>> lstm.input_gate_bias.shape (10, 1) # Cell candidate weights and bias >>> lstm.cell_candidate_weights.shape (10, 15) >>> lstm.cell_candidate_bias.shape (10, 1) # Output gate weights and bias >>> lstm.output_gate_weights.shape (10, 15) >>> lstm.output_gate_bias.shape (10, 1) # Output layer weights and bias >>> lstm.output_layer_weights.shape (5, 10) >>> lstm.output_layer_bias.shape (5, 1) .. py:method:: one_hot_encode(char: str) -> numpy.ndarray One-hot encode a character. :param char: The character to encode. :return: A one-hot encoded vector. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> output = lstm.one_hot_encode('a') >>> isinstance(output, np.ndarray) True >>> output.shape (5, 1) >>> output = lstm.one_hot_encode('c') >>> isinstance(output, np.ndarray) True >>> output.shape (5, 1) .. py:method:: reset_network_state() -> None Reset the LSTM network states. Resets the internal states of the LSTM network, including the combined inputs, hidden states, cell states, gate activations, and network outputs. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> lstm.reset_network_state() >>> lstm.hidden_states[-1].shape == (10, 1) True >>> lstm.cell_states[-1].shape == (10, 1) True >>> lstm.combined_inputs == {} True >>> lstm.network_outputs == {} True .. py:method:: sigmoid(input_array: numpy.ndarray, derivative: bool = False) -> numpy.ndarray Sigmoid activation function. :param input_array: The input array. :param derivative: Whether to compute the derivative. :return: The sigmoid activation or its derivative. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> output = lstm.sigmoid(input_array=np.array([[1, 2, 3]])) >>> isinstance(output, np.ndarray) True >>> np.round(output, 3) array([[0.731, 0.881, 0.953]]) >>> derivative_output = lstm.sigmoid(input_array=output, derivative=True) >>> np.round(derivative_output, 3) array([[0.197, 0.105, 0.045]]) .. py:method:: softmax(input_array: numpy.ndarray) -> numpy.ndarray Softmax activation function. :param input_array: The input array. :return: The softmax activation. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> output = lstm.softmax(input_array=np.array([1, 2, 3])) >>> isinstance(output, np.ndarray) True >>> np.round(output, 3) array([0.09 , 0.245, 0.665]) .. py:method:: tanh(input_array: numpy.ndarray, derivative: bool = False) -> numpy.ndarray Tanh activation function. :param input_array: The input array. :param derivative: Whether to compute the derivative. :return: The tanh activation or its derivative. >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> output = lstm.tanh(input_array=np.array([[1, 2, 3]])) >>> isinstance(output, np.ndarray) True >>> np.round(output, 3) array([[0.762, 0.964, 0.995]]) >>> derivative_output = lstm.tanh(input_array=output, derivative=True) >>> np.round(derivative_output, 3) array([[0.42 , 0.071, 0.01 ]]) .. py:method:: test() -> str Test the LSTM model. Returns: str: The output predictions. Example: >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> output = lstm.test() >>> isinstance(output, str) True >>> len(output) == len(lstm.input_sequence) True .. py:method:: train() -> None Train the LSTM model. Example: >>> lstm = LongShortTermMemory("abcde" * 50, hidden_layer_size=10) >>> lstm.train() .. py:attribute:: cell_state_candidates :type: dict[int, numpy.ndarray] .. py:attribute:: cell_states :type: dict[int, numpy.ndarray] .. py:attribute:: char_to_index :type: dict[str, int] .. py:attribute:: combined_inputs :type: dict[int, numpy.ndarray] .. py:attribute:: data_length :type: int .. py:attribute:: forget_gate_activations :type: dict[int, numpy.ndarray] .. py:attribute:: hidden_layer_size :type: int :value: 25 .. py:attribute:: hidden_states :type: dict[int, numpy.ndarray] .. py:attribute:: index_to_char :type: dict[int, str] .. py:attribute:: input_data :type: str .. py:attribute:: input_gate_activations :type: dict[int, numpy.ndarray] .. py:attribute:: input_sequence :type: str .. py:attribute:: learning_rate :type: float :value: 0.05 .. py:attribute:: network_outputs :type: dict[int, numpy.ndarray] .. py:attribute:: output_gate_activations :type: dict[int, numpy.ndarray] .. py:attribute:: random_generator :type: numpy.random.Generator .. py:attribute:: target_sequence :type: str .. py:attribute:: training_epochs :type: int :value: 100 .. py:attribute:: unique_chars :type: set .. py:attribute:: vocabulary_size :type: int .. py:function:: test_with_sample_data() -> None >>> test_with_sample_data()