neural_network.sliding_window_attention
=======================================

.. py:module:: neural_network.sliding_window_attention

.. autoapi-nested-parse::

    - - - - - -- - - - - - - - - - - - - - - - - - - - - - -
   Name - - sliding_window_attention.py
   Goal - - Implement a neural network architecture using sliding
           window attention for sequence modeling tasks.
   Detail: Total 5 layers neural network
           * Input layer
           * Sliding Window Attention Layer
           * Feedforward Layer
           * Output Layer
   Author: Stephen Lee
   Github: 245885195@qq.com
   Date: 2024.10.20
   References:
       1. Choromanska, A., et al. (2020). "On the Importance of
          Initialization and Momentum in Deep Learning." *Proceedings
          of the 37th International Conference on Machine Learning*.
       2. Dai, Z., et al. (2020). "Transformers are RNNs: Fast
          Autoregressive Transformers with Linear Attention."
          *arXiv preprint arXiv:2006.16236*.
       3. [Attention Mechanisms in Neural Networks](https://en.wikipedia.org/wiki/Attention_(machine_learning))
    - - - - - -- - - - - - - - - - - - - - - - - - - - - - -



Attributes
----------

.. autoapisummary::

   neural_network.sliding_window_attention.rng


Classes
-------

.. autoapisummary::

   neural_network.sliding_window_attention.SlidingWindowAttention


Module Contents
---------------

.. py:class:: SlidingWindowAttention(embed_dim: int, window_size: int)

   Sliding Window Attention Module.

   This class implements a sliding window attention mechanism where
   the model attends to a fixed-size window of context around each token.

   Attributes:
       window_size (int): The size of the attention window.
       embed_dim (int): The dimensionality of the input embeddings.


   .. py:method:: forward(input_tensor: numpy.ndarray) -> numpy.ndarray

      Forward pass for the sliding window attention.

      Args:
          input_tensor (np.ndarray): Input tensor of shape (batch_size,
                                     seq_length, embed_dim).

      Returns:
          np.ndarray: Output tensor of shape (batch_size, seq_length, embed_dim).

      >>> x = np.random.randn(2, 10, 4)  # Batch size 2, sequence
      >>> attention = SlidingWindowAttention(embed_dim=4, window_size=3)
      >>> output = attention.forward(x)
      >>> output.shape
      (2, 10, 4)
      >>> (output.sum() != 0).item()  # Check if output is non-zero
      True



   .. py:attribute:: attention_weights


   .. py:attribute:: embed_dim


   .. py:attribute:: window_size


.. py:data:: rng