neural_network.activation_functions.swish
=========================================

.. py:module:: neural_network.activation_functions.swish

.. autoapi-nested-parse::

   This script demonstrates the implementation of the Sigmoid Linear Unit (SiLU)
   or swish function.
   * https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
   * https://en.wikipedia.org/wiki/Swish_function

   The function takes a vector x of K real numbers as input and returns x * sigmoid(x).
   Swish is a smooth, non-monotonic function defined as f(x) = x * sigmoid(x).
   Extensive experiments shows that Swish consistently matches or outperforms ReLU
   on deep networks applied to a variety of challenging domains such as
   image classification and machine translation.

   This script is inspired by a corresponding research paper.
   * https://arxiv.org/abs/1710.05941
   * https://blog.paperspace.com/swish-activation-function/



Functions
---------

.. autoapisummary::

   neural_network.activation_functions.swish.sigmoid
   neural_network.activation_functions.swish.sigmoid_linear_unit
   neural_network.activation_functions.swish.swish


Module Contents
---------------

.. py:function:: sigmoid(vector: numpy.ndarray) -> numpy.ndarray

   Mathematical function sigmoid takes a vector x of K real numbers as input and
   returns 1/ (1 + e^-x).
   https://en.wikipedia.org/wiki/Sigmoid_function

   >>> sigmoid(np.array([-1.0, 1.0, 2.0]))
   array([0.26894142, 0.73105858, 0.88079708])


.. py:function:: sigmoid_linear_unit(vector: numpy.ndarray) -> numpy.ndarray

   Implements the Sigmoid Linear Unit (SiLU) or swish function

   Parameters:
       vector (np.ndarray): A  numpy array consisting of real values

   Returns:
       swish_vec (np.ndarray): The input numpy array, after applying swish

   Examples:
   >>> sigmoid_linear_unit(np.array([-1.0, 1.0, 2.0]))
   array([-0.26894142,  0.73105858,  1.76159416])

   >>> sigmoid_linear_unit(np.array([-2]))
   array([-0.23840584])


.. py:function:: swish(vector: numpy.ndarray, trainable_parameter: int) -> numpy.ndarray

   Parameters:
       vector (np.ndarray): A  numpy array consisting of real values
       trainable_parameter: Use to implement various Swish Activation Functions

   Returns:
       swish_vec (np.ndarray): The input numpy array, after applying swish

   Examples:
   >>> swish(np.array([-1.0, 1.0, 2.0]), 2)
   array([-0.11920292,  0.88079708,  1.96402758])

   >>> swish(np.array([-2]), 1)
   array([-0.23840584])