maths.softmax

This script demonstrates the implementation of the Softmax function.

It takes as input a vector of K real numbers and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. After applying softmax, the elements of the vector always sum up to 1.

Script inspired by its corresponding Wikipedia article: https://en.wikipedia.org/wiki/Softmax_function

Attributes

mat

Functions

softmax(→ numpy.ndarray)

Compute the softmax of vector along axis in a numerically-stable way.

Module Contents

maths.softmax.softmax(vector: numpy.ndarray, axis: int | None = -1) → numpy.ndarray

Compute the softmax of vector along axis in a numerically-stable way.

Parameters:

vector (np.ndarray | list | tuple): Input data (vector, matrix or

higher-rank tensor). It is converted to a float np.ndarray, so lists, tuples and integers are accepted too.

axis (int | None, optional): Axis along which softmax is computed so

that the probabilities sum to 1 along that axis. If None, the softmax is computed over the flattened array (a single distribution). Default is -1 (the last axis).

Returns:

np.ndarray: An array with the same shape as vector whose values along axis (or over the whole array when axis is None) form a probability distribution that sums to 1.

Raises:

ValueError: If vector is empty or cannot be converted to a numeric

float array (for example a string or a dict).

numpy.exceptions.AxisError: If axis is out of bounds for the input.

Note:

If the input contains NaN or inf the result will contain NaN along the affected axis; softmax is only meaningful for finite real inputs.

The softmax vector adds up to one. We need to ceil to mitigate precision.

>>> float(np.ceil(np.sum(softmax([1, 2, 3, 4]))))
1.0

Identical logits map to a uniform distribution:

>>> softmax(np.array([5, 5]))
array([0.5, 0.5])

A single element always maps to 1:

>>> softmax([0])
array([1.])

It is numerically stable for large logits (no overflow):

>>> softmax([1000.0, 1001.0, 1002.0])
array([0.09003057, 0.24472847, 0.66524096])

For a 2-D array the axis selects where probabilities sum to 1:

>>> mat = np.array([[1.0, 2.0, 3.0], [1.0, 2.0, 3.0]])
>>> np.round(softmax(mat, axis=-1), 3)
array([[0.09 , 0.245, 0.665],
       [0.09 , 0.245, 0.665]])
>>> np.round(softmax(mat, axis=0), 3)
array([[0.5, 0.5, 0.5],
       [0.5, 0.5, 0.5]])

With axis=None the whole array becomes one distribution that sums to 1:

>>> float(np.round(np.sum(softmax(mat, axis=None)), 6))
1.0

Empty, non-numeric and out-of-bounds inputs raise clear errors:

>>> softmax([])
Traceback (most recent call last):
    ...
ValueError: softmax input must be non-empty
>>> softmax("not a number")
Traceback (most recent call last):
    ...
ValueError: softmax input must be numeric, got str
>>> softmax([1, 2, 3], axis=3)
Traceback (most recent call last):
    ...
numpy.exceptions.AxisError: axis 3 is out of bounds for array of dimension 1

maths.softmax.mat