athena__initialiser_glorot Module

Module containing the implementation of the Glorot initialiser

This module implements Glorot (Xavier) initialisation, designed to maintain variance of gradients through layers with sigmoid/tanh.

Mathematical operation:

Uniform variant:

Normal variant:

where is fan-in, is fan-out.

Rationale: Maintains variance across layers, prevents vanishing/exploding gradients in deep networks

Best for: Tanh, Sigmoid, Softmax activations Reference: Glorot & Bengio (2010), AISTATS


Uses


Interfaces

public interface glorot_normal_init_type

  • private module function initialiser_normal_setup() result(initialiser)

    Interface for the Glorot normal initialiser

    Arguments

    None

    Return Value type(glorot_normal_init_type)

    Glorot normal initialiser object

public interface glorot_uniform_init_type

  • private module function initialiser_uniform_setup() result(initialiser)

    Interface for the Glorot uniform initialiser

    Arguments

    None

    Return Value type(glorot_uniform_init_type)

    Glorot uniform initialiser object


Derived Types

type, public, extends(base_init_type) ::  glorot_normal_init_type

Type for the Glorot initialiser (normal)

Components

Type Visibility Attributes Name Initial
real(kind=real32), public :: mean = 1._real32

Scale, mean, and standard deviation of the initialiser
character(len=20), public :: name

Name of the initialiser
real(kind=real32), public :: scale = 1._real32

Scale, mean, and standard deviation of the initialiser
real(kind=real32), public :: std = 0.01_real32

Scale, mean, and standard deviation of the initialiser

Constructor

private module function initialiser_normal_setup ()

Interface for the Glorot normal initialiser

Type-Bound Procedures

procedure, public, pass(this) :: initialise => glorot_normal_initialise

Initialise the weights and biases using the Glorot normal distribution

type, public, extends(base_init_type) ::  glorot_uniform_init_type

Type for the Glorot initialiser (uniform)

Components

Type Visibility Attributes Name Initial
real(kind=real32), public :: mean = 1._real32

Scale, mean, and standard deviation of the initialiser
character(len=20), public :: name

Name of the initialiser
real(kind=real32), public :: scale = 1._real32

Scale, mean, and standard deviation of the initialiser
real(kind=real32), public :: std = 0.01_real32

Scale, mean, and standard deviation of the initialiser

Constructor

private module function initialiser_uniform_setup ()

Interface for the Glorot uniform initialiser

Type-Bound Procedures

procedure, public, pass(this) :: initialise => glorot_uniform_initialise

Initialise the weights and biases using the Glorot uniform distribution

Functions

private module function initialiser_normal_setup() result(initialiser)

Arguments

None

Return Value type(glorot_normal_init_type)

Glorot normal initialiser object

private module function initialiser_uniform_setup() result(initialiser)

Arguments

None

Return Value type(glorot_uniform_init_type)

Glorot uniform initialiser object


Subroutines

private subroutine glorot_normal_initialise(this, input, fan_in, fan_out, spacing)

Initialise the weights and biases using the Glorot normal distribution

Arguments

Type IntentOptional Attributes Name
class(glorot_normal_init_type), intent(inout) :: this

Instance of the Glorot initialiser
real(kind=real32), intent(out), dimension(..) :: input

Weights to initialise
integer, intent(in), optional :: fan_in

Number of input and output units
integer, intent(in), optional :: fan_out

Number of input and output units
integer, intent(in), optional, dimension(:) :: spacing

Spacing of the input and output units (not used here, included for compatibility)

private subroutine glorot_uniform_initialise(this, input, fan_in, fan_out, spacing)

Initialise the weights and biases using the Glorot uniform distribution

Arguments

Type IntentOptional Attributes Name
class(glorot_uniform_init_type), intent(inout) :: this

Instance of the Glorot initialiser
real(kind=real32), intent(out), dimension(..) :: input

Weights and biases to initialise
integer, intent(in), optional :: fan_in

Number of input and output units
integer, intent(in), optional :: fan_out

Number of input and output units
integer, intent(in), optional, dimension(:) :: spacing

Spacing of the input and output units (not used)