athena__initialiser_lecun Module

Module containing the implementation of the LeCun initialiser

This module implements LeCun initialisation, the precursor to modern initialisation schemes, designed for efficient backpropagation.

Mathematical operation: Uniform variant: W ~ U(-limit, limit) where limit = sqrt(3 / fan_in) Normal variant: W ~ N(0, σ²) where σ = sqrt(1 / fan_in)

fan_in is the number of input units to the layer.

Rationale: Maintains variance of inputs through layers Helps prevent saturation of activation functions

Best for: SELU activation (with lecun_normal variant) Also works with: Tanh, Sigmoid Reference: LeCun et al. (1998), Neural Networks: Tricks of the Trade


Uses


Interfaces

public interface lecun_normal_init_type

  • private module function initialiser_lecun_normal_setup() result(initialiser)

    Interface for the LeCun normal initialiser

    Arguments

    None

    Return Value type(lecun_normal_init_type)

    LeCun normal initialiser object

public interface lecun_uniform_init_type

  • private module function initialiser_lecun_uniform_setup() result(initialiser)

    Interface for the LeCun uniform initialiser

    Arguments

    None

    Return Value type(lecun_uniform_init_type)

    LeCun uniform initialiser object


Derived Types

type, public, extends(base_init_type) ::  lecun_normal_init_type

Type for the LeCun initialiser (normal)

Components

Type Visibility Attributes Name Initial
real(kind=real32), public :: mean = 1._real32

Scale, mean, and standard deviation of the initialiser
character(len=20), public :: name

Name of the initialiser
real(kind=real32), public :: scale = 1._real32

Scale, mean, and standard deviation of the initialiser
real(kind=real32), public :: std = 0.01_real32

Scale, mean, and standard deviation of the initialiser

Constructor

private module function initialiser_lecun_normal_setup ()

Interface for the LeCun normal initialiser

Type-Bound Procedures

procedure, public, pass(this) :: initialise => lecun_normal_initialise

Initialise the weights and biases using the LeCun normal distribution

type, public, extends(base_init_type) ::  lecun_uniform_init_type

Type for the LeCun initialiser (uniform)

Components

Type Visibility Attributes Name Initial
real(kind=real32), public :: mean = 1._real32

Scale, mean, and standard deviation of the initialiser
character(len=20), public :: name

Name of the initialiser
real(kind=real32), public :: scale = 1._real32

Scale, mean, and standard deviation of the initialiser
real(kind=real32), public :: std = 0.01_real32

Scale, mean, and standard deviation of the initialiser

Constructor

private module function initialiser_lecun_uniform_setup ()

Interface for the LeCun uniform initialiser

Type-Bound Procedures

procedure, public, pass(this) :: initialise => lecun_uniform_initialise

Initialise the weights and biases using the LeCun uniform distribution

Functions

private module function initialiser_lecun_normal_setup() result(initialiser)

Interface for the LeCun normal initialiser

Arguments

None

Return Value type(lecun_normal_init_type)

LeCun normal initialiser object

private module function initialiser_lecun_uniform_setup() result(initialiser)

Interface for the LeCun uniform initialiser

Arguments

None

Return Value type(lecun_uniform_init_type)

LeCun uniform initialiser object


Subroutines

private subroutine lecun_normal_initialise(this, input, fan_in, fan_out, spacing)

Initialise the weights and biases using the LeCun normal distribution

Arguments

Type IntentOptional Attributes Name
class(lecun_normal_init_type), intent(inout) :: this

Instance of the LeCun initialiser
real(kind=real32), intent(out), dimension(..) :: input

Weights and biases to initialise
integer, intent(in), optional :: fan_in

Number of input and output parameters
integer, intent(in), optional :: fan_out

Number of input and output parameters
integer, intent(in), optional, dimension(:) :: spacing

Spacing of the input and output units (not used)

private subroutine lecun_uniform_initialise(this, input, fan_in, fan_out, spacing)

Initialise the weights and biases using the LeCun uniform distribution

Arguments

Type IntentOptional Attributes Name
class(lecun_uniform_init_type), intent(inout) :: this

Instance of the Glorot initialiser
real(kind=real32), intent(out), dimension(..) :: input

Weights and biases to initialise
integer, intent(in), optional :: fan_in

Number of input and output units
integer, intent(in), optional :: fan_out

Number of input and output units
integer, intent(in), optional, dimension(:) :: spacing

Spacing of the input and output units (not used)