athena__activation_swish Module

Module containing implementation of the swish activation function

This module implements Swish (also called SiLU), a smooth, non-monotonic activation function discovered by Google researchers.

Mathematical operation:

where is a parameter (typically , making it SiLU)

Derivative:

Properties: Smooth, self-gated, unbounded above, bounded below at 0 Often outperforms ReLU in deep networks Reference: Ramachandran et al. (2017), arXiv:1710.05941


Uses


Interfaces

public interface swish_actv_type

Interface for setting up swish activation function

  • private function initialise(scale, beta, attributes) result(activation)

    Initialise a swish activation function

    Arguments

    Type IntentOptional Attributes Name
    real(kind=real32), intent(in), optional :: scale

    Optional scale factor for activation output
    real(kind=real32), intent(in), optional :: beta

    Optional beta parameter for swish function
    type(onnx_attribute_type), intent(in), optional, dimension(:) :: attributes

    Optional array of ONNX attributes

    Return Value type(swish_actv_type)

    Swish activation type


Derived Types

type, public, extends(base_actv_type) ::  swish_actv_type

Type for swish activation function with overloaded procedures

Components

Type Visibility Attributes Name Initial
logical, public :: apply_scaling = .false.

Boolean to apply scaling or not
real(kind=real32), public :: beta = 1._real32

Beta parameter for swish function
character(len=10), public :: name

Name of the activation function
real(kind=real32), public :: scale = 1._real32

Scale of the activation function
real(kind=real32), public :: threshold

Threshold of the activation function

Constructor

Interface for setting up swish activation function

private function initialise (scale, beta, attributes)

Initialise a swish activation function

Type-Bound Procedures

procedure, public, pass(this) :: apply => apply_swish
procedure, public, pass(this) :: apply_attributes => apply_attributes_swish
procedure, public, pass(this) :: export_attributes => export_attributes_swish
procedure, public, pass(this) :: print_to_unit => print_to_unit_actv
procedure, public, pass(this) :: reset => reset_swish

Functions

public function create_from_onnx_swish_activation(attributes) result(activation)

Create swish activation function from ONNX attributes

Arguments

Type IntentOptional Attributes Name
type(onnx_attribute_type), intent(in), dimension(:) :: attributes

Array of ONNX attributes

Return Value class(base_actv_type), allocatable

Instance of activation type

private function apply_swish(this, val) result(output)

Apply swish activation to 1D array

Read more…

Arguments

Type IntentOptional Attributes Name
class(swish_actv_type), intent(in) :: this

Swish activation type
type(array_type), intent(in) :: val

Input values

Return Value type(array_type), pointer

Swish activation output

private pure function export_attributes_swish(this) result(attributes)

Export swish activation function attributes as ONNX attributes

Arguments

Type IntentOptional Attributes Name
class(swish_actv_type), intent(in) :: this

Swish activation type

Return Value type(onnx_attribute_type), allocatable, dimension(:)

Array of ONNX attributes

private function initialise(scale, beta, attributes) result(activation)

Initialise a swish activation function

Arguments

Type IntentOptional Attributes Name
real(kind=real32), intent(in), optional :: scale

Optional scale factor for activation output
real(kind=real32), intent(in), optional :: beta

Optional beta parameter for swish function
type(onnx_attribute_type), intent(in), optional, dimension(:) :: attributes

Optional array of ONNX attributes

Return Value type(swish_actv_type)

Swish activation type


Subroutines

private subroutine apply_attributes_swish(this, attributes)

Load ONNX attributes into swish activation function

Arguments

Type IntentOptional Attributes Name
class(swish_actv_type), intent(inout) :: this

Swish activation type
type(onnx_attribute_type), intent(in), dimension(:) :: attributes

Array of ONNX attributes

private pure subroutine reset_swish(this)

Reset swish activation function attributes and variables

Arguments

Type IntentOptional Attributes Name
class(swish_actv_type), intent(inout) :: this

Swish activation type