instanceNormalizationLayer

Instance normalization layer

Since R2021a

expand all in page

Description

An instance normalization layer normalizes a mini-batch of data across each channel for each observation independently. To improve the convergence of training the convolutional neural network and reduce the sensitivity to network hyperparameters, use instance normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

After normalization, the layer scales the input with a learnable scale factor γ and shifts it by a learnable offset β.

Creation

Syntax

layer = instanceNormalizationLayer
layer = instanceNormalizationLayer(Name,Value)

Description

layer = instanceNormalizationLayer creates an instance normalization layer.

example

layer = instanceNormalizationLayer(Name,Value) creates an instance normalization layer and sets the optional Epsilon, Parameters and Initialization, Learning Rate and Regularization, and Name properties using one or more name-value arguments. You can specify multiple name-value arguments. Enclose each property name in quotes.

Example: instanceNormalizationLayer('Name','instancenorm') creates an instance normalization layer with the name 'instancenorm'

example

Properties

expand all

Instance Normalization

Constant to add to the mini-batch variances, specified as a positive scalar.

The software adds this constant to the mini-batch variances before normalization to ensure numerical stability and avoid division by zero.

Before R2023a: Epsilon must be greater than or equal to 1e-5.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

This property is read-only.

Number of input channels, specified as one of the following:

  • "auto" — Automatically determine the number of input channels at training time.

  • Positive integer — Configure the layer for the specified number of input channels. NumChannels and the number of channels in the layer input data must match. For example, if the input is an RGB image, then NumChannels must be 3. If the input is the output of a convolutional layer with 16 filters, then NumChannels must be 16.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

Parameters and Initialization

Function to initialize the channel scale factors, specified as one of the following:

  • 'ones' – Initialize the channel scale factors with ones.

  • 'zeros' – Initialize the channel scale factors with zeros.

  • 'narrow-normal' – Initialize the channel scale factors by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.

  • Function handle – Initialize the channel scale factors with a custom function. If you specify a function handle, then the function must be of the form scale = func(sz), where sz is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel scale factors when the Scale property is empty.

Data Types: char | string | function_handle

Function to initialize the channel offsets, specified as one of the following:

  • 'zeros' – Initialize the channel offsets with zeros.

  • 'ones' – Initialize the channel offsets with ones.

  • 'narrow-normal' – Initialize the channel offsets by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.

  • Function handle – Initialize the channel offsets with a custom function. If you specify a function handle, then the function must be of the form offset = func(sz), where sz is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel offsets when the Offset property is empty.

Data Types: char | string | function_handle

Channel scale factors γ, specified as a numeric array.

The channel scale factors are learnable parameters. When you train a network using the trainnet function or initialize a dlnetwork object, if Scale is nonempty, then the software uses the Scale property as the initial value. If Scale is empty, then the software uses the initializer specified by ScaleInitializer.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer InputProperty Size
feature inputNumChannels-by-1
vector sequence input

1-D image input (since R2023a)

1-by-NumChannels

1-D image sequence input (since R2023a)

2-D image input1-by-1-by-NumChannels
2-D image sequence input
3-D image input1-by-1-by-1-by-NumChannels
3-D image sequence input

Data Types: single | double

Channel offsets β, specified as a numeric vector.

The channel offsets are learnable parameters. When you train a network using the trainnet function or initialize a dlnetwork object, if Offset is nonempty, then the software uses the Offset property as the initial value. If Offset is empty, then the software uses the initializer specified by OffsetInitializer.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer InputProperty Size
feature inputNumChannels-by-1
vector sequence input

1-D image input (since R2023a)

1-by-NumChannels

1-D image sequence input (since R2023a)

2-D image input1-by-1-by-NumChannels
2-D image sequence input
3-D image input1-by-1-by-1-by-NumChannels
3-D image sequence input

Data Types: single | double

Learning Rate and Regularization

Learning rate factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the scale factors in a layer. For example, if ScaleLearnRateFactor is 2, then the learning rate for the scale factors in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Learning rate factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the offsets in a layer. For example, if OffsetLearnRateFactor is 2, then the learning rate for the offsets in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

L2 regularization factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the scale factors in a layer. For example, if ScaleL2Factor is 2, then the L2 regularization for the offsets in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

L2 regularization factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the offsets in a layer. For example, if OffsetL2Factor is 2, then the L2 regularization for the offsets in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Layer

Layer name, specified as a character vector or a string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to unnamed layers.

The InstanceNormalizationLayer object stores this property as a character vector.

Data Types: char | string

This property is read-only.

Number of inputs to the layer, stored as 1. This layer accepts a single input only.

Data Types: double

This property is read-only.

Input names, stored as {'in'}. This layer accepts a single input only.

Data Types: cell

This property is read-only.

Number of outputs from the layer, stored as 1. This layer has a single output only.

Data Types: double

This property is read-only.

Output names, stored as {'out'}. This layer has a single output only.

Data Types: cell

Examples

collapse all

Open Live Script

Create an instance normalization layer with the name 'instancenorm'.

layer = instanceNormalizationLayer('Name','instancenorm')
layer = 
  InstanceNormalizationLayer with properties:

           Name: 'instancenorm'
    NumChannels: 'auto'

   Hyperparameters
        Epsilon: 1.0000e-05

   Learnable Parameters
         Offset: []
          Scale: []

  Show all properties

Include an instance normalization layer in a Layer array.

layers = [
    imageInputLayer([28 28 3])
    convolution2dLayer(5,20)
    instanceNormalizationLayer
    reluLayer
    maxPooling2dLayer(2,'Stride',2)
    fullyConnectedLayer(10)
    softmaxLayer]
layers = 
  7×1 Layer array with layers:

     1   ''   Image Input              28×28×3 images with 'zerocenter' normalization
     2   ''   2-D Convolution          20 5×5 convolutions with stride [1  1] and padding [0  0  0  0]
     3   ''   Instance Normalization   Instance normalization
     4   ''   ReLU                     ReLU
     5   ''   2-D Max Pooling          2×2 max pooling with stride [2  2] and padding [0  0  0  0]
     6   ''   Fully Connected          10 fully connected layer
     7   ''   Softmax                  softmax

Algorithms

expand all

Extended Capabilities

expand all

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Version History

Introduced in R2021a

expand all

See Also

| | | | | | | |

Topics