Check Custom Layer Validity

If you create a custom deep learning layer, then you can use the checkLayer function to check that the layer is valid. The function checks layers for validity, GPU compatibility, and correctly defined gradients. To check that a layer is valid, run the following command:

checkLayer(layer,validInputSize,'ObservationDimension',dim)
where layer is an instance of the layer, validInputSize is a vector or cell array specifying the valid input sizes to the layer, and dim specifies the dimension of the observations in the layer input data. For large input sizes, the gradient checks take longer to run. To speed up the tests, specify a smaller valid input size.

Check Layer Validity

Check the validity of the example custom layer preluLayer.

Define a custom PReLU layer. To create this layer, save the file preluLayer.m in the current folder.

Create an instance of the layer and check that it is valid using checkLayer. Set the valid input size to the typical size of a single observation input to the layer. For a single input, the layer expects observations of size h-by-w-by-c, where h, w, and c are the height, width, and number of channels of the previous layer output, respectively.

Specify validInputSize as the typical size of an input array.

layer = preluLayer(20,'prelu');
validInputSize = [5 5 20];
checkLayer(layer,validInputSize)
Skipping multi-observation tests. To enable tests with multiple observations, specify the 'ObservationDimension' option in checkLayer.
For 2-D image data, set 'ObservationDimension' to 4.
For 3-D image data, set 'ObservationDimension' to 5.
For sequence data, set 'ObservationDimension' to 2.
 
Skipping GPU tests. No compatible GPU device found.
 
Running nnet.checklayer.TestCase
.......... ...
Done nnet.checklayer.TestCase
__________

Test Summary:
	 13 Passed, 0 Failed, 0 Incomplete, 11 Skipped.
	 Time elapsed: 3.4818 seconds.

The results show the number of passed, failed, and skipped tests. If you do not specify the 'ObservationsDimension' option, or do not have a GPU, then the function skips the corresponding tests.

Check Multiple Observations

For multi-observation input, the layer expects an array of observations of size h-by-w-by-c-by-N, where h, w, and c are the height, width, and number of channels, respectively, and N is the number of observations.

To check the layer validity for multiple observations, specify the typical size of an observation and set 'ObservationDimension' to 4.

layer = preluLayer(20,'prelu');
validInputSize = [5 5 20];
checkLayer(layer,validInputSize,'ObservationDimension',4)
Skipping GPU tests. No compatible GPU device found.
 
Running nnet.checklayer.TestCase
.......... ........
Done nnet.checklayer.TestCase
__________

Test Summary:
	 18 Passed, 0 Failed, 0 Incomplete, 6 Skipped.
	 Time elapsed: 3.1679 seconds.

In this case, the function does not detect any issues with the layer.

List of Tests

The checkLayer function checks the validity of a custom layer by performing a series of tests.

Intermediate Layers

The checkLayer function uses these tests to check the validity of custom intermediate layers (layers of type nnet.layer.Layer).

TestDescription
methodSignaturesAreCorrectThe syntaxes of the layer functions are correctly defined.
predictDoesNotErrorpredict does not error.
forwardDoesNotError

forward does not error.

forwardPredictAreConsistentInSize

forward and predict output values of the same size.

backwardDoesNotErrorbackward does not error.
backwardIsConsistentInSize

The outputs of backward are consistent in size:

  • The derivatives with respect to each input are the same size as the corresponding input.

  • The derivatives with respect to each learnable parameter are the same size as the corresponding learnable parameter.

predictIsConsistentInType

The outputs of predict are consistent in type with the inputs.

forwardIsConsistentInType

The outputs of forward are consistent in type with the inputs.

backwardIsConsistentInType

The outputs of backward are consistent in type with the inputs.

gradientsAreNumericallyCorrectThe gradients computed in backward are consistent with the numerical gradients.

The tests predictIsConsistentInType, forwardIsConsistentInType, and backwardIsConsistentInType also check for GPU compatibility. To execute the layer functions on a GPU, the functions must support inputs and outputs of type gpuArray with the underlying data type single.

If you have not implemented forward, then checkLayer does not run the forwardDoesNotError, forwardPredictAreConsistentInSize, and forwardIsConsistentInType tests.

Output Layers

The checkLayer function uses these tests to check the validity of custom output layers (layers of type nnet.layer.ClassificationLayer or nnet.layer.RegressionLayer).

TestDescription
forwardLossDoesNotErrorforwardLoss does not error.
backwardLossDoesNotErrorbackwardLoss does not error.
forwardLossIsScalarThe output of forwardLoss is scalar.
backwardLossIsConsistentInSizeThe output of backwardLoss is consistent in size: dLdY is the same size as the predictions Y.
forwardLossIsConsistentInType

The output of forwardLoss is consistent in type: loss is the same type as the predictions Y.

backwardLossIsConsistentInType

The output of backwardLoss is consistent in type: dLdY must be the same type as the predictions Y.

gradientsAreNumericallyCorrectThe gradients computed in backwardLoss are numerically correct.

The forwardLossIsConsistentInType and backwardLossIsConsistentInType tests also check for GPU compatibility. To execute the layer functions on a GPU, the functions must support inputs and outputs of type gpuArray with the underlying data type single.

Generated Data

To check the layer validity, the checkLayer function generates data depending on the type of layer:

Layer TypeDescription of Generated Data
IntermediateValues in the range [-1,1]
Regression outputPredictions and targets with values in the range [-1,1]
Classification output

Predictions with values in the range [0,1].

If you specify the 'ObservationDimension' option, then the targets are one-hot encoded vectors (vectors containing a single 1, and 0 elsewhere).

If you do not specify the 'ObservationDimension' option, then the targets are values in the range [0,1].

To check for multiple observations, specify the observation dimension using the 'ObservationDimension' name-value pair. If you specify the observation dimension, then the checkLayer function checks that the layer functions are valid using generated data with mini-batches of size 1 and 2. If you do not specify this name-value pair, then the function skips the tests that check that the layer functions are valid for multiple observations.

Diagnostics

If a test fails when you use checkLayer, then the function provides a test diagnostic and a framework diagnostic. The test diagnostic highlights any issues found with the layer. The framework diagnostic provides more detailed information.

Function Syntaxes

The test methodSignaturesAreCorrect checks that the layer functions have correctly defined syntaxes.

Test DiagnosticDescriptionPossible Solution
Incorrect number of input arguments for 'predict' in Layer.The syntax for predict is not consistent with the number of layer inputs.

Specify the correct number of input and output arguments in predict.

The syntax for predict is

[Z1,…,Zm] = predict(layer,X1,…,Xn)
where X1,…,Xn are the n layer inputs and Z1,…,Zm are the m layer outputs. The values n and m must correspond to the NumInputs and NumOutputs properties of the layer.

Tip

If the number of inputs to predict can vary, then use varargin instead of X1,…,Xn. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi. If the number of outputs can vary, then use varargout instead of Z1,…,Zm. In this case, varargout is a cell array of the outputs, where varargout{j} corresponds to Zj.

Incorrect number of output arguments for 'predict' in LayerThe syntax for predict is not consistent with the number of layer outputs.
Incorrect number of input arguments for 'forward' in LayerThe syntax for forward is not consistent with the number of layer inputs.

Specify the correct number of input and output arguments in forward.

The syntax for forward is

[Z1,…,Zm,memory] = forward(layer,X1,…,Xn)
where X1,…,Xn are the n layer inputs, Z1,…,Zm are the m layer outputs, and memory is the memory of the layer.

Tip

If the number of inputs to forward can vary, then use varargin instead of X1,…,Xn. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi. If the number of outputs can vary, then use varargout instead of Z1,…,Zm. In this case, varargout is a cell array of the outputs, where varargout{j} corresponds to Zj for j=1,…,NumOutputs and varargout{NumOutputs+1} corresponds to memory.

Incorrect number of output arguments for 'forward' in LayerThe syntax for forward is not consistent with the number of layer outputs.
Incorrect number of input arguments for 'backward' in LayerThe syntax for backward is not consistent with the number of layer inputs and outputs.

Specify the correct number of input and output arguments in backward.

The syntax for backward is

[dLdX1,…,dLdXn,dLdW1,…,dLdWk] = backward(layer,X1,…,Xn,Z1,…,Zm,dLdZ1,…,dLdZm,memory)
where X1,…,Xn are the n layer inputs, Z1,…,Zm are the m outputs of forward, dLdZ1,…,dLdZm are the gradients backward propagated from the next layer, and memory is the memory output of forward. For the outputs, dLdX1,…,dLdXn are the derivatives of the loss with respect to the layer inputs and dLdW1,…,dLdWk are the derivatives of the loss with respect to the k learnable parameters. To reduce memory usage by preventing unused variables being saved between the forward and backward pass, replace the corresponding input arguments with ~.

Tip

If the number of inputs to backward can vary, then use varargin instead of the input arguments after layer. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi for i=1,…,NumInputs, varargin{NumInputs+j} and varargin{NumInputs+NumOutputs+j} correspond to Zj and dLdZj, respectively, for j=1,…,NumOutputs, and varargin{end} corresponds to memory.

If the number of outputs can vary, then use varargout instead of the output arguments. In this case, varargout is a cell array of the outputs, where varargout{i} corresponds to dLdXi for i=1,…,NumInputs and varargout{NumInputs+t} corresponds to dLdWt for t=1,…,k, where k is the number of learnable parameters.

Incorrect number of output arguments for 'backward' in LayerThe syntax for backward is not consistent with the number of layer outputs.

For layers with multiple inputs or outputs, you must set the values of the layer properties NumInputs (or alternatively, InputNames) and NumOutputs (or alternatively, OutputNames) in the layer constructor function, respectively.

Multiple Observations

The checkLayer function checks that the layer functions are valid for single and multiple observations. To check for multiple observations, specify the observation dimension using the 'ObservationDimension' name-value pair. If you specify the observation dimension, then the checkLayer function checks that the layer functions are valid using generated data with mini-batches of size 1 and 2. If you do not specify this name-value pair, then the function skips the tests that check that the layer functions are valid for multiple observations.

Test DiagnosticDescriptionPossible Solution
Skipping multi-observation tests. To enable checks with multiple observations, specify the 'ObservationDimension' parameter in checkLayer.If you do not specify the 'ObservationDimension' parameter in checkLayer, then the function skips the tests that check data with multiple observations.

Use the command checkLayer(layer,validInputSize,'ObservationDimension',dim), where layer is an instance of the custom layer, validInputSize is a vector specifying the valid input size to the layer, and dim specifies the dimension of the observations in the layer input.

For more information, see Layer Input Sizes.

Functions Do Not Error

These tests check that the layers do not error when passed input data of valid size.

Intermediate Layers.  The tests predictDoesNotError, forwardDoesNotError, and backwardDoesNotError check that the layer functions do not error when passed inputs of valid size. If you specify an observation dimension, then the function checks the layer for both a single observation and multiple observations.

Test DiagnosticDescriptionPossible Solution
The function 'predict' threw an error:The predict function errors when passed data of size validInputSize.Address the error described in the Framework Diagnostic section.
The function 'forward' threw an error:The forward function errors when passed data of size validInputSize.
The function 'backward' threw an error:The backward function errors when passed the output of predict.

Output Layers.  The tests forwardLossDoesNotError and backwardLossDoesNotError check that the layer functions do not error when passed inputs of valid size. If you specify an observation dimension, then the function checks the layer for both a single observation and multiple observations.

Test DiagnosticDescriptionPossible Solution
The function 'forwardLoss' threw an error:The forwardLoss function errors when passed data of size validInputSize.Address the error described in the Framework Diagnostic section.
The function 'backwardLoss' threw an error:The backwardLoss function errors when passed data of size validInputSize.

Outputs Are Consistent in Size

These tests check that the layer function outputs are consistent in size.

Intermediate Layers.  The test backwardIsConsistentInSize checks that the backward function outputs derivatives of the correct size.

The syntax for backward is

[dLdX1,…,dLdXn,dLdW1,…,dLdWk] = backward(layer,X1,…,Xn,Z1,…,Zm,dLdZ1,…,dLdZm,memory)
where X1,…,Xn are the n layer inputs, Z1,…,Zm are the m outputs of forward, dLdZ1,…,dLdZm are the gradients backward propagated from the next layer, and memory is the memory output of forward. For the outputs, dLdX1,…,dLdXn are the derivatives of the loss with respect to the layer inputs and dLdW1,…,dLdWk are the derivatives of the loss with respect to the k learnable parameters. To reduce memory usage by preventing unused variables being saved between the forward and backward pass, replace the corresponding input arguments with ~.

Tip

If the number of inputs to backward can vary, then use varargin instead of the input arguments after layer. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi for i=1,…,NumInputs, varargin{NumInputs+j} and varargin{NumInputs+NumOutputs+j} correspond to Zj and dLdZj, respectively, for j=1,…,NumOutputs, and varargin{end} corresponds to memory.

If the number of outputs can vary, then use varargout instead of the output arguments. In this case, varargout is a cell array of the outputs, where varargout{i} corresponds to dLdXi for i=1,…,NumInputs and varargout{NumInputs+t} corresponds to dLdWt for t=1,…,k, where k is the number of learnable parameters.

The derivatives dLdX1, …, dLdXn must be the same size as the corresponding layer inputs, and dLdW1,…,dLdWk must be the same size as the corresponding learnable parameters. The sizes must be consistent for input data with single and multiple observations.

Test DiagnosticDescriptionPossible Solution
Incorrect size of 'dLdX' for 'backward'.The derivatives of the loss with respect to the layer inputs must be the same size as the corresponding layer input.

Return the derivatives dLdX1,…,dLdXn with the same size as the corresponding layer inputs X1,…,Xn.

Incorrect size of the derivative of the loss with respect to the input 'in1' for 'backward'
The size of 'Z' returned from 'forward' must be the same as for 'predict'.The outputs of predict must be the same size as the corresponding outputs of forward.

Return the outputs Z1,…,Zm of predict with the same size as the corresponding outputs Z1,…,Zm of forward.

Incorrect size of the derivative of the loss with respect to 'W' for 'backward'.The derivatives of the loss with respect to the learnable parameters must be the same size as the corresponding learnable parameters.

Return the derivatives dLdW1,…,dLdWk with the same size as the corresponding learnable parameters W1,…,Wk.

Output Layers.  The tests forwardLossIsScalar and backwardLossIsConsistentInSize check that the outputs of forwardLoss and backwardLoss are of the correct size.

The syntax for forwardLoss is loss = forwardLoss(layer, Y, T). The input Y corresponds to the predictions made by the network. These predictions are the output of the previous layer. The input T corresponds to the training targets. The output loss is the loss between Y and T according to the specified loss function. The output loss must be scalar.

The syntax for backwardLoss is dLdY = backwardLoss(layer, Y, T). The inputs Y are the predictions made by the network and T are the training targets. The output dLdY is the derivative of the loss with respect to the predictions Y. The output dLdY must be the same size as the layer input Y.

Test DiagnosticDescriptionPossible Solution
Incorrect size of 'loss' for 'forwardLoss'.The output loss of forwardLoss must be a scalar.

Return the output loss as a scalar. For example, if you have multiple values of the loss, then you can use mean or sum.

Incorrect size of the derivative of loss 'dLdY' for 'backwardLoss'.The derivatives of the loss with respect to the layer input must be the same size as the layer input.Return derivative dLdY with the same size as the layer input Y.

Consistent Data Types and GPU Compatibility

These tests check that the layer function outputs are consistent in type and that the layer functions are GPU compatible.

For GPU compatibility, the layer functions must support inputs and return outputs of type gpuArray. Any other functions the layer uses must do the same. Many MATLAB® built-in functions support gpuArray input arguments. If you call any of these functions with at least one gpuArray input, then the function executes on the GPU and returns a gpuArray output. For a list of functions that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox). To use a GPU for deep learning, you must also have a CUDA® enabled NVIDIA® GPU with compute capability 3.0 or higher. For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).

Intermediate Layers.  The tests predictIsConsistentInType, forwardIsConsistentInType, and backwardIsConsistentInType check that the layer functions output variables of the correct data type. The tests check that the layer functions return consistent data types when given inputs of the data types single, double, and gpuArray with the underlying types single or double.

Tip

If you preallocate arrays using functions like zeros, then you must ensure that the data types of these arrays are consistent with the layer function inputs. To create an array of zeros of the same data type of another array, use the 'like' option of zeros. For example, to initialize an array of zeros of size sz with the same data type as the array X, use Z = zeros(sz,'like',X).

Test DiagnosticDescriptionPossible Solution
Incorrect type of 'Z' for 'predict'.The types of the outputs Z1,…,Zm of predict must be consistent with the inputs X1,…,Xn.

Return the outputs Z1,…,Zm with the same type as the inputs X1,…,Xn.

Incorrect type of output 'out1' for 'predict'.
Incorrect type of 'Z' for 'forward'.The types of the outputs Z1,…,Zm of forward must be consistent with the inputs X1,…,Xn.
Incorrect type of output 'out1' for 'forward'.
Incorrect type of 'dLdX' for 'backward'.The types of the derivatives dLdX1,…,dLdXn of backward must be consistent with the inputs X1,…,Xn.

Return the derivatives dLdX1,…,dLdXn with the same type as the inputs X1,…,Xn.

Incorrect type of the derivative of the loss with respect to the input 'in1' for 'backward'.
Incorrect type of the derivative of loss with respect to 'W' for 'backward'.The type of the derivative of the loss of the learnable parameters must be consistent with the corresponding learnable parameters.

For each learnable parameter, return the derivative with the same type as the corresponding learnable parameter.

Output Layers.  The tests forwardLossIsConsistentInType and backwardLossIsConsistentInType check that the layer functions output variables of the correct data type. The tests check that the layers return consistent data types when given inputs of the data types single, double, and gpuArray with the underlying types single or double.

Test DiagnosticDescriptionPossible Solution
Incorrect type of 'loss' for 'forwardLoss'.The type of the output loss of forwardLoss must be consistent with the input Y.

Return loss with the same type as the input Y.

Incorrect type of the derivative of loss 'dLdY' for 'backwardLoss'.The type of the output dLdY of backwardLoss must be consistent with the input Y.

Return dLdY with the same type as the input Y.

Gradients Are Numerically Correct

The test gradientsAreNumericallyCorrect checks that the gradients computed by the layer functions are numerically correct.

Intermediate Layers.  The test gradientsAreNumericallyCorrect tests that the gradients computed in backward are numerically correct.

Test DiagnosticDescriptionPossible Solution
The derivative 'dLdX' for 'backward' is inconsistent with the numerical gradient.

One or more of the following:

  • The specified derivative is incorrectly computed

  • Function is non-differentiable at some input points

  • Error tolerance is too small

Check that the derivatives in backward are correctly computed.

If the derivatives are correctly computed, then in the Framework Diagnostic section, manually check the absolute and relative error between the actual and expected values of the derivative.

If the absolute and relative errors are within an acceptable margin of the tolerance, then you can ignore this test diagnostic.

The derivative of the loss with respect to the input 'in1' for 'backward' is inconsistent with the numerical gradient.
The derivative of loss with respect to 'W' for 'backward' is inconsistent with the numerical gradient.

Output Layers.  The test gradientsAreNumericallyCorrect checks that the gradients computed in backwardLoss are numerically correct.

Test DiagnosticDescriptionPossible Solution
The derivative 'dLdY' for 'backwardLoss' is inconsistent with the numerical gradient.

One or more of the following:

  • The derivative with respect to the predictions Y is incorrectly computed

  • Function is non-differentiable at some input points

  • Error tolerance is too small

Check that the derivatives in backwardLoss are correctly computed.

If the derivatives are correctly computed, then in the Framework Diagnostic section, manually check the absolute and relative error between the actual and expected values of the derivative.

If the absolute and relative errors are within an acceptable margin of the tolerance, then you can ignore this test diagnostic.

See Also

|

Related Topics