Programmatically determine which Deep Learning layer properties contain learnables

In the Deep Learning Toolbox, there are a variety of layer object types. The display methods of these objects indicate which object properties contain learnable parameter data. For example, for LSTMLayer objects, the learnable parameters are stored in the properties "InputWeights", "RecurrentWeights", and "Bias" as shown below. My question is, is there a way, given a layer object, to programmatically determine the subset of its properties that are learnable?

layer = lstmLayer(100,'Name','lstm1')

layer = 
  LSTMLayer with properties:

                       Name: 'lstm1'
                 InputNames: {'in'}
                OutputNames: {'out'}
                  NumInputs: 1
                 NumOutputs: 1
             HasStateInputs: 0
            HasStateOutputs: 0

   Hyperparameters
                  InputSize: 'auto'
             NumHiddenUnits: 100
                 OutputMode: 'sequence'
    StateActivationFunction: 'tanh'
     GateActivationFunction: 'sigmoid'

   Learnable Parameters
               InputWeights: []
           RecurrentWeights: []
                       Bias: []

   State Parameters
                HiddenState: []
                  CellState: []

Use properties method to see a list of all properties.

Answer 1

[Pratyush Swain]

Hi Matt,

I understand you want to access the learnable properties of a deep learning layer object. There is no direct way fetch the trainable properties of a layer but a workaround can be to initialize a deep learning layer from the "lstm" layer object and then fetch its learnable parameters.

Please refer to the below example implementation:

% Define a lstm layer %
layer = lstmLayer(100,'Name','lstm1')

layer = 
  LSTMLayer with properties:

                       Name: 'lstm1'
                 InputNames: {'in'}
                OutputNames: {'out'}
                  NumInputs: 1
                 NumOutputs: 1
             HasStateInputs: 0
            HasStateOutputs: 0

   Hyperparameters
                  InputSize: 'auto'
             NumHiddenUnits: 100
                 OutputMode: 'sequence'
    StateActivationFunction: 'tanh'
     GateActivationFunction: 'sigmoid'

   Learnable Parameters
               InputWeights: []
           RecurrentWeights: []
                       Bias: []

   State Parameters
                HiddenState: []
                  CellState: []

Use properties method to see a list of all properties.

% Define a deep learning network %
net = dlnetwork(layer,Initialize=false);

% Retreive the learnable properties %
properties = net.Learnables;

% Display the learnable properties %
disp(properties)

     Layer         Parameter            Value    
    _______    __________________    ____________

    "lstm1"    "InputWeights"        {0×0 double}
    "lstm1"    "RecurrentWeights"    {0×0 double}
    "lstm1"    "Bias"                {0×0 double}

As we can observe, the "dlnetwork" forms a single layer network named as "lstm1" and we have successfully retreived its learnable properties as a table.

For more information , please refer to:

https://www.mathworks.com/help/deeplearning/ref/dlnetwork.initialize.html

Hope this helps.

[Matt J]

Perfect! Thanks.

[Matt J]

In order to use this technique, though, I must programmatically determine if the layer is a type of output layer, because dlnetworks may not contain output layers. Is there a way to detect that?

Answer 2

[Vishnu Keyen]

Let's define the a network

layers = [sequenceInputLayer(32, 'Name', 'input')
            lstmLayer(128, 'OutputMode', 'sequence', 'Name', 'lstm')]

layers = 
  2x1 Layer array with layers:

     1   'input'   Sequence Input   Sequence input with 32 dimensions
     2   'lstm'    LSTM             LSTM with 128 hidden units

since each layer is a nnet.cnn.layer type object, one can determine

you can determine the object type using class() command.

For example,

layerType = class(layers(2))

layerType = 'nnet.cnn.layer.LSTMLayer'

And then you can search for a match in that class for the type you are looking for

For example

contains(class(layers(2)),'lstm','IgnoreCase',true)

ans = logical
   1

you can check for specific parameters like RecurrentWeights once you identify its an LSTM, or a GRU layer..