How to customize Neural Networks' activation function

173 visualizzazioni (ultimi 30 giorni)
Shahar Hochma
Shahar Hochma il 18 Nov 2016
Risposto: km y il 8 Set 2023
Hi, I would like to implement, using Matlab, a neural network with 3 hidden layers, each using ReLU activation function. How can i do this?
Currently, I know i can set the activation function using:
net.layers{i}.transferFcn = reluLayer();
But this only allows to set a specific type of function that is predefined (like logsig), but ReLU is not one of those functions.
Is there a way to change the layer to the ReLU layer? Thanks

Risposte (6)

Darío Pérez
Darío Pérez il 24 Ott 2017
As far as I am concern, you can use the predefined function 'poslin' (which is a ReLU):
net.layers{i}.transferFcn = 'poslin';
but "other differentiable transfer functions can be created and used if desired": Multilayer Neural Network Architecture.
Not sure how discontinuity at x=0 would affect training stage. In addition, recent articles state that ReLU should be used for regression problems but it achieves worst results than 'tansig' or 'logsig' in one of my examples. Has anyone any thoughts/conclusions in this regard?

daniel il 30 Nov 2017
its useful for deeper nets so depends on your # layers, it's mostly for minimizing vanishing/exploding gradients
  1 Commento
Jan il 21 Gen 2019
Modificato: Jan il 21 Gen 2019
[MOVED from flags] Flagged by Amine Bendali on 18 Jan 2019 at 22:31.
i want to check it later
[Please use flags only to inform admins and editors about messages, which should be reviewed. Thanks]

Accedi per commentare.

peter chevo
peter chevo il 31 Ago 2018
Modificato: peter chevo il 31 Ago 2018
1. Copy folder and file of C:\Program Files\MATLAB\MATLAB Production Server\R2015a\toolbox\nnet\nnet\nntransfer\ such as +tansig and tansig.m to current path 2. edit file name such as tansig.m is my_transfer.m 3. edit folders name such as +tansig is +my_transfer 4. edit last line in apply.m to your formula equation
  1 Commento
Abdelwahab Afifi
Abdelwahab Afifi il 3 Mar 2021
This method doesn't work. Because each Activation function has its own files with its own sturcture/values/equation.

Accedi per commentare.

Maria Duarte Rosa
Maria Duarte Rosa il 5 Apr 2019
For Deep Learning networks one can create a custom activation layer using:

David Willingham
David Willingham il 19 Mag 2022
@Maria Duarte Rosa gave a good answer on how to create a custom activation layer by visiting this page: Define Custom Deep Learning Layers
This extended answer is aimed at addressing how to define a "Relu" function in MATLAB.
With MATLAB's current Deep Learning framework, ReLu function is a standard layer you can define.
Here is an example:
Create a ReLU layer with the name 'relu1'.
layer = reluLayer('Name','relu1')
layer =
ReLULayer with properties:
Name: 'relu1'
Include a ReLU layer in a Layer array.
layers = [ ...
imageInputLayer([28 28 1])
layers =
7x1 Layer array with layers:
1 '' Image Input 28x28x1 images with 'zerocenter' normalization
2 '' Convolution 20 5x5 convolutions with stride [1 1] and padding [0 0 0 0]
3 '' ReLU ReLU
4 '' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0]
5 '' Fully Connected 10 fully connected layer
6 '' Softmax softmax
7 '' Classification Output crossentropyex
For more information on ReLu layer see this page: reluLayer
  3 Commenti
Antoni Woss
Antoni Woss il 13 Set 2022
The reluLayer will apply the rectified linear unit operation to the outputs of the preceding layer. As you mentioned the reluLayer is exactly a layer of activation functions. For example, if the reluLayer follows a 2D convolutional layer, where the output of the convolution layer is say 10x10x5 (5 filters each of 10 pixels by 10 pixels), then the reluLayer will apply the rectified linear operation to each of the 10x10x5 values.
As the reluLayer operates elementwise, you do not need to specify architecture, such as specifying the outputSize in the fullyConnectedLayer. The reluLayer infers the correct architecture from the previous layer.
To specify the number of output activations (number of neurons as you are referring to) for MATLABs built in layers, you can take a look at the documentation for the particular layer. For example:
  • For the lstmLayer specify the number of hidden units, numHiddenUnits, to specify the number of output activations.
  • For the fullyConnectedLayer you can specify the outputSize.
Here are some documentation links to the layers discussed in this post for further reference:
Alexander Krauss
Alexander Krauss il 14 Set 2022
Thanks a lot Antoni, great and insightful answer. :)
I would just like to know how dropout layers are to be understood.
I mean I know what dropout and dropout rates respectively are and how they work.
But again my question refers to the implementation of dropout layers in MATLAB.
If I want to achieve a dropout rate of say 15 % (for each hidden layer), do I have to add a 15 % dropout layer after every activation layer or will it be enough to just add one dropout layer which then applies to the entire network of hidden layers.
Would be great if you could answer this one as well.
Best regards

Accedi per commentare.

km y
km y il 8 Set 2023

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by