can i use a custom activation function in this neural network and in what way?

x = linspace(0,1,10000)';

inputSize = 1;

layers = [

featureInputLayer(inputSize,Normalization="none")

fullyConnectedLayer(10)

sigmoidLayer

fullyConnectedLayer(1)

sigmoidLayer];

dlnet = dlnetwork(layers);

numEpochs = 15;

miniBatchSize =100 ;

initialLearnRate = 0.5;

learnRateDropFactor = 0.5;

learnRateDropPeriod = 5;

momentum = 0.9;

icCoeff = 7;

ads = arrayDatastore(x,IterationDimension=1);

mbq = minibatchqueue(ads,MiniBatchSize=miniBatchSize,MiniBatchFormat="BC");

figure

set(gca,YScale="log")

lineLossTrain = animatedline(Color=[0.85 0.325 0.098]);

ylim([0 inf])

xlabel("Iteration")

ylabel("Loss (log scale)")

grid on

velocity = [];

iteration = 0;

learnRate = initialLearnRate;

start = tic;

% Loop over epochs.

for epoch = 1:numEpochs

% Shuffle data.

mbq.shuffle

% Loop over mini-batches.

while hasdata(mbq)

iteration = iteration + 1;

% Read mini-batch of data.

dlX = next(mbq);

% Evaluate the model gradients and loss using dlfeval and the modelGradients function.

[gradients,loss] = dlfeval(@modelGradients, dlnet, dlX, icCoeff);

% Update network parameters using the SGDM optimizer.

[dlnet,velocity] = sgdmupdate(dlnet,gradients,velocity,learnRate,momentum);

% To plot, convert the loss to double.

loss = double(gather(extractdata(loss)));

% Display the training progress.

D = duration(0,0,toc(start),Format="mm:ss.SS");

addpoints(lineLossTrain,iteration,loss)

title("Epoch: " + epoch + " of " + numEpochs + ", Elapsed: " + string(D))

drawnow

end

% Reduce the learning rate.

if mod(epoch,learnRateDropPeriod)==0

learnRate = learnRate*learnRateDropFactor;

end

end

Answer 1

[Christopher Erickson]

There are two solutions I would suggest first:

If you have an elementwise activation function with no learnables (such as "exp" or "sin") you could use "functionLayer". For more generic workflows you could use a custom layer. I would only suggest using a custom layer if your activation function did not operate elementwise or if it has learnables; such as "prelu" in the examples.

Good luck!

[Dimitris Kastoris]

thanks!!

[Dimitris Kastoris]

custom layer doesnt support automatic differentiation...enablehigherderivatives cant be set true in modelLoss script...