Error in the regularization of Neural Networks and understanding 'perform'

Hello all,

In MATLAB 2017b..

I have a neural network for fitting pupose.

[I N] = size(x); % I = 3, N = 500
[O N] = size (t); % O = 1

Following is my code only for regularization purpose..

clc;
close all; rng(0);

x = input_vel_mass_pos_steps';
t = torque_steps';

trainFcn = 'trainrp';  

% Create a Fitting Network
hiddenLayerSize = 9;
net = fitnet(hiddenLayerSize,trainFcn);

%regularization Lambda
regularization_term = [0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.2 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.3 0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1];
mm = size(regularization_term,2);

x_train = x(:,sstrain);t_train = t(:,sstrain); %sstrain - indices got from a different program

x_test = x(:,sstest);t_test = t(:,sstest); %%sstest - indices got from a different program

trainPerformance = zeros(10,mm); %zeros(j,mm)
testPerformance = zeros(10,mm);
for j = 1:10
    for i = 1:mm
        net = fitnet(hiddenLayerSize,trainFcn);
        net.divideFcn = '';
        net.trainParam.epochs = 300;
        %nett.performFcn='msereg';
        %nett.performParam.ratio = regularization_term(i);
        net.performParam.regularization = regularization_term(i);
        [net trr] = train(net,x_train,t_train);
        y_train = nett(x_train);
        trainPerformance(j,i) = sqrt(perform(net,t_train,y_train));
        %trainPerformance(j,i) = norm((y_train-t_train)/sqrt(length(y_train)));
        y_test = net(x_test);
        testPerformance(j,i) = sqrt(perform(net,t_test,y_test));
        %testPerformance(j,i) = norm((y_test-t_test)/sqrt(length(y_test)));
    end
end

plot(regularization_term, mean(trainPerformance),regularization_term,mean(testPerformance))
hold on;
legend('trainperformance-RMSE','testperformacne-RMSE')
xlabel('Regularization Ratio')
ylabel('RMSE')

My questions are as following..

1. What does the 'perform' function does? As per the theory it takes into consideration

net.performFcn  and 
net.performParam

net.performFcn = mse by default and By using net.performFcn.regularization = regularization_term(i) why then at *last* (in the command window to check) when I do testPerformance = sqrt(perform(nett,t_test,y_test)) and testPerformance = norm((y_test-t_test)/sqrt(length(y_test))) the results are not equal? Ain't they suppose to be equal?

2. when my input code is net.performFcn.regularization = regularization_term(i) and testperformance = sqrt(perform(nett,t_test,y_test)) I get very weird graph as follows -

How can it be a straight line? More importantly how come test error was less than training error? By this graph, I won't be able to get the optimised lambda value from this. This should have been a curve. The testing should have gone from down to up as a curve and by penalty we should have seen the training error going up..

3. I used in mine(later) from the previous code(which I saw) as net.performFcn = 'msereg' and net.performParam.ratio = regularization(i) used to work for regularization but its not the case in new versions as I checked later (after using 'msereg') net.performFcn is still 'mse'. Question is what is then now net.performParam.ratio? and what's the difference between net.performParam.ratio and net.performParam.regularization? Because though net.performFcn is not 'msereg' my code above ran for net.performParam.ratio = regularization(i). Why did it ran then for net.performParam.ratio in the first place itself? I then used it anyhow to see my results..

4. Below are my cases for plot of RMSE vs regularization(i), all of which are abnormal. Any help understanding where am I going wrong is appreciated and concept of my regularization will be clear..

• using net.performParam.regularization = regularization_term(i) and testPerformance = norm(y_test-t_test)

• using net.performParam.ratio = regularization_term(i) and testPerformance = sqrt(perform(nett,t_test,y_test))

• using net.performParam.ratio = regularization_term(i) and testPerformance = norm(y_test-t_test)

[Greg Heath]

Why don't you demonstrate using MATLAB data ????

help nndatasets

doc nndatasets