How can I improve the performance of a feed-forward backpropagation neural network?

Hi, I am working with MATLAB R2013a to build a prediction neural network model. I have tried to use different training algorithms, activation functions and number of hidden neurons but still can't get the R more than 0.8 for training set, validation set and testing set. The R of training set for some networks can be more than 0.8 but provide low R values (around 0.4~0.5) for validation and testing set. Below are the codes. Is there any solutions to improve the performance and R value?

inputs<48x206>, targets<5x206>

inputs = inputs;

targets = targets;

hiddenLayerSize = 15;

net = fitnet(hiddenLayerSize);

net.layers{1}.transferFcn='tansig';

net.layers{2}.transferFcn='purelin';

net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};

net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};

net.divideFcn = 'dividerand';

net.divideMode = 'sample';

net.divideParam.trainRatio = 70/100;

net.divideParam.valRatio = 15/100;

net.divideParam.testRatio = 15/100;

net.trainFcn = 'traincgp';

net.performFcn = 'mse';

net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ... 'plotregression', 'plotfit'};

[net,tr] = train(net,inputs,targets);

outputs = net(inputs);

errors = gsubtract(targets,outputs);

performance = perform(net,targets,outputs)

trainTargets = targets .* tr.trainMask{1};

valTargets = targets .* tr.valMask{1};

testTargets = targets .* tr.testMask{1};

trainPerformance = perform(net,trainTargets,outputs)

valPerformance = perform(net,valTargets,outputs)

testPerformance = perform(net,testTargets,outputs)

view(net)

[pepper yuan]

Hi, Jocelyn, have you solve the problem of improving the performance of neural network? As I'm dealing the problem same with you, can you provide me your email, so I can ask you some questions via the email. Appreciate if you can rely me.

Answer 1

[Greg Heath]

% 1. This is REGRESSION, not PREDICTION.

%2. Placeholders:

 input = randn(48,206);
 target = randn(5,48)*input.^2+randn(5,48)*input+ randn(5,206);

 [ I N ] = size(input)   % [ 48 206 ]
 [ O N ] = size(target)  % [ 5  206 ]

%3. N = 206 doesn't provide enough information to be using I = 48:

 Ntrn   = N - 2*round(0.15*N)   % 144 training points
 Ntrneq = Ntrn*O                % 720 No. of training equations

 % Nw = (I+1)*H+(H+1)*O         No. of weights for an I-H-O net
 % Nw > Ntrneq     <==>  H > Hub

 Hub = (0.7*N-O)/(I+O+1)    % 2.6 to 2.9  for O = 5 to 1

% Therefore, regardless of O, the net is OVERFIT when H >= 3

% 4. Remedies:

 a.  H <= 2                   and/or 
 b. Validation Stopping       and/or
 c. TRAINBR Regularization    and/or 
 d. MSEREG Regularization     and/or
 e. Input variable Reduction

% 5. I recommend trying 4e first. To get a feel for the data, you could

 a. Standardize all variables with zscore or mapstd
 b. Create 48 graphs with the 5 targets plotted vs each input
 c. Remove or modify outliers
 d. Obtain the 54 x 54 correlation coefficient matrix
 e. Consider multiple single output models
 f. Use STEPWISEFIT and/or STEPWISE on a linear model

Good Luck.

P.S. I try not to waste my time by using statements that just assign default parameter values

Hope this helps.

Thank you for formally accepting my answer

Greg

[Greg Heath]

CORRECTION:

Hub = (Ntrneq-O)/(I+O+1)% 13.2 for O = 5

I have run 10 initial random number trials for each value of H = 0:13. This resulted in a DECREASE in performance as H increased!!!??? I was quite surprised since I had never experienced that before.

This became somewhat more believable when I plotted the data. The best Linear model (i.e., H=0 ) yielded NMSE = 0.44, Rsq = 1-NMSE = 0.56 and R = sqrt(Rsq) = 0.75.

Next I tried a non-neural quadratic model and hit paydirt!

If this were my problem I would use STEPWISEFIT to see which inputs, crossproducts and squares are really necessary.

Hope this helps.

Greg

Answer 2

[Jocelyn]

Thanks Greg for your suggested solutions. I have tried all the above remedies and neural network with MSEREG regularization provides the highest value of R for training and testing which are 0.73 and 0.71. However, I have to get the R value more than 0.8. Is there any other solution to improve it? It is really urgent and I really appreciate your help.

[Greg Heath]

1. You are still wasting time, space and attention by keeping statements that merely assign default values

2. How many inputs and outputs are you using after the variable reduction?

3. What happens when you use

   net.divideFcn = 'dividetrain'

and try to minimize H using a double for loop as in my posts?

Hope this helps.

Greg

Answer 3

[Jocelyn]

Sorry for the late reply. I didn't reduce the variables and have to use back the same number of variables as those variables are the tested significant variables from the structural equation modeling. So, the input and output sizes remain same. I have make some changes as below and would like to ask for your help to improve the performance.Thanks.

close all,clear all,clc,tic

load inputs&targets

x=KMinputs;

t=KMtargets;

[I N]=size(x) %[48 206]

[O N]=size(t) %[5 206]

rng('default')

h=4

net = fitnet(h);

net.layers{1}.transferFcn='tansig';

net.layers{2}.transferFcn='tansig';

net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};

net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};

Ntrneq=N*O

ynaive=mean(t,2)

Nw00=numel(ynaive)

Ndof00=Ntrneq-Nw00

y00=repmat(ynaive,1,N);

SSE00=sse(t-y00)

MSE00=SSE00/Ntrneq %0.6768

MSE00a=SSE00/Ndof00 %0.6801

Nw=(I+1)*h+(h+1)*O %221

Ndof=Ntrneq-Nw %809

net.divideFcn = 'dividerand';

net.divideParam.trainRatio = 0.9;

net.divideParam.valRatio = 0;

net.divideParam.testRatio = 0.1;

net.trainFcn = 'trainlm';

MSEgoal=0.01*Ndof*MSE00a/Ntrneq %0.0053

MinGrad=MSEgoal/100 %5.3415e-05

net.trainParam.goal=MSEgoal;

net.trainParam.min_grad=MinGrad;

net.performFcn = 'msereg';

net.performParam.regularization = 0.5;

net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ... 'plotregression', 'plotfit'};

[net tr y e] = train(net,x,t);

y = net(x);

errors = gsubtract(t,y);

performance = perform(net,t,y) %0.1574

trainTargets = t .* tr.trainMask{1};

testTargets = t .* tr.testMask{1};

trainPerformance = perform(net,trainTargets,y) %0.1531

testPerformance = perform(net,testTargets,y) %0.1953

[r,m,b] = regression(t,y)

NMSE= mse(e)/MSE00; %0.4181

R2= 1-NMSE; %0.5819

view(net)

%Rtrn=0.78149

%Rtst=0.78322

%Rtotal=0.78071

[Tien Tran]

Hi Jocelyn You can give me your data (input and target). I will try to help if I can

[Jocelyn]

Hi Tien Tran. I have attached my data file here. Thanks for your help.

[Tien Tran]

Hi Jocelyn

I have tried your data, but I find that your data is unrealistic or lack of data points to train ANN effective.

[Jocelyn]

Hi Tien Tran. I understand that there is lack of data points to train ANN but it is impossible for me to collect more data. So, I would like to ask for any solution to improve the correlation coefficient (R). I just need the R for training and testing to be more than 0.8.Thanks.

[Tien Tran]

If you just need high coefficient of determination (R or R^2) to present without require the quality, you can choose all data = training data; validation data = 70% of total data and testing data = 70% of total data. I do not recommend for this ideal.

[Greg Heath]

If you post the data in *.m or *.txt, I may be able to take a look at it.

Again: My first impression is that you don't have enough data to accurately deal with 48 inputs.

Greg

[Jocelyn]

Hi Greg. I have attached my data in *.txt here. Actually, the inputs can be classified into 3 categories(category 1=IN1->IN9, category 2=IN10->IN32, category 3=IN33->IN48). Does it applicable to build 3 NN for each category with the targets? Thanks.

Answer 4

[Jocelyn]

Is it feasible to set the rng to rng('default') or a fixed seed and generator for the neural network? How to predict the outputs of a new set of inputs based on the previous network training?

[Greg Heath]

I consider it not only feasible, but NECESSARY to initialize the RNG before training the 1st of multiple designs.

How else could you duplicate your work?

Have you searched for any of my design examples?

ynew = net(xnew)

yeids new answers

Hope this helps.

Greg

[Jocelyn]

Thanks Greg. I used back the previous msereg regularization codes by setting rng(22,'twister') and changing training algorithm to traincgb and regularization=0.4 without looping. I get better R and mse but I wonder is it feasible to have this neural network.I will get back the same values of R and mse no matter how many times I run it.

I did looked through your design examples but mostly are not MIMO and my work is more complicated. In the Ntrials looping, how to get Rtrn,Rval and Rtst for each Ntrials loop?

[Greg Heath]

 1. It is more appropriate to use the coefficient of determination 
(See Wikipedia), Rsq = 1-NMSE as the performance function.
 2. Rsq is the fraction of the mean target variance that is 
modeled by the net.
 3. NMSE is the normalized mean-square-error. The normalization 
denominator is the mean target variance.
 4. Your goal of R > 0.8 is equivalent to Rsq > 0.64, that is, you are 
only requiring the net to model 64% of the target variance.
 5. Typically, I advise multivariable regression designers to 
    a. Practice on MATLAB example data obtained from 
       help nndatasets
       doc  nndatasets
    b. Standardize data to zero-mean/unit-variance. Then 
       i. All variables are on an equal footing
       ii. NMSE = MSE
    c. Use MINMAX to find outliers that should be modified 
       or deleted.
    d. Plot inputs, targets and targets vs inputs to get a feel 
       for the data.
    e. Determine how well data is linearly related via the  
       correlation coefficient matrix and or a linear model.
       i. Strong correlations among inputs indicates that the 
       input dimensions should probably be reduced via input 
       elimination and/or combination.
       ii. Strong correlations among targets indicate that the 
       output dimensions could be reduced. However,  
       sometimes highly correlated outputs make designs easier.
6. In the case of 48 inputs for 5 targets, it is very likely that 
   the input dimension can be substantially reduced via input 
   elimination and/or combination. In the latter case, the  
   contribution of individual inputs can be sorted out later.

[Greg Heath]

Sometimes in high dimensional cases it is very fruitful to see how much of the target variance can be modeled by linear and quadratic classifiers.

[Jocelyn]

If I want to collect the Rsq of training, validation and testing sets for each Ntrials loop, what are the matlab codes?

Thanks Greg for the explanation. I can get the whole picture of it now. However, my data has been gone through the convergent and discriminant validity tests and redundant variables have been eliminated using structural equation modeling. So, I think there must be lack of data points to obtain better result. I decided to collect more data to run the neural network. Is there any formula or equation to calculate the minimum number of data set with 48 inputs and 5 outputs for neural network? Or just use the below equations?

Ntrn = N - 2*round(0.15*N)

Ntrneq = Ntrn*O

Nw = (I+1)*H+(H+1)*O

Hub = (0.7*N-O)/(I+O+1)

[Greg Heath]

For no overfitting

 Ntrneq >= Nw

Which leads to

 Hub = (Ntrneq - O)/(I + O + 1)
     = (Ntrn*O - O)/(I + O +1)
     ~ (0.7*N*O - O )/ (I + O + 1)

Hope this helps.

Greg