Data replication Neural Networks Matlab

Hello world.! I have recently been studying neural networks, so I may ask something obvious, but I figured out that when I replicate my inputs and outputs and then train the network for pattern recognition,it has far more accuracy than with the original data. I thought of that in order to replicate some of the extreme values I have. Can that make my network overfit? Thank you everyone

Answer 1

[Greg Heath]

1. I don't understand your question.

2. a. OVERFITTING means there are more unknown weights, Nw, than independent training equations, Ntrneq ( i.e., Nw > Ntrneq).

b. OVERTRAINING an overfit net CAN LEAD to loss of performance on NONTRAINING data.

3. There are several remedies to prevent OVERTRAINING AN OVERFIT NET. So, in general, overfitting need not be disastrous.

4. Methods for preventing loss of generalization via overtraining an overfit net

 a. Do not overfit: Nw < Ntrneq. Preferrably, 
Ntrneq >> Nw which yields design Stability and 
robustness w.r.t. noise and measurement error.

 For example:
    i. Increase the number of training examples
    ii. Reduce the number of hidden nodes 
 b. Use VALIDATION STOPPING to prevent overtraining
 c. Use the BAYESIAN REGULARIZATION 
training function TRAINBR with MSEREG 
    as a default.
 d. Replace the default performance function 
MSE with the regularized 
    modification MSEREG

Hope this helps.

Thank you for formally accepting my answer

Greg

[Andreas]

Dear Dr. Heath, in order to increase the number of training examples I decided to duplicate my data, as I have noticed that it increased the accuracy. Is this valid? Can it lead to overtrainig? Should I randomly change the duplicated data in order to look "new" but not totally different? I have followed your posts and your advice has been very helpful and productive. It is always an honour. Thank you.

[Greg Heath]

If you are doing it correctly, duplicating the data should basically yield the same result.

If you ask " What is correctly?"

I'll answer:

 " Do duplicated training examples remain 
training examples? 
Similarly for the val and test examples."

I have the sneaky feeling that some of the data is being used as both training and nontraining data:

i.e.,

The equivalent of using 'dividetrain'!

Obviously, you need to duplicate each of the division subsets separately.

Sometimes when I don't have enough original data, I use the following:

 1. Duplicate each subset individually. 
   a. Loop over multiple trials of
      adding random noise to the  
      duplicated data for training, 
      validation and testing.
   b. Rank the designs based on  
      val performance.
   c. Compare the test data results with 
the performance of the original data.

When I have enough data I use a double loop approach (I have posted zillions of examples)

Ntrials = 10
Hmin = ...
dH = ...
Hmax = ...

%Outer Loop over # of hidden nodes:

j=0
rng('defaullt')
for h = Hmin:dH:Hmax 
    j=j+1
   net = ...
   blah, blah, blah
%Inner Loop over different random 
   initial weights

     for i=1:Ntrials
         net = configure(...)
         blah, blah
     end
end

Hope this helps.

Greg

[Andreas]

Dear Dr. Heath,

so, to see if I got it right, I don't replicate the whole data set and then start training, I first separate the data sets and then I replicate each of them individually

e.g 
valInd=repmat(train.val,[100,1] etc etc)

and add some random noise e.g

for i=1:5
valInd(randi[1,10])=randi[1,5]   
end

and then I start the training procedure.

Is that correct? Also,could data balancing procedures (e.g oversampling through SMOTE) add the necessary noise?

Best regards,

Andreas Kampianakis

[Greg Heath]

No.

You add the random noise to the replicated data. Just make sure that the resulting signal to noise ratio is sufficiently high.

Obviously, one good way to approach the problem is as a function of RNG state (which determines the initial random weights) and SNR, given the number of hidden nodes.

However, I have done the reverse, i.e., trained and validated with noisy duplicated data and then used the original data for testing. Again results are presented in terms of the parameter SNR.

Hope this helps.

Greg

[Andreas]

Dear Dr. Heath,

thank you very much for your time.

Best regards,

Andreas Kampianakis