bootstraping for neural network

hi every body

I am using bootstrap sampling with neural network to select the best network architecture. For instance, if I use 30 bootstraping for each network, I would have 30 error for each training and test data related to them. To select the best model, should I use the variance of these errors and select the model with the lowest variance? and should the training and test error be considered separately for bootstrapping? Any other suggestion is welcomed.

[Greg Heath]

There are several different ways to implement training/testing bootstrapping with N data points AND there are several different ways to implement training/validation/testing bootstrapping.

How are you implementing it while avoiding overtraining an over-fit net?

Answer 1

[Greg Heath]

Training errors are biased because the same data is used for creation and evaluation.

Attempts to mitigate the bias involve reducing the number of training equations, Ntrneq, by the number of estimated unknown weights, Nw, to obtain the number of estimation degrees of freedom, Ndof ( see Wikipedia/R-squared), that is used to obtain the adjusted mean-squared-error estimate:

[Ntrn O ] = size(ttrn) % size of the training target matrix

Ntrneq = prod(size(ttrn)) = Ntrn*O % Number of training equations

etrn = ttrn-ytrn; % Training set errors

SSEtrn = ssetrn(etrn) % Sum-squared-errors

MSEtrn = SSEtrn/Ntrneq % Mean-squared-error estimate (biased)

Ndof = Ntrneq-Nw

MSEtrna = SSEtrn/Ndof % "a"djusted mean-squared-error estimate

Although I would only rely on the test set estimates and standard deviations, I would also keep track of the estimates and standard deviations of MSEtrn and MSEtrna.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Mohamad]

Hello Greg One of the methods that I use for avoiding over-fitting is to change the number of hidden nodes in a loop and plot the number of hidden nodes and MSe for the traing and test data set on the same figure. When the MSe for the test set started to increase in comparison with training set I would stop the process. Something like early stopping which is automatically implemented in Matlab. best