How to Create Validation Curve For Neural Networks

I'm trying to determine which value for lambda for my neural network architecture is the best. To do so, I want to create a validation curve (lambda on the x axis and the error on the y axis) using the training set and validation set. I have separate scripts for:

Forward propagation: returns cell array for activation matrices (A matrices). Takes the ϴ matrices and the input matrix X. Back propagation: returns cell array of the partial derivatives of the cost function. Take ϴ matrices, A matrices, Y matrix, and λ as input. Cost function: returns cost. Takes ϴ matrices, an estimates matrix, Y matrix, and λ as input.

I'm confused on how to start the code for the validation curve and what it should look like. I know I need to create a for loop that goes through different values of lambda, but my question is what exactly needs to go in that for loop?

Answer 1

[Rushil]

Hello

To create a validation curve for lambda (the regularization parameter), it is necessary to evaluate how the neural network performs for different lambda values. The typical approach is to train the network for each lambda, then record the training and validation errors (without the regularization term in the cost) for each value.

Given a setup with separate scripts for forward propagation, backpropagation, and cost calculation, the code structure can be organized as follows:

1) Define a range of lambda values

lambda_values = logspace(-4, 2, 10); % say, from 1e-4 to 1e2 
training_errors = zeros(length(lambda_values), 1); 
validation_errors = zeros(length(lambda_values), 1); 

2) For each lambda, train the network and compute errors

for i = 1:length(lambda_values) 

lambda = lambda_values(i); 
 
% initialize weights (Theta)  
Theta = initializeTheta(...); % initialize in your way 
 
for epoch = 1:max_epochs 
    A_train = forwardPropagation(Theta, X_train); % forward pass
 
    gradients = backPropagation(Theta, A_train, Y_train, lambda); % backprop
 
    Theta = updateTheta(Theta, gradients, learning_rate); % weight update
end 
 
% compute training error (no regularization)  
A_train = forwardPropagation(Theta, X_train); 
predictions_train = A_train{end}; 
training_errors(i) = costFunction(Theta, predictions_train, Y_train, 0); 
 
% compute validation error (no regularization) 
A_val = forwardPropagation(Theta, X_val); 
predictions_val = A_val{end}; 
validation_errors(i) = costFunction(Theta, predictions_val, Y_val, 0); 
  
end 

3) Plot the validation curve

figure; 
semilogx(lambda_values, training_errors, '-o', 'DisplayName', 'Training Error'); 
hold on; 
semilogx(lambda_values, validation_errors, '-o', 'DisplayName', 'Validation Error'); 
xlabel('Lambda'); 
ylabel('Error'); 
legend; 
title('Validation Curve: Lambda vs Error'); 
hold off; 

Here is an explanation of what happens inside the for loop:

• Weights (Theta) are initialized.
• The network is trained for the current lambda, using forward and backpropagation scripts and updating weights.
• After training, training and validation errors are computed (with lambda=0 in the cost function for these errors).
• Errors are stored for plotting.

This setup enables observation of how different values of lambda affect the network’s ability to fit the training data and generalize to validation data. The optimal lambda is typically the one with the lowest validation error.

The above makes use of the “semilogx” function, which can be found at the documentation link:

https://mathworks.com/help/matlab/ref/semilogx.html

Hope this helps