Bad classification accuracy... What could be making my pattern recognition network perform so poorly?

Hello Matlab Community. I have run into an issue with my neural network.

My network contains 2000 samples evenly distributed among 4 classes. Each sample contains 100 attributes. The data is best described as particular points in a time sequence (for example, one sample contains 100 data points over 100 seconds--or a 1*100 array). My inputs are a 100*2000 double, and my target is a 4*2000 binary matrix where ones describe the instance of a particular class.

Everything seems to work well, until I get my result.

My confusion matrices have been showing about a 25% accuracy rate.. and it varies every time I retrain. It looks like my neural network is clumping most of my data points into one class in the training phase, which is causing problems down the line. I would also like to note that each time I retrain, a bulk of the training data gets clumped into different classes. Sometimes, it's class 4 (as shown above), but other times it groups everything into class 1,2, or 3.

Here is what my data looks like plotted, each plot represents each of my four classes. I feel like these would be pretty easy to distinguish given their size alone, but that does not seem to be the case.

I have tried a number of things.. changing the training function, number of layers, layer sizes, ect... I'm still resulting in poor classification. Does anyone know what may be causing this? I would also like to note that I am using nnstart to work my way through the network design, then pulling the script from nntraintool. I have it set at the standard 70% train, 15% test, 15% train. Thanks

Edit: My code is below.

% Solve a Pattern Recognition Problem with a Neural Network
% Script generated by Neural Pattern Recognition app
% Created 11-Jul-2017 14:40:12
%
% This script assumes these variables are defined:
%
%   LargeRadar - input data.
%   LargeRadarTarget - target data.

x = LargeRadar;
t = LargeRadarTarget;

% Choose a Training Function
trainFcn = 'trainscg';  % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network
hiddenLayerSize = 15;
net = patternnet(hiddenLayerSize, trainFcn);

% Choose Input and Output Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};

% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;

% Choose a Performance Function
net.performFcn = 'crossentropy';  % Cross-Entropy

% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};

% Train the Network
[net,tr] = train(net,x,t);

% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);

% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)

% View the Network
view(net)

      genFunction(net,'myNeuralNetworkFunction');
      y = myNeuralNetworkFunction(x);
  end

Greg, I have re-uploaded my question with code included. Hope this helps

Answer 1

[Mukul Rao]

Hi,

You can try steps mentioned in the following links to improve neural network performance :

Improve Neural Network Generalization and Avoid Overfitting

How do I improve my neural network performance?

As far as I understand there is no recipe to make a good network, it takes some trial and error. I would also be vary of my dataset, does it make sense in terms of having adequate unique information for my network to learn well?

You could consider using "processpca" to check if there are redundant attributes and reduce the size of the training dataset.

[Greg Heath]

Classes 2 and 4 appear to have completely different scales

Classes one and three seem to be contaminated with outliers.

Sorry, have to go!