getting true positive and true negative rates from a neural network classification

Hi all,

I am trying to figure out how to get the true positive and true negative rates of a neural networks classifier (patternnet). Below is an example using the cancer dataset which is already in the MATLAB 2015a library. I ran the code below in MATLAB 2015a. The problem is that the ROC plot does not agree at all with the true positive and true negative rates returned by the confusion() function of MATLAB. When I run this code, the ROC plot has an optimal operating point at (0.05,0.97) which is 0.97 TPR and 0.95 TNR. But the TPR and TNR returned by the confusion function (Se1 and Sp1 in the code below) are 0 and 0.63, respectively, and those calculated from the confusion matrix (Se2 and Sp2) are 1 and 0, respectively. Can anyone help me understand why there are discrepancies in the TPR and TNR values from the different functions? I don't know which one is correct?

Thanks a ton.

Hadi

[inputs,targets] = cancer_dataset;

targets = targets(1,:);
nG1 = length(find(targets==1))
nG2 = length(find(targets==0))

setdemorandstream(672880951)

hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);

net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio   = 15/100;
net.divideParam.testRatio  = 15/100;
% net.trainParam.showWindow = false;

[net,tr] = train(net,inputs,targets);

outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)

testX = inputs(:,tr.testInd);
testT = targets(:,tr.testInd);

% plotroc(targets,outputs)

testY = net(testX);
figure
plotroc(testT,testY)

[c,cm,ind,per] = confusion(testT,testY);
Se1 = per(1,3)
Sp1 = per(1,4)
Ac1 = 1-c

N=length(testT);
Se2 = cm(2,2)/(cm(2,2)+cm(2,1))
Sp2 = cm(1,1)/(cm(1,1)+cm(1,2))
Ac2 = (cm(1,1)+cm(2,2))/N

Answer 1

[Brendan Hamm]

1. The ROC curve plots the TPR vs FPR.
2. How you are calculating the TPR and TNR is not correct:

TPR = cm(1,1)/sum(cm(:,1)) % Division is  by elements in the predicted true column
TNR = cm(2,2)/sum(cm(:,2)) % Same issues:
FNR = cm(1,2)/sum(cm(:,2))
FPR = cm(2,1)/sum(cm(:,1))

[Brendan Hamm]

Besides it is worth mentioning that what you are seeing as the optimal point on the ROC curve is not what you are using with the confusion function. Since the outputs of your patternet are really just the probability of belonging to the Positive class, confusion is creating a cutoff point (0.5) such that all values above this are assigned to Positive and everything below is negative. However, this would not be the same cutoff level which you are seeing on the roc curve. To determine this use:

[X,Y,T,AUC,OPTROCPT] = perfcurve(testT,testY,1);
cutoff = OPTROCPT(2);
testY = double(testY >= cutoff); % Assigns at optimal ROC value

[Hadi Tadayyon]

Hi Brendan,

Thanks for your reply. In your definition of TPR, you are assuming that c(1,1) is the number of true positives, but it is actually the number of true negatives. According to the definition of TPR = TP/(TP+FN), it should be calculated as cm(2,2)/(cm(2,2)+cm(2,1)), where cm(2,2)=TP and cm(2,1)=FN. The documentation for "confusion" says "cm is a S-by-S confusion matrix, where cm(i,j) is the number of samples whose target is the ith class that was classified as j". In this example, cm was returned as [0 39; 0 66], and the total number of samples in the positive class is 66 and the total number of negative samples is 39. So cm(2,2) has to be the TP. If you run this example, you will find that the plotted ROC curve does not contain the TPR/TNR values computed from the cm, even using your definition (actually TPR returns NAN with your definition since it is division by zero). I don't understand why there is this discrepancy?

[Hadi Tadayyon]

Hi Brendan,

I just saw your second comment. Actually I thought about that too. What you are saying is that after finding the optimal ROC point (TPR, FNR) using perfcurve and calculating the classifier prediction using the optimal cutoff value, the confusion matrix calculated from this prediction should give you the same TPR,FNR values as that from perfcurve, correct? Well, unfortunately this is not the case.

Here is the code to prove it:

[X,Y,T,AUC,OPTROCPT] = perfcurve(testT,testY,1);

cutoff = OPTROCPT(2);
pred = double(testY >= cutoff); % Assigns at optimal ROC value

N=length(testT);
TP = sum(pred.*testT==1);
FP = sum(pred-testT==1);
FN = sum(pred-testT==-1);
TN = N-TP-FP-FN;
TPR = TP/(TP+FN)
FNR = 1-TN/(TN+FP)

Here I calculated the TPR and TNR values manually in order to make sure the terms are clearly defined. I'm pretty sure I've calculated them correctly based on their standard definition. But TPR and FNR values calculated are not the same as OPTROCPT values from perfcurve. Shoudln't these be the same?

[Brendan Hamm]

Firstly, I think you are misunderstanding what I said before. I am not saying that perfcurve will give you the results of the confusion matrix. I am saying that if you want to use the optimal point on the ROC curve, then you need to find the cutoff value of the probabilities using perfcurve. Without this information the confusion function is simply using a cutoff value of 0.5. So, running the section of code here (note I leave out stuff which is not necessary):

[inputs,targets] = cancer_dataset;
targets = targets(1,:);
setdemorandstream(672880951)
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);
[net,tr] = train(net,inputs,targets);
testX = inputs(:,tr.testInd);
testT = targets(:,tr.testInd);
testY = net(testX);
>> testY(1:4)
ans =
    0.9985    0.2932    0.9984    0.4398

We can see that our output for the training set is actually a probability. This is the probability that the cancer is benign (as opposed to malignant). So really we want to provide a cutoff value of probability such that any probability above that will classify as benign and below that cutoff it is classified as malignant. Due to the nature of mis-classifying a malignant tumor as benign we likely want a high value for the cutoff. But confusion does not know what is more important to us, so it just uses a 0.5 cutoff (which I will use for demonstration). So we will then convert these probabilities to be an indicator of class. A 1 will represent benign and a 0 will represent malignant.

    testY = double(testY > 0.5); % Convert to 1s or 0s
    [c,cm,ind,per] = confusion(testT,testY);
    >> cm
    cm =
        32     4
         2    67
    >> per
    per =
        0.0563    0.0588    0.9412    0.9437
        0.0588    0.0563    0.9437    0.9412

As our classes are labeled 0 and 1 respectively (malignant and benign) the first columns are those predicted to be malignant (0) and the second column those predicted to be benign (0). We can verify this:

>> nnz(testY) % Count the 1s in the test predictions 
ans =
    71

This is the same as the number of elements in the second column. (Those that were predicted benign)

>> nnz(testT)
ans =
    69

This is the number of elements in the second row. (Those that actually are benign).

(cont'd)

[Brendan Hamm]

So, we say that malignant is the positive class and benign is the negative class. This statement corresponds to looking at the first row of per. (Note: If we defined this in the opposite manner we would look at the second row).

There are multiple definitions of the True Positive Rate, False Positive Rate, etc. What I mean when I say you are not calculating these correct, I mean you are using an alternative definition (probably the more prevalent today). But, in the documentation for confusion this is expressly mentioned to be:

TPR = #(Predicted positive and is positive)/#(predicted positive). TNR = #(Predicted negative and is negative)/#(predicted negative) The reason for this is it has a straightforward meaning (i.e. TPR is the proportion of the population that will test positive). This representation is useful in conditional probability applications.From these definitions the Sensitivity and Specificity cannot be derived, so they remain what your definition of TPR etc. is.

FNR = cm(1,2)/sum(cm(:,2))
FPR = cm(2,1)/sum(cm(:,1))
TPR = cm(1,1)/sum(cm(:,1))
TNR = cm(2,2)/sum(cm(:,2))

This exactly agrees with the first row of per.

But, the sensitivity and specificity still remain:

Sens = cm(1,1)/sum(cm(1,:))
Spec = cm(2,2)/sum(cm(2,:))

[green]

How to find the fnr,fpr,tpr,tnr for each classes? please sent it to [e-mail]