Cross-Entropy Loss in Classification Layers

The MATLAB documentation is very clear to say that a classificationLayer computes the cross-entropy loss for problems with "mutually exclusive classes". In general, cross-entropy doesn't require mutually exclusive classes, however, and a training label does not need to be "one-hot" (i.e., a 1 in the true class component, 0 elsewhere), but can be discounted for class imbalance (see Custom Weighted Classification Layer) or can be "fuzzy", where the sum of the training label components is 1, but several components are non-zero.

In the MATLAB documents, they show what appears to be the generalized expression for cross-entropy, but they repeatedly state that the "classificationLayer" is for mutually exclusive classes only. My questions are:

1. Can anyone verify that the classificationLayer is, in fact, coded so that it will not work with fuzzy training labels? i.e., in a 2-class problem we could have a [0.8 , 0.2 ] label that works fine with cross-entropy loss as written in any cross-entropy Equation, including some equations sitting around in MATLAB documentation.

2. If MATLAB's "cross-entropy" in their classificationLayer is really only "mutually exclusive class cross-entropy", then does anyone know how to write a custom classification output layer to implement the real, general cross-entropy that can accept "fuzzy labels"? Examples such as define a Custom Classification Output Layer seem to make some sense when used in conjunction with invoking trainNetwork as "trainedNet = trainNetwork(X,Y,layers,options)", but you would have a 'FuzzyLabels' properties and the fuzzy labels would automatically get passed in from 'Y' when invoking trainNetwork if your constructor function accepted a 'fuzzyLabels' that came from 'Y' in your layers contruct? i.e., one of your layers at the end is 'GeneralCrossEntropyLayer(Y)?

Anyone get what I'm asking here, or is this just rambling nonsense?? :O :)

[kira]

i have the same question. My idea was to create a custom logsig layer, but I don't know if the training function will allow that, and i'm studying how to make that layer...

[Andreas]

Did you ever find a solution to this problem? I know that I should probably just give up on Matlab for CNNs since they cannot provide a solution for this very common situation, but it would be very practical to be able to not having to change coding language in general.

[Greg Heath]

I believe I have covered this topic in both the NEWSGROUP and ANSWERS.

However, I do not have the time to find it. Try searching on variations of

greg crossentropy 65

greg cross entropy 28

crossentropy 97

cross entropy 61

Hope this helps.

Greg

[Andreas]

An answer would consist of a link to a custom output layer that can handle a probability distribution, i.e., with cross-entropy. Have you written such a layer or seen it? Otherwise I think you have not answered the underlying premise of the question.

Answer 1

[J]

I ended up just writing my own custom layers. Another language is an option, but I also sympathize with those who have a large code base already in MATLAB.

My solution worked, but was highy "jerry-rigged" because:

1. MATLAB complained when I wanted to use more generic soft labels with even custom classification layers.(it was like an error check of some sort)
2. Due to (1), I had to use a custom regression layer, but then MATLAB complained that you can't do softmax on regression layer outputs!
3. Due to 2, I could have written a custom softmax layer, but I just 'tricked' MATLAB by writing a custom 'identity layer' (outputs the same values that are input!) and apparently their error checking will allow you to connect regression layer --> custom identity layer --> softmax even if it complains about regression layer --> softmax

FWIW, someone may have addressed this in 2019 (I have not kept up), but the related posts 2018 and earlier did not answer the question. They typically answered a related question, but not this one, or they "answered" this question in a way that would work in theory, but actually doesn't work (i.e., "just write a custom classification layer" type of answer that is ignorant of MATLAB's error checking that won't allow that to be done in a straight-forward way).

Hope this helps???

[Andreas]

I understand, pity that this was the only solution. Are you willing to share these layers?