Hightly imbalanced class weights

I’m building a semantic segmentation network, which contains 8 different classes. The occurence of elements for each of these classes vary greatly, so i’ve aimed to balance the weights associated with each class. Since this is a common problem, I’ve followed some examples proposed by the MATLAB help center, such as:

https://www.mathworks.com/help/deeplearning/ug/sequence-classification-using-inverse-frequency-class-weights.html?utm_source=chatgpt.com and

https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-dilated-convolutions.html ,

where the inverse of the frequency for each class is used.

I must specify that one of these classes is hightly represented (represents maybe around 90% of the samples) and it is the least important one, as I want my network to focus on the other 7 (identify them correctly).

In my case, to calculate my class weights, I tried the following (freq is the count of elements in the data for each subclasses):

weights = sum(freq) ./ freq;
norm_weights = weights ./ max(weights);

The resulting weight vector for each class is

norm_weights = [0.0001 0.0199 0.0380 0.1118 1.0000 0.0995 0.0576 0.0464]

where norm_weights(1) corresponds to my hightly represented class and norm_weights(5) corresponds to the least one.

When I train my network (with a smaller dataset than what I will train it with once this problem will be resolved), my training accuracy seems stagnant, my validation accuracy looks a bit unstable, and the training loss doesnt go down much. I also want to specify that I'm trying to reproduce results, where the research paper mentions that they've calculated the weight coefficients of the classes using the inverse of the population sample sizes, and that they range between 0 and 1, so unless there is a mistake elsewhere in my code, this should be possible.

What I would like to know is, are my training results normal considering there were only 300 iterations in my training***, or is there a similar but better way to calculate my class weights to avoid the unstable training results that I seem to be observing?

***on the research paper i'm trying to reproduce, it took around 2000 iterations to reach a training accuracy of 90%, while my training accuracy began at ~10% and finished at 11-12% after 300 iterations

Any help is greatly appreciated, thank you for your time!

[Meg Noah]

Is it possible to add more samples by adding small amounts of noise to the underrepresented ones?

[Ève]

@Meg Noah are you referring to data augmentation? if so, i've created a function that augments my data which is in a combined datastore, since the imageDataAugmenter and augmentedImageDatastore don't work on this type of datastore. Also, when i used too many geometric transformations in my function, it didn't work.

(I realise that i wasn't quite clear, it isn't that most of my samples belong to the overrepresented class and the rest belong to the other classes. Rather, on each of my images, around 90%+ of the pixels belong to the overrepresented class and the other pixels belong to some of the other classes. )

I think that too many random geometric transformations (such as shear, scale, translation) probably resulted in the small fraction of the pixels belonging to the underrepresented classes getting even smaller or disappearing (for example, too big of a translation could result in the small portion of the image containing rare classes to be framed out). So, currently, the only transformations I have are (this is a small portion of my augmentation function):

tform = randomAffine2d(XReflection=true,YReflection=true,Scale=[1 1.1]); % generating random transformations

[Meg Noah]

@Ève Yes. Would you like to share a few examples of your underrepresented data and then maybe I can write a script to create some data augmentation for your truth library.

[Ève]

thats super kind! although i wouldn't know how to share it, honestly, because my input data is satellite data that i've processed and then resized to 400x545x3 tensors (and every matching truth data is of size 400x545, with each pixel now containig a number from 1:8 representing a class)... I do still think that the problem is related to the way i calculate my weights but I can send here the augmentation function that i've created and if you understand it, you can tell me what you think:

function [dataOut] = augment(data)
% The AUGMENTED DATASTORE function does data augmentation on a combined
% datastore, allowing both the spectral and «truth»/vfm data to be
% augmented. The goal is to reproduce what the imageDataAugmenter and augmentedImageDatastore
% functions do on image datastores, but on combined datastores.
% Following examples from [https://www.mathworks.com/help/deeplearning/ug/image-to-image-regression-using-deep-learning.html],
% [https://www.mathworks.com/help/images/ref/randomaffine2d.html?searchHighlight=randomAffine2d&s_tid=srchtitle_support_results_1_randomAffine2d]
% and [https://www.mathworks.com/help/images/ref/imwarp.html?searchHighlight=imwarp&s_tid=srchtitle_support_results_1_imwarp].

    dataOut = cell(size(data));

    for idx = 1:size(data,1) % my minibatchsize is currently 1 but could be more
        
        tform = randomAffine2d(XReflection=true,YReflection=true,Scale=[1 1.1]); % generating random transformations

        X = data{idx,1}; % spectral/input data
        Y = data{idx,2}; % truth data

        targetSizeX = [size(X,1), size(X,2)];
        targetSizeY = [size(Y,1), size(Y,2)];

        routX = affineOutputView(size(X), tform, "BoundsStyle", "CenterOutput"); % affineOutputView takes an image of size X, returns a spatial
        routY = affineOutputView(size(Y), tform, "BoundsStyle", "CenterOutput"); % referencing object for imwarp to use

        % transformation of input and output/truth data
        X_aug = imwarp(X, tform, 'OutputView', routX, 'InterpolationMethod', 'linear'); % tform will apply the same transformations to each of the 3 channels
        Y_aug = imwarp(Y, tform, 'OutputView', routY, 'InterpolationMethod', 'nearest'); 

        % making sure that the output size is [400x545]
        X_aug = imresize(X_aug, targetSizeX, 'bilinear'); % resizing applied to all the three channels
        Y_aug = imresize(Y_aug, targetSizeY, 'nearest');

        dataOut(idx,:) = {X_aug, Y_aug}; % cell array

    end
end

% POSSIBLE TRANSFORMATIONS:
% aug = imagedataAugmenter( ...
%     RandXReflection=1, ...
%     RandRotation=[0 360], ...
%     RandScale=[0.5 4], ...
%     RandYShear=[-10 15], ...
%     RandXTranslation=[-50 50], ...
%     RandyTranslation=[-30 30]);

[Meg Noah]

@Ève

Are these 3-band or 3-color satellite Earth image windows/chips? If so, are the data from someplace like https://earthexplorer.usgs.gov/ - if so, which satellite?

[Ève]

the data is from https://search.earthdata.nasa.gov/search/granules?p=C2667982885-LARC_ASDC (it's CALIOP) and the 3 refers to 3 different spectral quantities derived from the satellite data

Answer 1

[saish]

Hey Eve,

Yes, it is quite normal for training to be slow and accuracy to be low after only 300 iterations, especially with a highly imbalanced dataset. But this is not necessarily true and will depend on the results obtained on 2000 or more iterations and when you use the entire dataset.

In this Accuracy metrics can be misleading in imbalanced settings, since the network can just predict the majority class and still get high accuracy. So I think main issue here is to get the weights in normalized range.

It is better to use the any of the following and check the results:

1. Log inverse frequency to reduce the difference between the class weights.

weights = log(sum(freq) ./ freq);

norm_weights = weights ./ max(weights);

2. Median frequency balancing which is most commonly used for image segmentation tasks.

median_freq = median(freq);

weights = median_freq ./ freq;

norm_weights = weights ./ max(weights);

3. Focal loss, which down-weights majority class examples and focuses learning on minority class.

https://www.mathworks.com/help/releases/R2025a/vision/ref/nnet.cnn.layer.focallosslayer.html

As I can see in the comments, you have used transformed the data, which worsen up the issue. Instead of random transformations, consider augmenting only those image patches that contain a significant portion of rare classes. Apply augmentations to these patches and then reinsert them into the dataset, or use them as additional training samples. This ensures that rare classes are not lost during augmentation.

[Ève]

Thank you for your response! I will try to train it with my imbalanced weight values on my hole dataset and if it doesn't work, i'll try one of your suggestions. Currently i'm having problems with my server so this is blocking my progress