Hi Yan, t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used in semantic segmentation to visualize the high-dimensional features corresponding to each class in a lower-dimensional space. Below is a high-level overview of how to apply t-SNE in this context:
- Obtain the feature maps: Perform semantic segmentation on your input image (H x W x C). This outputs a feature map of size H x W x numClasses (in its penultimate layer), where each layer corresponds to the predicted probabilities or scores for a specific class.
- Extract class-specific features: Before applying the argmax across the classes to obtain the final segmentation map, extract the feature map corresponding to each class. Each of these feature maps will have a size of H x W x 1 (or simply H x W).
- Flatten the feature maps: For each class-specific feature map, flatten it into a vector of size H*W x 1. This allows to reshape the spatial information into a format suitable for t-SNE.
- Apply t-SNE: Use the 'tsne' function in MATLAB to reduce the dimensionality of each flattened vector. This outputs low-dimensional embeddings for each class, which can be plotted to visualize the feature distribution.
- Visualize the results: After running t-SNE, plotting the low-dimensional embeddings gives information on how the features for each class are distributed in the reduced space.
Additionaly refer to the MathWorks documentation for more information:
