t-sne visualization of Vision transformer

How to visualize the features of the vision transfomer after training using t-sne visualizion?

example of vision transformer: https://se.mathworks.com/help/deeplearning/ug/train-vision-transformer-network-for-image-classification.html

and t-sne visualization of CNN: https://se.mathworks.com/help/deeplearning/ug/view-network-behavior-using-tsne.html

I could not apply the same in vision transformer beacuse there are an error during using actication function as follows:

earlyLayerName = "'encoderblock1_mha1";

pool1Activations = activations(net,...

augimdsTrain,earlyLayerName,"OutputAs","rows");

Answer 1

[Garmit Pant]

Hello MAHMOUD EID

From what I gather, you are following the MATLAB examples “Train Vision Transformer Network for Image Classification” and “View Network Behavior Using tsne” to visualise the results of the network activations of a trained vision transformer. You are encountering an error in the process.

The line of code where you are encountering the error is trying to refer to a non-existent layer, assuming you are using the MATLAB default implementation of the “visionTransformer” network. By analysing the network, we can see the layer is named “encoderblock1_mha” and not “encoderblock1_mha1”. Unless you have renamed and replaced the layer, passing the correct layer name shall eliminate the result.

I hope you find the above explanation and suggestions useful!

[MAHMOUD EID]

Hi Garmit,

I think this is not the problem.

I ensured from the layer name . if you choose any layer for visulazion in the activation command :

pool1Activations = activations(net,...

augimdsTrain,earlyLayerName,"OutputAs","rows");

usually, I have this error "Incorrect number or types of inputs or outputs for function activations."

[Garmit Pant]

Hi Mahmoud

Are you using the same data as mentioned and included in the examples?

[MAHMOUD EID]

Hi Garmit

The problem is the same during using any dataset.

you can use the examples mentioned above and try to visualize the transformer after training using the same data of the example, and share the code for the visualiztion .