GRAD-CAM++のコード作成時のエラー

GRAD-CAMより高解像度の画像の特徴量抽出が可能になるだろうとのことで、下記のようなGRAD-CAM＋＋のコードを作成しましたが、エラーが発生しました。function中の2階微分３階微分のdlgradientのトレースがうまくいっていないようなのです。１階微分のgradientsはトレースできているようです。色々ためしましたが原因と対策が分かりません。どなたかご教授頂けないでしょうか？よろしくお願いいたします。以下コードと発生したエラーです。よろしくお願いいたします。

（コード）

%%%%サポートベクターマシンによる分類 %%%%%%%

%%%%%転移学習 ネットワークを選択可能　GRAD-CAM++で可視化

%%%%% 2024年10月27日 2階微分と3階微分の計算でエラー発生中。可能性あり

%データストアの準備

imds2 = imageDatastore("C:\Users\user\MATLAB Drive\Examples\Ultrasonic_analysis\mltest2","IncludeSubfolders",true,"FileExtensions",".png","LabelSource","foldernames");

%動画N増し（５つずつ）

%imds2 = imageDatastore("C:\Users\user\MATLAB Drive\Examples\Ultrasonic_analysis\actual_videono_normalized_cropped","IncludeSubfolders",true,"FileExtensions",".png","LabelSource","foldernames");

%加工した動画

%imds2 = imageDatastore("C:\Users\user\MATLAB Drive\Examples\Ultrasonic_analysis\actual_videono_normalized_cropped3","IncludeSubfolders",true,"FileExtensions",".png","LabelSource","foldernames");

%加工した動画(振動子除去、正規化など）

%imds3 = imageDatastore("C:\Users\user\MATLAB Drive\Examples\Ultrasonic_analysis\actual_videono_normalized_cropped3","IncludeSubfolders",true,"FileExtensions",".png","LabelSource","foldernames");

%ネットワークの準備

%net=googlenet;

%net=efficientnetb0;

%net=inceptionresnetv2;

net=nasnetlarge;

%訓練データとテストデータに分割

[imds2Train,imds2Validation]=splitEachLabel(imds2,0.7,'randomized');

analyzeNetwork(net)

net.Layers

imageAugmenter = imageDataAugmenter( ...

'RandRotation',[-1,1], ...

'RandXTranslation',[-50 50], ...

'RandYTranslation',[50 50], ...

'RandScale',[0.95,1.05], ...

'FillValue',0);

%augdsTrain2= augmentedImageDatastore([224 224 3],imds2Train,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

%augdsValidation2=augmentedImageDatastore([224 224 3],imds2Validation,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

%augdsTrain2= augmentedImageDatastore([299 299 3],imds2Train,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

%augdsValidation2=augmentedImageDatastore([299 299 3],imds2Validation,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

augdsTrain2= augmentedImageDatastore([331 331 3],imds2Train,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

augdsValidation2=augmentedImageDatastore([331 331 3],imds2Validation,'DataAugmentation',imageAugmenter,'ColorPreprocessing','gray2rgb');

%%%データの表示と確認

minibatch_Train2 = preview(augdsTrain2);

figure(10)

imshow(imtile(minibatch_Train2.input));

%%%%%%%%%% 機械学習による分類 %%%%%%%%%%

%%%% 特徴の抽出

% 特徴マップを取得する層の名前

%layer = 'activation_295';

featureLayer = 'global_average_pooling2d_2';

%%%特徴量抽出

featuresTrain = activations(net,augdsTrain2,featureLayer,OutputAs="rows");

featuresTest = activations(net,augdsValidation2,featureLayer,OutputAs="rows");

%%% ラベルの取得

TTrain = imds2Train.Labels;

TTest = imds2Validation.Labels;

%%% SVMに当てはめ

rng default

mdl = fitcecoc(featuresTrain,TTrain,'OptimizeHyperparameters','auto',...;

'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName',...;

'expected-improvement-plus'));

%%% テストイメージの分類

Ypreds2 = predict(mdl,featuresTest);

YValidation2=imds2Validation.Labels;

figure(1);

confusionchart(YValidation2,Ypreds2,'RowSummary','row-normalized');

accuracy = mean(Ypreds2 == imds2Validation.Labels);

%%%%% 離型剤の場合 %%%%%

% 画像の読み込みと前処理

imageFiles = ["normalized_frame_vra_12.png", "normalized_frame_vra82_12.png", ...

"normalized_frame_vra90_12.png", "normalized_frame_vra117_12.png", ...

"normalized_frame_vra214_12.png"];

imageDir = "C:\Users\user\MATLAB Drive\Examples\Ultrasonic_analysis\actual_videono_normalized_cropped3\releaseagent\";

inputSize = net.Layers(1).InputSize(1:2);

images = cell(1, numel(imageFiles));

for i = 1:numel(imageFiles)

img = imread(fullfile(imageDir, imageFiles(i)));

img = imresize(img, inputSize);

images{i} = im2double(img);

end

% 画像を4次元配列に変換

dlImages = cat(4, images{:});

dlImages = dlarray(dlImages, 'SSCB');

% ネットワークの層名を一覧表示

layerNames = arrayfun(@(x) x.Name, net.Layers, 'UniformOutput', false);

disp(layerNames);

% Grad-CAMの計算

layerName = 'global_average_pooling2d_2'; % 例として使用する層

classIdx = 1; % 解析したいクラスのインデックス

% Grad-CAM++の計算

gradCAMMaps = cell(1, numel(images));

for i = 1:numel(images)

img = images{i};

dlImg = dlarray(img, 'SSCB');

% ネットワークをLayerGraphに変換

lgraph = layerGraph(net);

% 出力層を削除

lgraph = removeLayers(lgraph, 'ClassificationLayer_predictions');

% ネットワークをdlnetworkに変換

dlnet = dlnetwork(lgraph);

% Grad-CAMの計算

[scoreMap, featureLayer, reductionLayer] = gradCAM(dlnet, dlImg, classIdx);

% Grad-CAM++の重みを計算

[gradients, scores, secondOrderGradients, thirdOrderGradients] = dlfeval(@modelGradients, dlnet, dlImg, classIdx);

%[gradients, scores] = dlfeval(@modelGradients, dlnet, dlImg, classIdx);

% 勾配をリサイズして featureMap のサイズに合わせる

resizedGradients = gradients;

% 重みの計算

%alpha = gradients .^ 2;

%beta = gradients .^ 3;

alpha = secondOrderGradients .^ 2;

beta = thirdOrderGradients .^ 3;

weights = sum(alpha .* beta, [1 2]) ./ (2 * alpha .* beta + 1e-7);

% クラス活性化マップの生成

camMap = sum(scoreMap .* weights, 3);

% マップのリサイズと正規化

camMap = max(camMap, 0);

camMap = camMap / max(camMap(:));

gradCAMMaps{i} = extractdata(camMap);

figure(2)

subplot(1,5,i); %release agent

imshow(img);

hold on

imagesc(gradCAMMaps{i}, 'AlphaData', 0.5);

colorbar

colormap jet

title(sprintf('Grad-CAM++ of 離型剤 image %d', i));

end

function [gradients, scores, secondOrderGradients, thirdOrderGradients] = modelGradients(dlnet, dlImg, classIdx)

% 特定のクラスに対するスコアを計算

scores = predict(dlnet, dlImg);

score = scores(classIdx);

% 損失関数を定義

loss = -sum(score, 'all');

% 勾配を計算

gradients = dlgradient(loss, dlImg);

% 2階微分の計算

gradientsSum = sum(gradients, 'all');

secondOrderGradients = dlgradient(gradientsSum, dlImg);

% 3階微分の計算

secondOrderGradientsSum = sum(secondOrderGradients, 'all');

thirdOrderGradients = dlgradient(secondOrderGradientsSum, dlImg);

end　　

（エラー）

次を使用中のエラー: dlarray/dlgradient (行 105)

微分する値は、トレースされていません。これは、実数のトレース付き dlarray スカラーでなければなりませ

ん。dlfeval によって呼び出される関数内で dlgradient を使用して変数をトレースしてください。

エラー: svm_nasnetlarge_actualvideo_gradcam_plusplus__20241027m>modelGradients (行

175)

secondOrderGradients = dlgradient(gradientsSum, dlImg);

エラー: deep.internal.dlfeval (行 17)

[varargout{1:nargout}] = fun(x{:});

エラー: deep.internal.dlfevalWithNestingCheck (行 19)

[varargout{1:nargout}] = deep.internal.dlfeval(fun,varargin{:});

エラー: dlfeval (行 31)

[varargout{1:nargout}] = deep.internal.dlfevalWithNestingCheck(fun,varargin{:});

エラー: svm_nasnetlarge_actualvideo_gradcam_plusplus__20241027m (行 130)

[gradients, scores, secondOrderGradients, thirdOrderGradients] =

dlfeval(@modelGradients, dlnet, dlImg, classIdx);

Answer 1

[Gayathri]

Hi @敬,

I understand that you are trying to develop a code for GradCAM++ but facing the error

“Error using dlarray/dlgradient (line 105) The values to differentiate are not traced. They must be real, scalar dlarray with tracing. Use dlgradient within a function called by dlfeval to trace variables.”

This is because we have to keep each differentiable in a different function and then call it using the “dlfeval” function. Please refer to the below code for the same.

function [gradients, scores] = modelGradients(dlnet, dlImg, classIdx)
    scores = predict(dlnet, dlImg);
    score = scores(classIdx);
    loss = -sum(score, 'all');
    gradients = dlgradient(loss, dlImg); 
end

function secondOrderGradients= modelGradients2(gradients, dlImg)
    secondOrderGradients = dlgradient(sum(gradients,"all"), dlImg);
end

function thirdOrderGradients= modelGradients3(secondOrderGradients, dlImg)
    thirdOrderGradients = dlgradient(sum(secondOrderGradients,"all"), dlImg);
end

To call these functions use the following lines of codes.

[gradients, scores] = dlfeval(@modelGradients, net, dlImg, label);
secondOrderGradients = dlfeval(@modelGradients2, gradients,dlImg);
thirdOrderGradients = dlfeval(@modelGradients3, secondOrderGradients,dlImg);

The above-mentioned changes would help resolve the issue of gradients not being traced.

For more information on “dlfeval” and “dlgradient” please refer to the following links.

• https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/dlfeval.html
• https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/dlarray.dlgradient.html

Hope you find this information helpful.

[敬]

Thank you for your kind and insightful answer. I'll definitely try it out.