Error using trainNetwork for network with multiple input

Hi,

I having a CNN that using two input where one of the input using zscore for normalization while the another input using zerocentre normalization (The input data are the same except for normalization setting). However when i tried to train the network using the following command the error occur (Invalid training datastore. For network with multiple inputs, used combined or transformed datastore.). I'm not sure how to insert 2 input into trainNewtork function. Hope someone could help me on this. Thank you very much.

imds = imageDatastore('MerchData', 'IncludeSubfolders',true, 'LabelSource','foldernames'); 
[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7);
augimdsTrain = augmentedImageDatastore(inputSize(1:2),imdsTrain)
augimdsValidation = augmentedImageDatastore(inputSize(1:2),imdsValidation)
% the rest of the code -----
[testnet,traininfo] = trainNetwork(augimdsTrain,lgraph_1,options);

The two input are as followed

input 1 (normalization: zscore)

input 2 (normalization: zerocentre)

Answer 1

[Srivardhan Gadila]

As per the above information, data for both the inputs is same except for normalizations on the input data, so you can create a combined datastore as follows:

cds = combine(imds,imds,labelds);

For more information, you can refer to the following pages on what should be the data format for multiple input networks trainNetwork - Data format & Multiple-Input and Multiple-Output Networks.

Since we are using augmentedImageDatastore and there is a knwon issue on combining the augmentedImageDatastores directly. The workaround is to apply transform on the augmentedImageDatastore and combine it. Also we have to do some other changes to make the workflow possible, as follows:

augimdsTrain.MiniBatchSize = 1;
taugimdsTrain = transform(augimdsTrain,@(x)x{1,1});
labelsTrain = transform(augimdsTrain,@(x){x{1,2}});
% combine the datastores
cds = combine(taugimdsTrain,taugimdsTrain,labelsTrain);

For more information, refer to the following answer on similar issue with other datastore when combining Error Combining two pixelLabelImageDatastores or randomPatchExtractionDatastores.

[Teo]

Hi @Srivardhan Gadila,

thank you for your suggestion. I had tried it but not sure does the code below correctly implemented since i also need to do combination for validation data aside from the train data.

imds = imageDatastore('MerchData', 'IncludeSubfolders',true, 'LabelSource','foldernames'); 
[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7);
figure('Units','normalized','Position',[0.3 0.3 0.4 0.4]);
augimdsTrain = augmentedImageDatastore([224 224 3],imdsTrain)
augimdsValidation = augmentedImageDatastore([224 224 3],imdsValidation)

augimdsTrain.MiniBatchSize =1
taugimdsTrain = transform(augimdsTrain,@(x)x{1,1})
labelsTrain = transform(augimdsTrain,@(x){x{1,2}})
cds = combine(taugimdsTrain,taugimdsTrain,labelsTrain)

augimdsValidation.MiniBatchSize =1
taugimdsValidation = transform(augimdsValidation,@(x)x{1,1})
labelsTrain = transform(augimdsValidation,@(x){x{1,2}})
cds2 = combine(taugimdsValidation,taugimdsValidation,labelsTrain)

miniBatchSize = 10;
valFrequency = floor(numel(augimdsTrain.Files)/miniBatchSize);
options = trainingOptions('sgdm', ...
    'MiniBatchSize',miniBatchSize, ...
    'MaxEpochs',1, ...
    'InitialLearnRate',1e-4, ...
    'Shuffle','every-epoch', ...
    'ValidationData',cds2, ...
    'ValidationFrequency',valFrequency, ...
    'Verbose',true, ...
    'VerboseFrequency', 1, ...
    'Plots','training-progress');

[testnet,traininfo] = trainNetwork(cds,lgraph_2,options);

Hope you are able to help clarify on this.

Thank you very much

[Srivardhan Gadila]

For the validation datastore code, rename the variable name of datastore for validation labels from "labelsTrain" to "labelsValidation". Apart from that I think there should not be any other issues.

[Teo]

Thanks @Srivardhan Gadila for your explanation and pointing out the "labelsValidation" that i miss out. once again thank you very much for your time and help.

[Ali Hajnayeb]

I have applied your suggested approach, but when I added validation option, I saw this message:

"Multi-input networks do not support validation."

[Xuan Yang]

Hi @Teo,

Thanks for your question and code. But I have a question. I don't known how to evaluate the test set on the trained multiple input neural network. Since there is no test set in your source code, I just use the validation set as the test set. I run the following code

[YPred,scores] = classify(testnet,cds2);
accuracy = mean(YPred == labelsValidation);

however, it doesn't work. There are two problems:

1. There is only one sample in YPred.
2. The error information is 'Incorrect use== There is no comparison defined between categorical and matlab.io.datastore.TransformedDatastore arrays.'

I suppose the datastore of cds and cds2 is not suitble for testing. I want to get the final accuracy on the test set. I don't know how to arrange the test samples and test set labels.

How did you solve this problem? Do you have any suggestion? Thank you very much!

[Xuan Yang]

Hi @Srivardhan Gadila,

Thanks for your answer. But I have a question. I don't known how to evaluate the test set on the trained multiple input neural network. Since there is no test set in Teo's source code, I just use the validation set as the test set. I run the following code

[YPred,scores] = classify(testnet,cds2);
accuracy = mean(YPred == labelsValidation);

however, it doesn't work. There are two problems:

1. There is only one sample in YPred.
2. The error information is 'Incorrect use== There is no comparison defined between categorical and matlab.io.datastore.TransformedDatastore arrays.'

I suppose the datastore of cds and cds2 is not suitble for testing. I want to get the final accuracy on the test set. I don't know how to arrange the test samples and test set labels.

How did you solve this problem? Do you have any suggestion? Thank you very much!