multi-step prediction in time-series data using 1D CNN model

• I construct a deep learning model based on 1D CNN
• Using data is time-series data (17 channels)
• Input : data at T (17 channels)
• Output : data a T+1 (17 channels)
• It predicts very accurately for one-step, but it cannot predict for multi-step. I think if the model could predict the one-step-ahead data accurately, then the model also can predict two, three, four step ahead. Because predictions are based on previous prediciton which is very accurate.

% model code
net = dlnetwork;

tempNet = [
    sequenceInputLayer(17,"Name","sequenceinput")
    batchNormalizationLayer("Name","batchnorm")
    convolution1dLayer(3,32,"Name","conv1d_4","Padding","same")
    reluLayer("Name","relu_5")];
net = addLayers(net,tempNet);

tempNet = identityLayer("Name","identity");
net = addLayers(net,tempNet);

tempNet = [
    convolution1dLayer(4,32,"Name","conv1d","Padding","same")
    maxPooling1dLayer(3,"Name","maxpool1d","Padding","same")
    reluLayer("Name","relu")];
net = addLayers(net,tempNet);

tempNet = [
    additionLayer(2,"Name","addition")
    reluLayer("Name","relu_6")
    identityLayer("Name","identity_1")];
net = addLayers(net,tempNet);

tempNet = identityLayer("Name","identity_2");
net = addLayers(net,tempNet);

tempNet = [
    convolution1dLayer(32,32,"Name","conv1d_2","Padding","same")
    maxPooling1dLayer(3,"Name","maxpool1d_2","Padding","same")
    reluLayer("Name","relu_2")];
net = addLayers(net,tempNet);

tempNet = [
    additionLayer(2,"Name","addition_2")
    reluLayer("Name","relu_7")
    identityLayer("Name","identity_3")];
net = addLayers(net,tempNet);

tempNet = [
    convolution1dLayer(4,32,"Name","conv1d_1","Padding","same")
    maxPooling1dLayer(3,"Name","maxpool1d_1","Padding","same")
    reluLayer("Name","relu_1")];
net = addLayers(net,tempNet);

tempNet = [
    additionLayer(2,"Name","addition_1")
    reluLayer("Name","relu_8")];
net = addLayers(net,tempNet);

tempNet = [
    convolution1dLayer(32,32,"Name","conv1d_3","Padding","same")
    maxPooling1dLayer(3,"Name","maxpool1d_3","Padding","same")
    reluLayer("Name","relu_3")];
net = addLayers(net,tempNet);

tempNet = [
    additionLayer(2,"Name","addition_3")
    reluLayer("Name","relu_9")];
net = addLayers(net,tempNet);

tempNet = [
    additionLayer(2,"Name","addition_4")
    reluLayer("Name","relu_10")
    fullyConnectedLayer(128,"Name","fc")
    reluLayer("Name","relu_4")
    fullyConnectedLayer(17,"Name","fc_1")];
net = addLayers(net,tempNet);

% 헬퍼 변수 정리
clear tempNet;

net = connectLayers(net,"relu_5","identity");
net = connectLayers(net,"relu_5","identity_2");
net = connectLayers(net,"identity","conv1d");
net = connectLayers(net,"identity","addition/in1");
net = connectLayers(net,"relu","addition/in2");
net = connectLayers(net,"identity_2","conv1d_2");
net = connectLayers(net,"identity_2","addition_2/in1");
net = connectLayers(net,"relu_2","addition_2/in2");
net = connectLayers(net,"identity_1","conv1d_1");
net = connectLayers(net,"identity_1","addition_1/in1");
net = connectLayers(net,"relu_1","addition_1/in2");
net = connectLayers(net,"identity_3","conv1d_3");
net = connectLayers(net,"identity_3","addition_3/in1");
net = connectLayers(net,"relu_3","addition_3/in2");
net = connectLayers(net,"relu_8","addition_4/in1");
net = connectLayers(net,"relu_9","addition_4/in2");
net = initialize(net);

analyzeNetwork(net);

net = dlupdate(@double, net);
net.State = dlupdate(@double, net.State);
net.Learnables = dlupdate(@double, net.Learnables);

x_train = rand(9665,17); % data at T, normalized
y_train = rand(9665,17); % data at T+1 (10 minute is the measurement period), normalized

x_test = rand(2417,17); % data for Test (T)
y_Test  = rand(2417,17); % data for Test (T+1), non normalized

%%Train net

%%
options = trainingOptions('adam', ...
    'MaxEpochs',2000, ...
    'InitialLearnRate',0.01, ...
    'GradientThresholdMethod','l2norm',...
    'L2Regularization',1e-4,...
    'MiniBatchSize',200,...
    'GradientThreshold',1,...
    'Shuffle','every-epoch',...
    'LearnRateSchedule','piecewise',...
    'LearnRateDropPeriod',700,... 
    'VerboseFrequency',200,...
    'LearnRateDropFactor',0.2,...
    'ExecutionEnvironment','auto',...
    'Verbose',true, ...
    'Plots','none');
tic
net1 = trainnet(x_train, y_train, net,'mse' ,options);
toc

%% one step ahead prediction code (very accurate in results)

a = zeros(size(y_Test,2),size(y_Test,1));
tic
for i=1:size(y_Test,1)
    aa = predict(net1,x_test(1:i,:)); %% 17 * 1 형태로 출력됨
    a(:,i) = aa(end,:)';
end
toc
pred_accum = a' .* std1(38:end) + mu(38:end);

%%multi step ahead prediction code (Prediction results is very poor, and I tried several times for predict multi-step)

% first try
net = resetState(net);
a = zeros(2417,17);
aa = [];
[aaa, state] = predict(net1,x_test(1,:))
net1.State = state;
for i=2:2417
    aa = [aa; aaa(i-1,:)];
    [preddummy, state] = predict(net1,aa(1:i-1,:));
    aaa(i,:) = preddummy(end,:);
    net1.State = state;
end

% second try
net = resetState(net);
a = zeros(2417,17);
aa = [];
[aaa, state] = predict(net1,x_test(1,:))
net1.State = state;
for i=2:2417
    aa = [aa; aaa(i-1,:)];
    [preddummy, state] = predict(net1,aa(i-1,:));
    aaa(i,:) = preddummy(end,:);
    net1.State = state;
end

% third try
a = zeros(2417,17);
aa = [];
aaa = predict(net1,x_test(1,:))
for i=2:2417
    aa = [aa; aaa(i-1,:)];
    preddummy = predict(net1,aa(i-1,:));
    aaa(i,:) = preddummy(end,:);
end

Answer 1

[Snehal]

Hello @jingoo lee,

I understand that you’re trying to implement one-step and multi-step predictions using a 1D CNN model in MATLAB R2024b.

In a multi-step approach, predictions from earlier steps are used for future predictions, often leading to error accumulation. Accumulation of small errors over multiple steps leads to a magnified error, which is evident in the lower accuracy of predicted outputs.

You can try to improve the accuracy of your multi-step predictions by implementing one or more of the following strategies:

1. Use Sequence-to-Sequence architectures as they work very well with multi-step predictions
2. Implement mechanisms like Ensemble to mitigate accumulation of errors

You may refer to the following documentation links for more details:

• https://www.mathworks.com/help/deeplearning/ug/sequence-to-sequence-regression-using-deep-learning.html
• https://www.mathworks.com/help/stats/ensemble-algorithms.html

Hope this helps.