Is it possible to apply upper and lower bounds to predictions in an LSTM?

My predictions, which have been running very well as late, have hit an issue, where they are forecasting well below what should be the minimum possible value, i.e. forecasting at ~200, when the minimum should be 0. Is there anyway of simply applying a constraint/bounds, or indeed using the historical data to apply this?

Answer 1

[Ben]

I believe the default LSTM has outputs bounded in (-1,1) due to the activation functions used.

In any case you can try using activations to add constraints to your model, e.g. a tanhLayer has outputs bounded in (-1,1) and you can add a fixed linear transform to modify that to an arbitrary interval (a,b) as follows:

a = 0; b = 10;

% since tanh(x) is in (-1,1), (tanh(x) + 1)/2 is in (0,1).

% then scale to (0,b-a) and translate to (a,b);

boundToIntervalLayer = functionLayer(@(x) a + (b-a)*(tanh(x)+1)/2);

layers = [

sequenceInputLayer(1)

lstmLayer(1)

fullyConnectedLayer(1)

boundToIntervalLayer];

net = dlnetwork(layers);

% test it out

seqLen = 10;

x = dlarray(randn(1,seqLen),"CT");

y = forward(net,x);

% you can check y is in (a,b)

Alternatively if you use a custom training loop you could add a "soft constraint" by adding a penalty term to the loss function that is large when the network predicts values outside of your bounds.