Neural network for regression that has mulitple outputs of different order

Hello, I am asking to see if there is any advice to create the neural net that has multiple outputs (for regression) with differnt order(say one of output has order of 10^-5, while the other output has order of 10^5).

Is there any tips for this? Shall I have to make seperate neural nets for each of those outputs?

Thank you very much!

Answer 1

[Varun Sai Alaparthi]

Hello byungchan,

I understand that you want to effectively train a regression model with multiple output heads and the outputs are of different orders.

My advice would be to use weights while adding L2 regression losses. This would normalise the magnitude effect caused due to difference in order.

For example:

Loss = Loss1*W1+Loss2*W2;
% If Loss1 s order > Loss2’s order W1<W2 (example: W1 = 0.001, W2=1) 

Here W1 and W2 can be treated as hyper parameters and can be tuned accordingly for effective training.

Please refer to the following link for implementing custom training loop and loss function with weights for multiple outputs:

TrainNetworkWithMultipleOutputsExample

Please try this and reach out to us for any further issues.

Sincerely

Varun

[byungchan]

Thank you so very much for your kind answers, Varun,

I am trying to implement lstm while using mae instead of mse as a loss function.

While doing so it says dlgradient does not support the lstm.

Is there any ways I could detour this matter?

Again, thank you very much.