If ntstool use test set values to predict further values?
1 visualizzazione (ultimi 30 giorni)
Mostra commenti meno recenti
Jan Kostrzewa
il 30 Ago 2015
Commentato: Jan Kostrzewa
il 18 Set 2015
Hi !
I use Matlab ntstool toolbox to predict time series forecasting. Ntstool gives me suspiciously good results. For example for load ice_data and train,valid,test ratio 0.55/0.15/0.3 show me MSE equal to (more or less) 0.015 when I try to forecast using Neural Network manually I get MSE qual to (more or less) 2.45. Similarly using ARIMA method I MSE 1.3. Error of ARIMA depends of course of parameters of the model but I would like to present order of magnitude.
On this background results returned by ntstool look very suspicious. This is why I would like to ask the question if ntstool use test set values to predict further values and calculate MSE? I mean.. do ntstool predict values that way that:
1. Use neural network to predict value, then put this value to neural network as an input, then predict another value, put predicted value to neural network as an input and repeat that process till the end. or
2. Use neural network to predict value, then put as an input CORRESPONING value from TEST SET to neural network as an input, then predict another value, put CORRESPONING value from TEST SET to neural network as an input and repeat that process.
In first case error accumulate from first prediction to the last otherwise in second case input is updated to correct value on every step. Which method use by default nstool ?!
Thanks in advance for replying :) ! Janek
0 Commenti
Risposta accettata
Greg Heath
il 30 Ago 2015
Since MSE is scale dependent, it is better to report the normalized value NMSE and or the coefficient of determination (AKA Rsquare). See a statistic reference e.g., Wikipedia for more clarity
Average Target Variance (Minimum MSE when the output is constant)
MSE00 = mean(var(target',1))
Normalized MSE
NMSE = MSE/MSE00
Coefficient of variation (Rsquared), the fraction of MSE00 that is modeled by the net.
Rsq = 1 - NMSE
I think you are way off course. Try the command line approach.
See the documentation examples
help narnet
doc narnet
Then search ANSWERS and the NEWSGROUP for my tutorial posts. Start with search words
greg narnet
greg narnet tutorial
Hope this helps.
Thank you for formally accepting my answer
Greg
2 Commenti
Più risposte (0)
Vedere anche
Categorie
Scopri di più su Deep Learning Toolbox in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!