Problem with processing large data set
1 view (last 30 days)
Show older comments
Andrei-Ieronim Constantinescu on 9 Dec 2020
Commented: Andrei-Ieronim Constantinescu on 17 Dec 2020
I want to train a BiLSTM-CNN (Bidirectional Long Short-Term Memory Convolutional Neural Network) with 203457 timeseries. I am trying to store them in a table with one line and 203457 columns, each cell containing an array with 100,000 values, but my server freezes whenever I run my code. I don’t get any error messages so I am not sure what the problem is and how to fix it. Do you have any idea why this happens and how to solve this?
Eric Sofen on 14 Dec 2020
I'm not sure of the BiLSTM-CNN input requirements, but for starters you're probably better off storing those timeseries as rows in a table rather than variables (columns). You've got some 20 billion total values, so you may be hitting memory limits. You may want to look into using a tall variable for out-of-memory computation.
Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!