Azzera filtri
Azzera filtri

Problem with processing large data set

2 visualizzazioni (ultimi 30 giorni)
I want to train a BiLSTM-CNN (Bidirectional Long Short-Term Memory Convolutional Neural Network) with 203457 timeseries. I am trying to store them in a table with one line and 203457 columns, each cell containing an array with 100,000 values, but my server freezes whenever I run my code. I don’t get any error messages so I am not sure what the problem is and how to fix it. Do you have any idea why this happens and how to solve this?
Thank you!

Risposte (1)

Eric Sofen
Eric Sofen il 14 Dic 2020
I'm not sure of the BiLSTM-CNN input requirements, but for starters you're probably better off storing those timeseries as rows in a table rather than variables (columns). You've got some 20 billion total values, so you may be hitting memory limits. You may want to look into using a tall variable for out-of-memory computation.
  1 Commento
Andrei-Ieronim Constantinescu
Thank you very much for the response. Meanwhile, I found a forum suggesting that a fileDatastore is good for processing MAT files and big data due to the fact that it is a type of datastore. I will try both options and update here wich one is better.

Accedi per commentare.

Categorie

Scopri di più su Sequence and Numeric Feature Data Workflows in Help Center e File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by