Handling memory issues by connecting to database and retrieving only required data
2 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
I have an excel file as large as 20MB with roughly abt 1.5 million rows. I run out of memory while I perform any sort of operation on the data though I am able to read data completely from this huge file. As an alternative would it be a good idea to connect to a database and fetch only the data required for each of my operation (Although this could mean connecting to the database several number of times.) ?
0 Commenti
Risposta accettata
Walter Roberson
il 7 Mar 2011
It seems odd that you would run out of memory on basic operations on such a small file. You are only averaging about 12 characters per row, which would tend to suggest one double precision number per row, which would add up to only 12 megabytes of storage. That's relatively small.
Please check that you are not accidentally using "*" (matrix multiplication) instead of ".*" (element-by-element multiplication) in your routines.
Does even something as simple as
B = A + 0;
cause you to run out of memory? If not then just read the file once and access the sub-ranges you need. If it does run out of memory with something that simple, please use "whos" to figure out what is taking up so much memory. If you are working on the 32 bit Windows version, the command "memstats" can tell you about available memory.
0 Commenti
Più risposte (1)
Vedere anche
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!