importing large 4GB+ tab-delimited ascii file makes MATLAB crash
1 visualizzazione (ultimi 30 giorni)
Mostra commenti meno recenti
Sean Phillips
il 17 Mag 2019
Risposto: Sean Phillips
il 18 Mag 2019
i didnt have a problem until now the data files have been scaled up in complexity
i usually use
IMPORT = importdata('components.dat')
in this case components.dat is 4 GB+
it is a table of numerical values consisting of 12 columns and millions of rows
for smaller models this has worked but now MATLAB crashes when i try to import at this size.
I have 96GB of RAM and dual Xeons.
Do i need to change setting in MATLAB or is there better ways to import this data
this is an exmaple of the file
1 0.01 -50.00 0.00 -3950.00 -2.0000E+04 0.000 -4000. -1.409E-15 -2.138E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9900E+04 0.000 -4000. -1.410E-15 -2.193E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9800E+04 0.000 -4000. -1.411E-15 -2.249E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9700E+04 0.000 -4000. -1.412E-15 -2.306E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9600E+04 0.000 -4000. -1.411E-15 -2.364E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9500E+04 0.000 -4000. -1.411E-15 -2.424E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9400E+04 0.000 -4000. -1.409E-15 -2.485E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9300E+04 0.000 -4000. -1.407E-15 -2.547E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9200E+04 0.000 -4000. -1.405E-15 -2.611E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9100E+04 0.000 -4000. -1.401E-15 -2.676E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9000E+04 0.000 -4000. -1.397E-15 -2.742E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8900E+04 0.000 -4000. -1.392E-15 -2.810E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8800E+04 0.000 -4000. -1.386E-15 -2.879E-15 0.00E+00 0.00E+00
it is tab delimited
0 Commenti
Risposta accettata
Più risposte (2)
Steven Lord
il 17 Mag 2019
Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.
per isakson
il 18 Mag 2019
Modificato: per isakson
il 18 Mag 2019
That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?
Anyhow try
>> num = cssm( 'components.dat' );
>> whos num
Name Size Bytes Class Attributes
num 13x12 1248 double
where
function num = cssm( ffs )
fid = fopen( ffs );
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
, 'Delimiter','\t', 'CollectOutput',true );
fclose( fid );
num = cac{1};
end
If that doesn't work, I guess there is some strange stuff in the middle of your text file.
I added
, 'ReturnOnError',false
to the textscan call and the string, "an error", to the text file an run
>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
>>
textscan() found my string "an error" on the row 1005147.
0 Commenti
Vedere anche
Categorie
Scopri di più su Text Data Preparation in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!