Import data and text in huge csv files into matlab and convert to matfiles

3 views (last 30 days)
Jay Ramadugu
Jay Ramadugu on 17 Oct 2014
Edited: per isakson on 20 Oct 2014
Hello All,
Struggling with converting my csv files to matfiles. xlsread and csvread works great with small files but with bigger ones my matlab freezes forever. The file that I need to convert has about 2000 rows and 1200 columns. The first row contains text and the remaining rows contain data. Ultimately I want to be able write the data in each column into the text in the first row of the corresponding column. I also tried using xlsread1 that was shared online but I keep getting errors. I am using Matlab R2010a. Please help if you guys know how to do it.
  1 Comment
per isakson
per isakson on 19 Oct 2014
"remaining rows contain data" &nbsp does that mean pure numerical data, i.e. no datetime?

Sign in to comment.

Answers (3)

Robert Cumming
Robert Cumming on 17 Oct 2014
Use low level routines to read the file.
fopen
fgetl
strread or textscan
Using this you are in complete control.

Image Analyst
Image Analyst on 18 Oct 2014
Honestly, that's not huge. Even at double precision, that's only 19.2 megabytes - much smaller than a run of the mill digital photo. I deal with images that are like 20 GB - a thousand times larger than yours. Maybe try dlmread(). If that or fread() or fgetl(), textscan, sscanf(), etc. don't work then I'll look into it.

per isakson
per isakson on 19 Oct 2014
Edited: per isakson on 20 Oct 2014
Here are two alternative sets of code to read your file. That is, if "remaining rows contain data" &nbsp means pure numerical data, i.e. no text like datetime.
In this case textscan is twice as fast as dmlread
>> cssm
Elapsed time is 1.972024 seconds.
Elapsed time is 4.248113 seconds.
ans =
1
M(1:3,1:5)
ans =
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
where cssm is the script below
hdr = repmat( 'colhead,', [1,1200] );
str = sprintf( '%.1f,', [1:1200] );
str(end) = [];
%%write a sample file
fid = fopen( 'cssm.txt', 'w' );
fprintf( fid, '%s\n', hdr );
for jj = 1 : 2000
fprintf( fid, '%s\n', str );
end
fclose( fid );
tic
fid = fopen( 'cssm.txt', 'r' );
rw1 = fgetl( fid );
num = textscan( fid, '%f', inf, 'Delimiter', ',\n', 'CollectOutput', true );
fclose( fid );
Out = transpose( reshape( num{:}, [1200,2000] ) );
toc
tic
M = dlmread( 'cssm.txt', ',', 1,0 );
toc
all(M(:)==Out(:))
M(1:3,1:5)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by