To read a text file using matlab

Hi,

I have a text files, which contains the numeric data of 1800 rows and 16 columns. Like this, I have a 17280 files. How to read these multiple files faster in matlab efficiently ?....

Awaiting for the earliest reply...

Regards, M. SARFUDEEN

[Jan]

Faster than what? Please post the existing code, otehrwise it is too likely that we suggest functions, which you are using already. We cannot guess how the values are stored. "Numeric data" could mean, that the values are stored in binary format, or as text files, or Excel files, or what ever.

If you read the files line by line and forget a pre-allocation, the allocation of memory might take much more time than the reading of the files. If you apply the standard fscanf method already, the hard disk migth be the bottleneck, such that there is no way for an acceleration but buying a SSD.

So please mention much more details.

Answer 1

[per isakson]

IIRC: fscanf and textscan are the fastest Matlab functions to read and parse text. The others and there are a few are wrappers of these two. I did guess that fscanf would be the fastest. However, it seems I was mistaken.

Below is the result of a test I made now. It shows that textscan is a tiny bit faster than fscanf. The result of this test depends strongly on the performance of the system cache. I have run the test several times with somewhat varying result. However, textscan was always the fastest. I changed the order of the code in the function. That didn't affect the result.

Assumptions:

• text file, csv-file or similar
• only numeric data, no dates or similar
• not fixed width format

Test on R2013a, 64bit, Win7, old PC with 8GB ram and a spinning HD

    >> [ out, S ] = cssm()  % after several runs in a row
    out =
        1.1594    1.0924    1.4397
    S = 
          fscanf: [500000x2 double]
        textscan: [500000x2 double]
        textread: [500000x2 double]

    >> all(all(S.fscanf == S.textscan ) )
    ans =
         1
    >>

where

    function    [ out, S ] = cssm(  )
        ffs = 'h:\m\cssm\two_numerical_columns.txt';
        tic
        fid = fopen( ffs, 'r' );
        m1  = fscanf( fid, '%f', [2,inf] );
        sts = fclose( fid );
        out(1) = toc;
        tic
        fid = fopen( ffs, 'r' );
        m2  = textscan( fid, '%f%f', 'CollectOutput', true );
        sts = fclose( fid );
        out(2) = toc;
        tic
        [a,b] = textread( ffs, '%f%f' ); %#ok<REMFF1>
        out(3) = toc;
        S.fscanf    = transpose( m1 );
        S.textscan  = m2{1};
        S.textread  = [a,b];
    end

and where two_numerical_columns.txt contains

    206.58598 71.5589
    216.87904 78.0681
    219.93297 91.3855
    232.68272 80.8046
    234.24895 78.49
    240.34258 83.9841
    ...