reading huge data files with repeating pattern

Hi!

I want to read a file as I have attached in the example. Each timestep is independent (there are about 2M timesteps in each file) and I know the number of lines in each file. Can someone please help me with this?

Answer 1

[Star Strider]

This is not an easy file to work with.

One approach —

fidi = fopen('example.txt','rt');
k1 = 1;
while ~feof(fidi)
    for k = 1:9
        first9{k1,k} = fgetl(fidi)                                              % First 9 Lines
    end
    C = textscan(fidi, ['%f%f%s' repmat('%f',1,9)], 'CollectOutput',true);
    C3{k1,:} = [C{:,2}]                                                         % Column #3
    M = cell2mat(C(:,[1 3]));
    if isempty(M)                                                               % Empty Matrix Indicates End-Of-File
        break
    end
    D{k1,:} = M;
    fseek(fidi, 0, 0);
    k1 = k1 + 1
end
fclose(fidi);
C3v = cat(1,C3{:})                                                              % Column #3 (Column Vector)
Out = cell2mat(D);                                                              % Numeric Data Matrix

This reads the entire file, saving the data as a (48x11) double matrix. The third column of the data are saved as ‘C3v’ and here is a (48x1) cell array of strings corresponding to the rows of ‘Out’ The first 9 lines of each section are saved in the ‘first9’ cell array for you to do with them as you wish later. When the code encounters an empty matrix ‘M’ it exits the loop. This prevents an infinite loop if a valid end-of-file indicator is not found.

.

[GUNJAN AUTI]

Thanks a lot! I was trying to loop for each value and created an infinite loop. This will be much faster than my method!

[Star Strider]

As always, my pleasure!

This is the approach I usually use with such files.