How to import data from a non-symmetric .txt file.

We have the following data file:

1

2 3

5 6 7

How can I import that into MATLAB easily, so the empty block become zero? Like this:

1 0 0

2 3 0

4 5 6

load does not work and importdata gives:

1 NaN

2 3

4 5

6 NaN

Answer 1

[Cam Salzberger]

Hello Jeppe,

If you give an explicit format to use, the data read functions will try to stick to it as best as possible. My personal preferred data read function is readtable. Just tell it not to assume there are header lines (which is probably why it's skipping the "1" by default), and read using your desired format:

t = readtable('datafile.txt','ReadVariableNames',false,'HeaderLines',0,'Format','%f %f %f');

-Cam

[Jeppe Sørensen]

Thanks for your answer. Your suggestion works for the data presented above. In my real life case however, the file is much bigger and there need another way to represent it than '%f, %f, %f'. I would need about 100 %f for it to work. An example is attached to this comment. There must be some other way to do it, which if more convenient for my case. I hope I have been clear

[Walter Roberson]

repmat('%f', 1, 100)

[Cam Salzberger]

What many of the built-in file parsing functions do is go into the file and try to determine the structure of it from the first few lines. Then it tries to read the whole file using that structure. In the case of failure, it may try a couple of other ways as well.

Since this depends on the data being consistent throughout the file, which yours isn't, the data isn't being read correctly. You could write your own code that checks every line of the file to see which is the longest, and then parses the whole file in that way. It would essentially double the amount of time needed to read the file, but it's doable.

Or, if you know your expected file structure exactly, then just programmatically create your format string, as Walter suggested. Although you'd probably want a space or tab in there:

formatStr = repmat('%f\t', 1, 100);

More ideally, you would modify whatever it is that is creating the files to put something in the missing areas, rather than leaving them blank. Then you can use the TreatAsEmpty specifier to mark those values uniformly. But this isn't always an option.

[Walter Roberson]

readtable() delegates to textscan() which does not need spaces between the % format elements.

[Jeppe Sørensen]

By converting NaN to zeros after importing, this works excellently for me. Thank you!

Answer 2

[Cedric]

If you are stuck with usual tools, try this:

 content = fileread( 'virusDat.txt' ) ;

 % - Split in rows, remove extra empty ones.
 data = strsplit( content, '\n' ) ;
 while isempty( data{end} )
    data(end) = [] ;
 end

 % - Convert to numeric and get max number of columns.
 data  = cellfun( @(x) sscanf( x, '%d' ), data, 'UniformOutput', false ) ;
 nCols = max( cellfun( @numel, data )) ;

 % - Define padding function and pad with e.g. NaNs.
 pad_fun = @(x) [reshape( x, 1, [] ), repelem( NaN, 1, nCols-numel( x ))] ;
 data    = cellfun( pad_fun, data, 'UniformOutput', false  ) ;

 % - Concatenate padded rows.
 data = vertcat( data{:} ) ;

EDIT : replaced

 nCols = numel( data{end} ) ;

with

 nCols = max( cellfun( @numel, data )) ;

in case you don't always have this pyramidal structure.

[Jeppe Sørensen]

This works as intended as well, and is a very good answer. Thank you for this. Cam Salzburgs answer gets accepted, however, as it is a bit simpler.

[Cedric]

My pleasure! It was just to give you an alternate approach for dealing with "unorthodox" files. Note that it doesn't require that you hard-code 24 for defining a format string and that it is faster than READTABLE, essentially because of the detection of options in READTABLE for importing that takes time.