Read non-tabular text file using datastore or mapreduce

How can I read non-tabular text file using datastore or mapreduce? Thanks

Answer 1

[yashwanth annapureddy]

Hi Darek,

As you might already know datastore is primarily meant for large tabular data sets. Non-tabular/blocked reading is something we want to look at also in the near future.

However you should be able to play with the TextscanFormats to read the data you care about just as you would using textscan for reading non-tabular data. For example, as you may already know, you can set the TextscanFormats as '%q' and read in a table with one variable name with the block of text for further processing.

I could probably help more, if you provide a sample text file?

hope this helps.

Answer 2

[Darek]

I tried this approach, but Matlab is poor at managing large files and runs out of memory quickly. This is my sample text file. Thanks

##################################

0|FRM|FM MLLD FILE|20141001|

1|FRM|A4014X|20051201|

5|FRM|A4014X000005|8K0ER0|C|R|SF|O|01|WA

5|FRM|A4014X000006|8K0ER0|P|R|SF|O|01|WA

8|FRM|A4014X|000000002|

90|FRM|FM MLLD FILE|20141001|

##################################

[yashwanth annapureddy]

How about

numRows = value that is suitable for your memory requirements;

>> ds = datastore('file.txt', 'ReadV', false, 'Delimiter', '', 'RowsPerRead', numRows);

while hasdata(ds)

t = read(ds); % post process the data

end

I guess you should be able to control RowsPerRead to not run out of memory, unless I am missing something.

thanks

[Darek]

This approach with mapreduce works, but it's extremely SLOW. Just to count records in small file (3GB, 8 million rows) it takes 20+ minutes on my new PC. Similar project can be done using other tools (R, Powershell, Python) in seconds. Thanks for your help.

[yashwanth annapureddy]

By records if you mean lines in the file, datastore is probably not the right tool to count the number of records in a file. Lower or even system level functions like the ones mentioned in this thread might be useful for this kind of analysis.