Loading Large .txt files

Let's say I have a very very large .txt file with (200millions row & 11 columns= 200m-by-11 matrix). All data are numeric number value (e.g., 10, 100 ,200...) . My file is ~ 20GB

When I load this data in Matlab, the errors occurs: "Out of Memory"

clear;clc;filename = 'test42.txt'; load('test42.txt');P = test42(:,1:3);%get data=coordinate(x,y,z) from set of data "column" at (all row & column 1,2,3)

My PC system: win10-64 bit, RAM 16GB, core-i7, HDD:1TB; SSD 1TB

Actually, I just want to load the data contain only first 3 columns. It mean, the matrix that I want to get is: 200m-by-3 matrix. And with the reduce column, I hope Matlab is able to load data.

Do you know any way to read the whole dataset, or read the reduce data with only first 3 columns? Thanks.

The format of my file is like this.

Answer 1

[per isakson]

You didn't say how much physical memory is in your system.

Matlab provides ways to handle large text files, Large Files and Big Data, but forget that for a moment. It's not a free lunch.

" (e.g., 10, 100 ,200...)" Does that mean positive whole numbers only? If so, do you know the maximum value of the three first columns, respectively?

"When I load this data" What exactly did you do?

The three first columns will take 4.8GB to store as double.

>> 200*1e6*3*8/1e9
ans =
          4.8

But do you need to use double?

Simplest first, convert the three first columns to double and skip the remaining seven columns. Try

%%
fid = fopen( 'c:\whatever\the_huge_text_file.txt', 'r' );
cac = textscan( fid, '%f%f%f%*f%*f%*f%*f%*f%*f%*f', 'HeaderLines',0 );
fclose( fid );

Or use an alternative formatspec, which is a bit easier to read. It says read the first three columns and skip the rest up til a new-line character.

cac = textscan( num2str([1:10]), '%f%f%f%*[^\n]', 'HeaderLines',0 ); 

Next step requires input from you on the numbers in the three first columns.

In response to the edited question

To keep the precision of the numbers in the text file you need to use double. I'm positive that

%%
fid = fopen( 'c:\whatever\test42.txt', 'r' );
cac = textscan( fid, '%f%f%f%*f%*f%*f%*f%*f%*f%*f', 'CollectOutput',true );
fclose( fid );

will load the three first columns without problems (on your system).

[per isakson]

See the addendum to my answer.

[ha ha]

I try with your code, but it did not load the data???Or i did s.thing wrong?

[per isakson]

Fifty years ago a worked at a printshop for some months. There I learned that a typesetter shall follow the manuscript even if it blows out of the window. It's different at this forum.

I included the line

cac = textscan( num2str([1:10]), '%f%f%f%*[^\n]', 'HeaderLines',0 ); 

to illustrate an alternate formatspec.

Try this script (The alternative formatspec is more robust, because it reads the first three regardless of the total number of columns.)

%%
fid = fopen( 'c:\whatever\test42.txt', 'r' );
cac = textscan( fid, '%f%f%f%*[^\n]', 'CollectOutput',true );
fclose( fid );

and peek

figure; plot( cac{1}(:,1) )

[ha ha]

Thanks @per isakson. Have a nice day.

%%
tic
fid = fopen( 'C:\Users\USER\Desktop\PLO\Rama9_A_with_car.txt', 'r' );
cac = textscan( fid, '%f%f%f%*[^\n]', 'CollectOutput',true );
fclose( fid );
P=cat(1, cac{:});
toc

[ha ha]

%%
tic
clear;clc;filename = 'Rama9_A.txt'; 
inputfile = importfile_upgrade(filename);%this is 1x1 cell contain kx3 matric of all data
P=cat(1, inputfile{:});%get data=coordinate(x,y,z) from set of data "column" at (all row & column 1,2,3)
toc

[per isakson]

Thank you for showing these results.

The three elapsed times puzzles me. The differences are so large.

However, one thing I know is that it's difficult to reproduce results from reading files. A specific result depends on the state of the system cache. Large parts of the file may already be in the cache and furthermore the cache may be more or less "fragmented".

You use the statement

P=cat(1, inputfile{:}); 

It's possible to increase readability (imo) and save a second or two. Try

>> cssm
Elapsed time is 1.812582 seconds.
Elapsed time is 2.085810 seconds.
Elapsed time is 0.000041 seconds.
Elapsed time is 0.314024 seconds.
>> cssm
Elapsed time is 1.700285 seconds.
Elapsed time is 2.239885 seconds.
Elapsed time is 0.000040 seconds.
Elapsed time is 0.370981 seconds.

where the script cssm reads

%% Sample data 
tic, cac = { reshape( [1:194500412*3], [],3 ) }; toc
%%
tic, P1 = cat(1,cac{:}); toc
%%
tic, P2 = cac{1}; toc
%%
tic, P3 = cell2mat( cac ); toc

and P1 and P2 are equal

 >> all( P1==P2, 1 )
ans =
  1×3 logical array
   1   1   1

Answer 2

[Walter Roberson]

textscan() is more likely to succeed than some of the other alternatives.

Most reliable would be to pre-allocate all of the storage, and then to process chunks of the file at a time (for efficiency). For example, if you told textscan() to read 50 lines of the file, that would be just under 4 Kb, which would fit easily into MATLAB's "small blocks" storage strategy where it can extend an array in place if the array is sufficiently small. Copy the 50 rows into the master matrix, proceed to next chunk.