Importing big .ASC file

Good morning, I searched on the web but I din't find anything useful for me.

I have a big .ASC file that begins like this:

Time  1 - default sample rate [s]	MX840A_0_CH 1 [µm/m]	MX840A_0_CH 2 [µm/m]	MX840A_0_CH 3 [µm/m]	MX840A_0_CH 4 [µm/m]	MX840A_0_CH 5 [µm/m]	MX840A_0_CH 6 [µm/m]	MX840A_0_CH 7 [µm/m]	MX840A_0_CH 8 [µm/m]	Time  2 - default sample rate [s]	Time  2 - fast sample rate [s]	MX410_0_CH 1 [µm/m]	MX410_1_CH 2 [g]
0	0,1980	0,3258	-0,2376	-0,5230	1,680	-0,1286	-0,3765	-0,2802	0	0	0,2373	-0,00105	
0,00083	0,07124	0,3033	-0,04901	-0,4132	1,524	0,05268	-0,2449	-0,1356	0,00083	0,00021	0,2639	-0,00067	
0,00167	0,05543	0,2692	0,05971	-0,3727	1,215	0,2908	-0,1258	-0,1223	0,00167	0,00042	0,3119	0,000	
0,00250	0,1738	0,2790	0,2098	-0,3734	0,7158	0,1389	-0,1260	-0,1299	0,00250	0,00063	0,3839	0,00058	
0,00333	0,2204	0,2843	0,3669	-0,2345	0,3564	0,09048	0,01820	-0,04594	0,00333	0,00083	0,3732	0,00080	

...

It is basically n x 13 matrix with a big "n" (millions of rows).

I would like to import all these data in a suitable file for Matlab. I tried with the code:

filename = 'myfile01.txt';
delimiterIn = ' ';
headerlinesIn = 1;
A = importdata(filename,delimiterIn,headerlinesIn);

The result of this simple script (A) is only a 1x1 cell. I can't understand what I am missing. I would like to handle this data in Matlab then, in order to perform an analysis.

Thank you for your help!

[Ive J]

Have you tried datastore?

doc tabularTextDatastore

[Walter Roberson]

Do you have enough memory that you could read in the entire file as text?

importdata() is not going to understand using comma as decimal separator. readtable() would be more likely to understand it -- or you could read the file as text, replace the comma with decimal point, and then textscan() the characters.

[Guglielmo Giambartolomei]

@Ive Ive: no, I didn't try. Thanks!

[Guglielmo Giambartolomei]

@Walter Roberson: I'm able to read the text file with the Matlab button "Import data" on the top of the window but it's very slow on my computer.

Answer 1

[Walter Roberson]

T = readtable('big.ASC','filetype','text', 'decimal', ',');

(I'm not sure why it loses the variable names.)

When I copy and paste from what you posted, I end up with an empty 14th column, indicating that there is a tab after the 13th column. You can do

T{:,1:13}

to extract a numeric array of 13 columns, or you can

T(:,14:end) = [];

to delete anything after column 13.

[Guglielmo Giambartolomei]

Thank you Walter, I tried the command you suggested me but I received the message:

Error using readtable (line 216)

Invalid parameter name: decimal.

Is it maybe because the numerical values have a comma instead of a point?

Thank you!

[Walter Roberson]

Which MATLAB release are you using? The 'decimal' option was relatively recent.

filename = 'myfile01.txt';
S = fileread(filename);
S = regexprep(S, ',', '.');

fmt = [repmat('%f', 1, 13), '%*[^\n]'];
datacell = textscan(S, fmt, 'HeaderLines', 1, 'delimiter', '\t');

rawvariables = regexp(regexp(S, '^.*$', 'match', 'once'), '\t', 'split');
if isempty(rawvariables{end}); rawvariables(end) = []; end
variables = matlab.lang.makeUniqueStrings( matlab.lang.makevalidname(rawvariables) );

A = cell2table(datacell, 'VariableNames', variables);

[Guglielmo Giambartolomei]

R2018b Walter. Thank you!

[Walter Roberson]

I think the above code might work. I am not certain, as the sample you copyied and pasted might not have exactly the same format as your actual file. Testing would be easier if you could attach a sample as a file.

Answer 2

[Guglielmo Giambartolomei]

Dear Walter, I downloaded the 2020a version and it works! I have a question: now I have a n x 1 table (one column of the T table). How can I covert that object in a n x 1 double in order to analyse the data? Thank you!

[Walter Roberson]

T{:, ColumnNumber}

or you can use table2array() but I have not found that to fit into my style.

[Guglielmo Giambartolomei]

I used table2array...Thank you very much!