Coalesce data from 1000 monte Carlo simulations with different data file lengths?

Hi all,

I'm trying to coalesce data from 1000 monte Carlo simulations of contaminant infiltration into the ground. Each Monte Carlo run generated a file Obs_Node.txt. I want to skip the first 11 lines of this .txt file, then isolate each data column beneath that to save for future analysis. My challenge is that each Monte Carlo run has unique time convergence intervals...ie the length of the data columns are different for each MC iteration's Obs_Node.txt file. In other words, in one Path(i) (ie Monte Carlo iteration) there would be 123456 rows of data for each of the 5 columns and in another Path(i) there may be 65432 rows of data for each of the .txt's 5 data columns. So the ## changes with each Path(i).

My questions are:

1) how do I preallocate when the length of the data changes for each Path(i) at the top of the code?

2) how do I tell Matlab to scan the data when I don't know the length of the data colums ?

I attached an example of Obs_Node.txt for reference....there are only 5 rows in this reference file for simplicity.

num_sim = 1000;  %number of moonte carlo simulations

%Obs_Node.out  -  preallocation of desired output data from each monte carlo run
TIME_INTERVAL = zeros(##,num_sim);
Node_CONC=zeros(##,num_sim);
THETA_T1 = zeros(##,num_sim);
FLUX_T1 = zeros(##,num_sim);
CONC_T1 = zeros(##,num_sim);

%~~~~~~~~~~Coalesce data from Obs_Node.out files~~~~~~~~~~~~
for i=1:num_sim

    Obs_Node = fopen(["/Users/apple/Dropbox/Desktop/Simulations/MC_"+num2str(i)+'/Obs_Node.out']);   % Open monte carlo output file in Path (i) 
    
    skip_lines=11;   %skip all the lines until the output data of interest
    
    for k=1:(skip_lines)
        x=fgetl(Obs_Node);
    end
    
  
    
    temp1 = fscanf(Obs_Node,'%f',[5,##]);  %scan the matrix of data (QUESTION 2 ABOVE)
    TEMP1 = temp1';       % transpose data
    
    time_interval = TEMP1(:,1);      % Time intervals - 1st data column
    TIME_INTERVAL(:,i) = time_interval(:); % saves each iteration's data 
    
    theta_T1 = TEMP1(:,3);      % Hydraulic Conductivity - 3rd data column
    THETA_T1(:,i) = theta_T1(:);  % saves each iteration's data 
    
    fclose(Obs_Node);
end

[Benjamin Thompson]

My example works on text files regardless of the extension. So *.out is fine.

Answer 1

[Benjamin Thompson]

With the data import tool you can generate a function like this. It needs some editing to handle some of the odd formatting of your file. Then you can pass [12 Inf] as the datalines argument. The "end" at the end of your file is difficult, so you can either remove the last line from the output after loading the file data, or have a separate tool or function that removes the last row with "end" from files before you read them into MATLAB.

Answer 2

[David Hill]

opts = detectImportOptions('Obs_Node.txt');
opts.VariableTypes={'double','double','double'};
opts.Delimiter={'\t'};
opts.DataLines=[12 Inf];
t=readmatrix('Obs_Node.txt',opts);
t(:,1)=[];
t(isnan(t(:,1)),:)=[]

t = 5×5

0.0010 -25.0000 0.2827 -382.0000 0 0.0020 -25.0000 0.2827 -244.0000 0 0.0030 -25.0000 0.2827 -170.0000 0 0.0043 -25.0000 0.2827 -114.0000 0 0.0060 -25.0000 0.2827 -75.4000 0

[Wesser]

This works conceptually but the file that is being read is actually "Obs_Node.out" and not "Obs_Node.txt". I had changed the file ending for the question above as I couldn't upload the example file without changing the file from .out to .txt. The opts route seems feasible, but how do I get matlab to read the .out file path? It doesn't seem that there is a name-value pair that can access the .out files.

Name-Value Pairs supported with Import Options OPTS:

% See also spreadsheetImportOptions, delimitedTextImportOptions, fixedWidthImportOptions,xmlImportOptions, readtable, readtimetable, readmatrix, readcell