Coalesce data from 1000 monte Carlo simulations with different data file lengths?

4 visualizzazioni (ultimi 30 giorni)
Hi all,
I'm trying to coalesce data from 1000 monte Carlo simulations of contaminant infiltration into the ground. Each Monte Carlo run generated a file Obs_Node.txt. I want to skip the first 11 lines of this .txt file, then isolate each data column beneath that to save for future analysis. My challenge is that each Monte Carlo run has unique time convergence the length of the data columns are different for each MC iteration's Obs_Node.txt file. In other words, in one Path(i) (ie Monte Carlo iteration) there would be 123456 rows of data for each of the 5 columns and in another Path(i) there may be 65432 rows of data for each of the .txt's 5 data columns. So the ## changes with each Path(i).
My questions are:
1) how do I preallocate when the length of the data changes for each Path(i) at the top of the code?
2) how do I tell Matlab to scan the data when I don't know the length of the data colums ?
I attached an example of Obs_Node.txt for reference....there are only 5 rows in this reference file for simplicity.
num_sim = 1000; %number of moonte carlo simulations
%Obs_Node.out - preallocation of desired output data from each monte carlo run
TIME_INTERVAL = zeros(##,num_sim);
THETA_T1 = zeros(##,num_sim);
FLUX_T1 = zeros(##,num_sim);
CONC_T1 = zeros(##,num_sim);
%~~~~~~~~~~Coalesce data from Obs_Node.out files~~~~~~~~~~~~
for i=1:num_sim
Obs_Node = fopen(["/Users/apple/Dropbox/Desktop/Simulations/MC_"+num2str(i)+'/Obs_Node.out']); % Open monte carlo output file in Path (i)
skip_lines=11; %skip all the lines until the output data of interest
for k=1:(skip_lines)
temp1 = fscanf(Obs_Node,'%f',[5,##]); %scan the matrix of data (QUESTION 2 ABOVE)
TEMP1 = temp1'; % transpose data
time_interval = TEMP1(:,1); % Time intervals - 1st data column
TIME_INTERVAL(:,i) = time_interval(:); % saves each iteration's data
theta_T1 = TEMP1(:,3); % Hydraulic Conductivity - 3rd data column
THETA_T1(:,i) = theta_T1(:); % saves each iteration's data

Risposte (2)

Benjamin Thompson
Benjamin Thompson il 24 Ott 2022
With the data import tool you can generate a function like this. It needs some editing to handle some of the odd formatting of your file. Then you can pass [12 Inf] as the datalines argument. The "end" at the end of your file is difficult, so you can either remove the last line from the output after loading the file data, or have a separate tool or function that removes the last row with "end" from files before you read them into MATLAB.

David Hill
David Hill il 24 Ott 2022
opts = detectImportOptions('Obs_Node.txt');
opts.DataLines=[12 Inf];
t = 5×5
0.0010 -25.0000 0.2827 -382.0000 0 0.0020 -25.0000 0.2827 -244.0000 0 0.0030 -25.0000 0.2827 -170.0000 0 0.0043 -25.0000 0.2827 -114.0000 0 0.0060 -25.0000 0.2827 -75.4000 0
  1 Commento
Wesser il 24 Ott 2022
This works conceptually but the file that is being read is actually "Obs_Node.out" and not "Obs_Node.txt". I had changed the file ending for the question above as I couldn't upload the example file without changing the file from .out to .txt. The opts route seems feasible, but how do I get matlab to read the .out file path? It doesn't seem that there is a name-value pair that can access the .out files.
Name-Value Pairs supported with Import Options OPTS:
% See also spreadsheetImportOptions, delimitedTextImportOptions, fixedWidthImportOptions,xmlImportOptions, readtable, readtimetable, readmatrix, readcell

Accedi per commentare.





Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by