Data extraction from txt file with changing number of columns

Hi everyone,

I am running simulations in a software where the output is essentially a massive amount of data in .txt files. I am trying to extract and plot the data using matlab. This is as far as I have made it, with the trial data set for 10 Monte Carlo runs. The problem is that the number of Monte Carlo runs vary so the output which is in coloumns that I will attach below varies as well. Is there maybe an alternative way to extract the data thats more proficient than what I am doing right now? or is there maybe a loop that could make it faster?

clc; close all; clear all;

%import data from DAMAGE
fid = fopen('All-In-19-QUICK_EVAL.txt');
%file 1 - lifetime
lifetime = fscanf(fid, '%s %s %s %s', [1 4]);
lifetimedata = fscanf(fid,'\n %f %f',[2 24]);
lifetimedata';
proportion = lifetimedata(1:2:39);
countlifetime = lifetimedata(2:2:40);

figure(1)
plot(proportion, countlifetime,'b');
title('Lifetime Proportion')
xlabel('proportion [.1%]'); ylabel('count'); xlim([0 1]); 

%file 2 - altitude
altituded = fscanf(fid, '\n %s %s %s', [1 3])
altitudedata = fscanf(fid, '\n %f %f', [2 36]);
altitudedata'
altitude = altitudedata(1:2:72);
countaltitude = altitudedata(2:2:72);

% figure(2)
% plot(altitude, countaltitude,'b');
% title('Altitude');
% xlabel('altitude [km]'); ylabel('count'); xlim([0 2000]); 

%file 3 - cumulative number of collisions - this section is my main problem
file3 = fscanf(fid, '\n %s %s %s', [1 30]);
file3data = fscanf(fid, '\n %f %f %f', [14 1800])
file3data'
year3 = file3data(1:14:1414);
run1in3 = file3data(2:14:1414);
run2in3 = file3data(3:14:1414);
run3in3 = file3data(4:14:1414);
run4in3 = file3data(5:14:1414);
run5in3 = file3data(6:14:1414);
run6in3 = file3data(7:14:1414);
run7in3 = file3data(8:14:1414);
run8in3 = file3data(9:14:1414);
run9in3 = file3data(10:14:1414);
run10in3 = file3data(11:14:1414);
mean3 = file3data(12:14:1414);
negStandardDev3 = file3data(13:14:1414);
posStandardDev4 = file3data(14:14:1414);

i have attached the .txt this evaluates. NOTE: code is only extracting data from the first 3 data blocks, ideally i would want a way to extract all data.

[dpb]

It is easy enough to read each section; the problem is they left no trail of bread crumbs through the forest/file to use in reading the file as for number of variables, number of observations, etc., etc., etc., ... so you've got to parse the file to find all of those things --

I'd probably start a file like this by reading the whole thing into a string array and finding the empty lines as the sections -- then you'll need a corollary database of the section types and what is the unique section heading that can be used to distinguish which section you're reading.

It's all doable and not terribly difficult, but the tedium factor is immense in working through all the details.

Answer 1

[per isakson]

The file All-In-19-QUICK_EVAL.txt contains 73 sections. Each section

• starts with two consequtive empty rows followed by
• in the final 27 sections a simulation case ID, e.g. 2028_86, followed by
• a descriptive header row followed by
• a column header row followed by
• a block of numerical data

I noticed that

• neither the header nor the column header provides names, which are legal in Matlab
• not all section types have unique headers, e.g. more than one section type has the header, Number of Objects > 10 cm
• there are nine (not ten) unique simulation case ID

I created a function to read and parse All-In-19-QUICK_EVAL.txt. It's a first attempt. The data of each section is stored in one structure. (73 structures in total.) The names of the structures are modified versions of the headers and the names of the fields are modified versions of the column headers. (That's simple to implement.) To create dynamically named structures I use the function, matfile. Thus acknowledging: TUTORIAL: Why Variables Should Not Be Named Dynamically (eval).

avdb_3( 'All-In-19-QUICK_EVAL.txt', 'test.mat' )
load test.mat
whos

  Name                                                                 Size             Bytes  Class     Attributes

  Altitude                                                             1x1               1240  struct              
  CumulativeNumberOfCatastrophicCollisions                             1x1               6120  struct              
  CumulativeNumberOfCatastrophicCollisionsWithoutConstellation         1x1               6120  struct              
  CumulativeNumberOfCatastrophicCollisionsWithoutSentinel              1x1               6120  struct              
  CumulativeNumberOfCollisions                                         1x1               6120  struct              
  CumulativeNumberOfCollisionsWithoutConstellation                     1x1               6120  struct              
  CumulativeNumberOfCollisionsWithoutSentinel                          1x1               6120  struct              
  CumulativeNumberOfCriticalCollisions                                 1x1               6120  struct              
  CumulativeNumberOfDamagingCollisions                                 1x1               6120  struct              
  CumulativeNumberOfDamagingCollisionsWithoutConstellation             1x1               6120  struct              
  CumulativeNumberOfDamagingCollisionsWithoutSentinel                  1x1               6120  struct              
  LifetimeProportion                                                   1x1                656  struct              
  NumberOfCatastrophicCollisionsByAltitude                             1x1               4040  struct              
  NumberOfCatastrophicCollisionsByAltitudeWithoutConstellation         1x1               4040  struct              
  NumberOfCollisionFragmentsGt10Cm                                     1x1             108600  struct              
  NumberOfCollisionFragmentsGt10CmWithoutConstellation                 1x1             108600  struct              
  NumberOfCollisionFragmentsGt10CmWithoutSentinel                      1x1             108600  struct              
  NumberOfCollisionFragmentsGt1Cm                                      1x1             108600  struct              
  NumberOfDeadObjectsGt10Cm                                            1x1             108600  struct              
  NumberOfDeadObjectsGt10CmWithoutConstellation                        1x1             108600  struct              
  NumberOfDeadObjectsGt10CmWithoutSentinel                             1x1             108600  struct              
  NumberOfExplosionFragmentsGt10Cm                                     1x1             108600  struct              
  NumberOfExplosionFragmentsGt1Cm                                      1x1             108600  struct              
  NumberOfIntactsGt10Cm                                                1x1             108600  struct              
  NumberOfIntactsGt10CmWithoutConstellation                            1x1             108600  struct              
  NumberOfIntactsGt10CmWithoutSentinel                                 1x1             108600  struct              
  NumberOfIntactsGt1Cm                                                 1x1             108600  struct              
  NumberOfNEWCollisionFragmentsGt10Cm                                  1x1             108600  struct              
  NumberOfNEWCollisionFragmentsGt10CmWithoutConstellation              1x1             108600  struct              
  NumberOfNEWCollisionFragmentsGt10CmWithoutSentinel                   1x1             108600  struct              
  NumberOfNon_CatastrophicCollisionsByAltitude                         1x1               4040  struct              
  NumberOfNon_CatastrophicCollisionsByAltitudeWithoutConstellatio      1x1               4040  struct              
  NumberOfNon_CatastrophicCollisionsByAltitudeWithoutSentinel          1x1               4040  struct              
  NumberOfOLDCollisionFragmentsGt10Cm                                  1x1             108600  struct              
  NumberOfObjectsGt10Cm                                                1x1              10024  struct              
  NumberOfObjectsGt10CmWithoutConstellation                            1x1             108600  struct              
  NumberOfObjectsGt10CmWithoutConstellation_1                          1x1             108600  struct              
  NumberOfObjectsGt10CmWithoutSentinel                                 1x1             108600  struct              
  NumberOfObjectsGt10CmWithoutSentinel_1                               1x1             108600  struct              
  NumberOfObjectsGt10Cm_1                                              1x1             108600  struct              
  NumberOfObjectsGt10Cm_2                                              1x1             108600  struct              
  NumberOfObjectsGt1Cm                                                 1x1               6120  struct              
  SpatialDensity_ObjectsGt10Cm                                         1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation                     1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_1                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_2                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_3                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_4                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_5                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_6                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_7                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_8                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutConstellation_9                   1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel                          1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_1                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_2                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_3                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_4                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_5                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_6                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_7                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_8                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10CmWithoutSentinel_9                        1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_1                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_2                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_3                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_4                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_5                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_6                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_7                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_8                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt10Cm_9                                       1x1             106520  struct              
  SpatialDensity_ObjectsGt1Cm                                          1x1             106520  struct              
  ans                                                                  1x49                98  char                
  cmdout                                                               1x33                66  char                

With help of tab-completion and copy&paste it's possible to use these names.

plot( LifetimeProportion.Proportion, LifetimeProportion.Count,'b');
title('Lifetime Proportion')
xlabel('proportion [.1%]'); ylabel('count'); xlim([0 1]); 
grid