MATLAB Answers

Averaging each .STD files

1 view (last 30 days)
Greetings. I have been pondering and working hard on this issue since I came here to ask for solutions. I have over 100 .std files in which each of them has 10 headers and each header has 24 hours data of which each hour has 60 values. I want to loop through the hundred data files and get the average of the 24hrs data with graphs. I was only able to do for each data at a time and it's very tedious.

  2 Comments

Steve Eddins
Steve Eddins on 28 Oct 2020
If you can provide more information about the content and structure of your files, it might help people come up with answers for you. It would be ideal if you could attach a couple of sample data files.
Based on your description so far, I would probably be looking to create a timetable containing all the data and then use the retime function to get 24-hour averages.
Moses Joseph
Moses Joseph on 28 Oct 2020
I have sent the attached files. I would be grateful if I can get it done.
Thank you.

Sign in to comment.

Accepted Answer

Steve Eddins
Steve Eddins on 29 Oct 2020
Edited: Steve Eddins on 29 Oct 2020
[Update: Based on comments from Moses, I submitted another answer to this question. See below.]
Try this:
files = dir("*.Std");
N = numel(files);
Filename = strings(N,1);
Date = NaT(N,1);
Mean1 = zeros(N,1);
Mean2 = zeros(N,1);
Mean3 = zeros(N,1);
Mean4 = zeros(N,1);
for k = 1:numel(files)
name = string(files(k).name);
A = readmatrix(name,"FileType","text");
[~,base_name,~] = fileparts(name);
base_name_parts = split(base_name,"-");
Filename(k) = name;
Date(k) = datetime(join(base_name_parts(2:4),"-"));
means = mean(A,1);
Mean1(k) = means(1);
Mean2(k) = means(2);
Mean3(k) = means(3);
Mean4(k) = means(4);
end
T = timetable(Date,Filename,Mean1,Mean2,Mean3,Mean4);
When I run this with your data files, T is this:
8×5 timetable
Date Filename Mean1 Mean2 Mean3 Mean4
___________ ________________________ ______ ______ ______ _____
01-Jan-2015 "bjco001-2015-01-01.Std" 11.992 24.82 2.6294 6.38
02-Jan-2015 "bjco002-2015-01-02.Std" 11.992 22.25 2.1003 6.38
03-Jan-2015 "bjco003-2015-01-03.Std" 11.992 25.834 1.9662 6.38
04-Jan-2015 "bjco004-2015-01-04.Std" 11.992 32.088 1.6014 6.38
05-Jan-2015 "bjco005-2015-01-05.Std" 11.992 25.03 1.8935 6.38
06-Jan-2015 "bjco006-2015-01-06.Std" 11.992 31.423 1.5822 6.38
07-Jan-2015 "bjco007-2015-01-07.Std" 11.992 31.528 1.8784 6.38
08-Jan-2015 "bjco008-2015-01-08.Std" 11.992 31.08 1.6912 6.38
...
Plot sample:
plot(T.Date,T.Mean2)

  1 Comment

Moses Joseph
Moses Joseph on 29 Oct 2020
In short, I am short of words for this kind of response from you Sir. I am really grateful Sir for your valuable help only that it hasn't answered my question. If you noticed let's say for instance, in bjco001-2015-01-01.std, we have 4 columns and each column has 0-23 of which 0 has almost 60 values ie, 0.00-0.93 which represent a day value. Another day value starts with 1.00 till we no longer see 1 then continue from 2 again till we get to 23.
So for the four columns, I need the average value of 0.00-0.93, 1.000-1.983, 2.000-2.983 till 23.000-23.983.
Which means, for the first file, I should have 4 columns each column having 24 rows values. So I want to do this for the whole files instantaneously without having doing them one file at a time.
D = load('bjco001-2015-01-01.txt'); M = ceil(D(:,1)+1E-4); HrMeans = accumarray(M, (1:numel(M)).', [], @(x){mean(D(x,2:4))}); Result = [(0:23).' cell2mat(HrMeans)]
figure plot(Result(:,1), Result(:,2)) grid xlabel('Time') ylabel('VTEC')
The above was the code I got from here but it only worked for one file at a time and I have to convert the STD file to txt file before I could use it.
It's so stressful using it but I still appreciated it.
So I need the one that can loop through the who STD files I sent to you and give give me 4 columns with 24 rows data for each file with a graph for each. I have tried many possibilities before coming to ask online here but no result.
Please to use the above code I sent, it can only work after converting the file to txt which I believe you know it sir.
I would be indebted to you if this is achieved here. Nothing is impossible and that's why I am here.
Thank you in anticipation for a positive result.

Sign in to comment.

More Answers (3)

Moses Joseph
Moses Joseph on 28 Oct 2020
Attached are the files but compressed

  0 Comments

Sign in to comment.


Moses Joseph
Moses Joseph on 28 Oct 2020
Attached are the files in zip format

  0 Comments

Sign in to comment.


Steve Eddins
Steve Eddins on 29 Oct 2020
Try this:
files = dir("*.Std");
N = numel(files);
Date = NaT(0,1);
X = zeros(0,3);
for k = 1:numel(files)
name = string(files(k).name);
A = readmatrix(name,"FileType","text");
[~,base_name,~] = fileparts(name);
base_name_parts = split(base_name,"-");
Date = [Date ; datetime(join(base_name_parts(2:4),"-")) + (A(:,1)/24)];
X = [X ; A(:,2:4)];
end
T = timetable(Date,X(:,1),X(:,2),X(:,3));
T = sortrows(T);
head(T)
ans =
8×3 timetable
Date Var1 Var2 Var3
____________________ _____ ____ ____
01-Jan-2015 00:00:00 12.72 2.69 6.38
01-Jan-2015 00:01:01 12.69 2.47 6.38
01-Jan-2015 00:01:58 12.66 2.54 6.38
01-Jan-2015 00:03:00 12.63 2.86 6.38
01-Jan-2015 00:04:01 12.6 2.39 6.38
01-Jan-2015 00:04:58 12.57 2.52 6.38
01-Jan-2015 00:06:00 12.53 2.81 6.38
01-Jan-2015 00:07:01 12.49 2.65 6.38
T2 = retime(T,'hourly','mean');
head(T2)
ans =
8×3 timetable
Date Var1 Var2 Var3
____________________ ______ _______ ____
01-Jan-2015 00:00:00 10.85 2.0272 6.38
01-Jan-2015 01:00:00 8.4937 1.8062 6.38
01-Jan-2015 02:00:00 7.681 1.1152 6.38
01-Jan-2015 03:00:00 6.0948 1.016 6.38
01-Jan-2015 04:00:00 3.7198 1.0692 6.38
01-Jan-2015 05:00:00 3.066 1.1472 6.38
01-Jan-2015 06:00:00 8.8583 0.97183 6.38
01-Jan-2015 07:00:00 18.533 1.1425 6.38

  14 Comments

Steve Eddins
Steve Eddins on 30 Oct 2020
I'm glad you were able to figure it out!
Moses Joseph
Moses Joseph on 30 Oct 2020
Kudos sir 👍
Moses Joseph
Moses Joseph on 31 Oct 2020
Greetings Sir.
Once again thank you.
If you wouldn't mind, I have 2 questions to ask from you as regards this work.
1. Can you please explain each line of this codes as to a lame man cause is of no use of I can't explain well to people or understand its functionalities. Differentiate which is MATLAB inbuilt functions/variables and personal functions/variables.
2. The table of results gotten from this computation, can you help in splitting it into days and hours for each month which means 31days(31 rows) and 24 hrs(24 columns) for each month and calculating the mean, media, standard deviation, variation for each of the hour or column?
If this can be automated using MATLAB, I will so much appreciate. I have computed the one you did for me manually for one year using Excel spreadsheet but man, it's too much work for me going for 5 years data. Copying and pasting which is error prone.
Hope I am not asking for too much sir? I believe the solution will be of help to many people doing this kind of project.
Happily waiting your positive response sir.
Thank you

Sign in to comment.

Tags

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by