Processing Multiple CSV files one at a time and plotting them on the same graph/
50 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
I wrote a code which takes in two CSV files and synchronizes their data and plots it on a graph. With bigger files, it shows a memory issue and the program does not run. I have tried to split the big CSV file into multiple files and I want the program to read it one by one (not storing the data since doing this also lead to program failure with memory overload). I have attached my very first code (which worked fine for smaller CSV files) and the second code that I modified for larger memory files but I still cant seem to run it.
INITIAL CODE
%Initial Code
%reading and modifying the ring Data
sleeponData = readtable('x.csv');
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring
%converting epoch time to datetime
sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;
%reading the pillow
data = importdata('sensordata.csv');
data(:,2) = [];
baseline = data(1,:);
baseline(1) = 0;
minus_bl = data - baseline;
minus_bl(end,:) = [];
tabledata = array2table(minus_bl);
%pillowTime = array2table(data(:,1));
sleeponTT = table2timetable(sleeponData);
pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
tabledata.data1 = pillow_time;
pilowTT = table2timetable(tabledata);
%synchronizing the timestamp
%new = synchronize(sleeponData, data, 'secondly','linear');
new = synchronize(sleeponTT, pilowTT);
% plotyy(new.time, new.sportRaw, new.time, new{:,3:end});
figure
hold on
x = new.time;
y = new.sportRaw;
yyaxis right;
plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
ylabel('Movement');
title('Participant x');
z = new{:,3:end};
yyaxis left;
plot(x,z, '-');
ylabel('Sensor Readings');
MODIFIED CODE BELOW:
%MODIFIED CODE
%reading and plotting the Ring Data
sleeponData = readtable('sleepring.csv'); %small file
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring
%converting epoch time to datetime
sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;
%reading the multiple sensor files
data = [];
filename = "sensordata_%i.csv";
figure
title('Sensor Data 1');
hold on
for i = 1:100;
new_filename = sprintf(filename, i);
if isfile(new_filename)
newdata = importdata(new_filename);
newdata(:,2) = [];
data = [data; newdata];
baseline = data(1,:);
baseline(1) = 0;
minus_bl = data - baseline;
minus_bl(end,:) = [];
tabledata = array2table(minus_bl);
sleeponTT = table2timetable(sleeponData);
pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
tabledata.data1 = pillow_time;
pilowTT = table2timetable(tabledata);
new = synchronize(sleeponTT, pilowTT);
x = new.time;
y = new.sportRaw;
yyaxis right;
plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
ylabel('Movement');
z = new{:,3:end};
yyaxis left;
plot(x,z, '-');
ylabel('Sensor Readings');
else
break;
end
end
2 Commenti
Eric Sofen
il 21 Mar 2022
I'm not sure what's wrong with your new code. It will be easier to diagnose with some sample data.
That said, this sounds like a situation wheretall arrays and tables might be useful. They use lazy evaluation to work with data that doesn't fit in memory.
Risposte (0)
Vedere anche
Categorie
Scopri di più su Timetables in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!