Filling gaps in time series with Nan

Hi there!

I^m currently working with several series of data from different sensor.Each sensor is measuring different quantities, and, theoretically, should provide a value each 30s, so that in 1 day I should have 2880 values for each sensor. I have a matrix for each sensor with 2 columns: timestamp and the corresponding measured data. The problem is that sometime some sensors just misses the measure (very randomly) and they do not record gaps as anything, they simply skip to the next measure, so the length of the matrixes is never 2880, but always shorter and different between the sensors. What i would like to do is to detect these gaps in the timestamp column and to insert Nan values in the corresponding data column. If anyone could offer any suggestions it would be very much appreciated!

PS: the timestamp column is already in Matlab time format: (2013-03-01=735294) (30s=-3.4722e-004)

Thanks!!!!

Answer 1

[José-Luis]

Sounds like a job for intersect():

%Generating random dataset:
numVals = 2000;
all_seconds = 0:30/86400:1;
all_seconds(end) = [];
all_seconds = all_seconds' + floor(now);
your_seconds = all_seconds(sort(randperm(numel(all_seconds),2000)));
your_data = [your_seconds rand(numVals,1)];

%Actually doing what you asked:
filled_data = [all_seconds NaN(numel(all_seconds),1)];
[bla ia ib] = intersect(filled_data(:,1),your_data(:,1));
filled_data(ia,2) = your_data(ib,2);

Please accept an answer if it helps you.

[Karina]

Fantastic and elegant solution. Thanks!

[tqou37]

What is numVals?

Answer 2

[Matt Kindig]

One way that considers floating point errors:

startValue = datenum(2013,5,17,0,0,0);  %your starting observation
endValue = datenum(2013,5,17,23,59,59);    %your ending observation
increment = datenum(2013,1,1,0,0,30)-datenum(2013,1,1,0,0,0);  %30s increments
timePoints = startValue:increment:endValue;  %desired time points

% timestamps is your existing time points
% values is your existing sensor measurements
% to illustrate, we will just remove some random time points
ix = sort(ceil(rand(1,10)*length(timePoints)));
timestamps = timePoints;
timestamps(ix) = []; 
values = rand(size(timestamps));    %random values

%now get good points
 n= histc(timestamps, [timePoints-(increment/3), timePoints(end)+(increment/3)]);
% n will be 1 if sample time is present, zero otherwise.
allValues = NaN(size(timePoints));    %initially set to NaN
allValues(n==1) = values;  %fill in with "real" data