Signal processing timetable data in segments defined by time ranges

Hi,

I have large recordings stored as tall timetables on which I would like to do signal processing in consecutive time range blocks.

Big data processing is new to me, and would like to ask if there is an efficient way to do this with mapreduce, or some other technique? (I am currently selecting each subset at a time, processing, and storing the result, but this is taking long.)

I've been thinking that writing a "readfcn()" that gets the time period number might be the way, but its not clear how to do this.

Could anyone please give me some advice?

Many thanks, Kevin

[Julian]

What exactly do you want to do?

[Kevin Williams]

I would like to calculate certain statistics on burst transmissions.

[Star Strider]

I’m not certain what those statistics are, however the approach in Using MapReduce to Fit a Logistic Regression Model could provide some guidance. (I have no experience with mapreduce since none of my data sets ever required it.)

[Mathieu NOE]

it would certainly help to have some data samples

[Kevin Williams]

Hi, the data is just a timetable of complex samples, 1 sample per row.

[Kevin Williams]

Experimenting with mapreduce, I see that the mapper gets blocks of a certain number of rows (I get 65532).

This results in some "leftover" rows at the end of the block that don't necessarily form a complete burst. (I am trying to process my recording in frames of a fixed number of samples.)

I noticed the "CustomDatastore" has a read() method that needs to be implemented, and wondered whether this could be used.

However, I'm not sure this is the best way to do this burst processing? (The statistics I need are things like the peak power, average power, etc. in the burst.)

[Sam Marshalik]

If you have a tall array, are you able to use overloaded functions that works on tall to do your analysis? If so, you can leverage Parallel Computing Toolbox or MATLAB Parallel Server to speed up that analysis to run simultenously across multiple cores on one or multiple machines.