Divide a data set into sub data sets

Hi,

I am running a simulation program.

There, in every simulation run it creates a data set of size T. I would like to know if there's a loop I can use to divide this data set into 20 sub data sets of equal size and randomly choose a data set, among the 20 sub data sets, to be used in a different pat of the program.

any help is greatly appreciated.

Dav.

[DGM]

@Mehri Mehrnia's question-as-answer moved here:

Hi,

I read all Q&As, but I have another question:

I have a big data set, due to the algorithim, I have a vector with a size in milions so Matlab is not responsive.

what is an efficient way to divide this data to smaller parts that Matlab can handle it?

In fact, it should be unbiased data segmentation

Answer 1

[Azzi Abdelmalek]

A=1:100  % Example
data=reshape(A,[],20) % each column represent your part of data
id=randi(20)
random_vector=data(:,id)

[dav]

Thanks a lot.

Does randi randomly pick a vector among the available 20? And is it similar to sampling with replacement?

because i need to sample a vector under sampling with replacement.

Thanks again!

[Azzi Abdelmalek]

I do not know what (sample a vector under sampling with replacement) means. The code pick randomly a vector from the 20 parts, but it can pick the same another time

[Azzi Abdelmalek]

If you want to avoid repetition

A=1:100  % Example
data=reshape(A,[],20) % each column represent your part of data
[n,m]=size(data);
id=randi(m)
random_vector=data(:,id)
data(:,id)=[]

[dav]

thank you very much!

[Khan Engr]

I have similar type of question with one step ahead to ask;

If I have a data set 'A' of 50000 samples (sampling time 1e-4sec) and want to divide this data set into small data sets, each small data set having 100 samples. And then have them in the form such as; A1, A2, A3, ....A500, so that I can use them further. (the small data sets are in a sequence such as A1= A(1:100), A2=A(101:200), A3=A(201:300), ..... A500)

I Will be grateful for kind help.

Answer 2

[Jan]

The question is not clear. You have T "data sets" (what ever this mean) and want to divide it in 20 "sets" of equal size. Is T dividable by 20? Should the new data sets contain repetitions, e.g. is this allowed:

aNewDataSet = [T(1), T(1), T(3:end)]

Should the new data sets be elementwise distinct or is this allowed:

newDataSet(1) = [T(1), T(3:10)]
newDataSet(2) = [T(1), T(4:11)]

When only one data set is used afterwards, why creating 20 of them at first?