Can we manipulate a file without opening it
9 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Hello,
I have a question which I explain in bellow. Consider the following loop:
for i=1:10^6
A = Read a csv file;
A = perform some operations on A;
A= save the performed operations;
end
Apparently, the most time conssumming part is reading the file. If I use A=csvread(); then this is very time consumming. If I use fopen stuff it is
computationally cheaper but still time conssuming.
Do you have an idea to rewduce the computational time for what I intend to do?
I hope there is a way to do the above operations without actually opening any file (updating an existing file and saving the updates to the same file without opening it).
Any idea?
Thanks in advance!
Babak
8 Commenti
Stephen23
il 25 Nov 2022
Don't read and write the file on every iteration. Just use an array and indexing.
Risposte (2)
Matt J
il 25 Nov 2022
Modificato: Matt J
il 25 Nov 2022
If you have one single file, the reading and saving of the file should probably happen outside the loop. Use the parfor loop to loop over sections of the data and keep them in Matlab memory until you are ready to save all of the results.
A = Read a csv file;
parfor i=1:10^6
A(i,:) = perform some operations on A(i,:);
end
A= save the performed operations;
3 Commenti
Matt J
il 25 Nov 2022
Modificato: Matt J
il 25 Nov 2022
I don't think you should be using files to store and retrieve optimization results. I would structure the loops like this,
I=1000;
J=300;
bestValue=inf;
bestSolution=[];
for i=1:I %Loop over batches
s(1:J).Value=nan;
s(1:J).Solution=nan;
parfor j=1:J %Do a batch of optimizations in parallel
[x,fval,exitflag]=Run the optimizer
if exiflag<0 %optimization failed
continue
end
s(j).Value=fval;
s(j).Solution=x;
end
[minf,k]=min([s.Value]);
if minf<bestVal
bestVal=minf;
bestSolution=s(k).Solution;
end
end
Walter Roberson
il 25 Nov 2022
I suggest that you switch to using parfeval() . Approximately
while you haven't gotten tired of it all
while number of active workers is less than number of cores
use parfeval() to create a new worker passing in a different initial condition
end
wait for a worker to finish, using a timeout
if any worker has been active longer than you want, cancel() the worker, end
if any workers have finished, fetch their results and update the notion of best, end
end
when you get tired of it all, cancel all remaining workers
Vedere anche
Categorie
Scopri di più su Parallel Computing Fundamentals in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!