Parfor with options = statset('UseParallel',true);
4 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Christopher McCausland
il 22 Nov 2023
Commentato: Edric Ellis
il 28 Nov 2023
Hi
I have access to a large cluster many CPU's. I am hoping to prefrom the same operations on several hundred .edf input files with a parfor loop. So far so good. Part of this processing would benifit from an additional pararellel loop for all-vs-one testing.
I am therefore wondering if it is possiable to use both of these options at once. For example if I have a parpool(6) and a classifier with five classes. Therefore 6*5 = 30 cpu cores, which I have access too. I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
Alternatively I could just use parfor(30) for example, however I am intrested to see what the preformacne gains/losses are of implementing this.
Kind regards,
Christopher
0 Commenti
Risposta accettata
Edric Ellis
il 23 Nov 2023
If you are able to "flatten" the parallelism so that it is all exposed to a single parfor loop with enough iterations to keep all your workers busy, that will always be the most efficient. But that's not always possible.
The restriction on nested parfor loops is that you cannot nest a parfor loop directly inside another parfor loop. However, you can "hide" the inner parfor loop inside a function. Today, achieving nested parallelism is a bit tricky. One way to achieve that is to have each worker create its own parpool. Depending on the workload, you might be able to set your cluster to launch one multi-threaded worker per host on the cluster, and then use parpool("Threads") on the cluster workers to launch an inner thread-based pool.
2 Commenti
Edric Ellis
il 28 Nov 2023
The thing you're trying to avoid is having idle workers at the back end of your computation because they've run out of things to do. When you try to use nested parallelism, this might increase the amount of time wasted in that way. The happiest scenario is when you've got plenty of iterations of a top-level parallel construct so that they can be load-balanced by the parfor internals to smooth out any differences in execution times between the iterations etc.
Più risposte (1)
Walter Roberson
il 22 Nov 2023
I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
You would need to start separate MATLAB sessions.
You should perhaps also consider queuing up at lot of parfeval() sessions.
Hmmmm -- I wonder if parfeval() can call parfeval() to queue more work? I don't see why not... though you do risk deadlock if all of the sessions are locked up waiting for other sessions that cannot execute because there are no available cores...
2 Commenti
Edric Ellis
il 23 Nov 2023
A worker cannot call parfeval to schedule more work on the pool that it is part of (one reason is to avoid the risk of deadlock). You can use afterEach to have the client schedule more work.
Vedere anche
Categorie
Scopri di più su Parallel Computing Fundamentals in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!