Parallel workers automatically shutting down in the middle of RL parallel training.
18 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Matteo D'Ambrosio
il 10 Mag 2023
Commentato: Matteo D'Ambrosio
il 11 Mag 2023
Hello,
I am currently training a reinforcement learning PPO agent on a Simulink model with UseParallel=true. The total episodes that it should be running for are 5000 (about 10/11 hours of training), but i'm noticing that as the training goes on, more and more workers of the parallel pool are automatically shutting down, making training slower and slower as it progresses. I start with 8 workers, and they consistently decrease one at a time, until errors are generated.
I've been noticing this consistently in each training that i do, and would like to know if there are any workarounds.
For the parpool, i am letting Matlab start it automatically with all options set to default. I have also tried playing around with the number of workers, but the same thing happens.
2 Commenti
Emmanouil Tzorakoleftherakis
il 10 Mag 2023
What errors are you seeing? Maybe try training on a single worker initially to make sure you don't see any errors before moving to parallel.
Risposta accettata
Edric Ellis
il 11 Mag 2023
If workers are leaving the pool one at a time while the pool is busy, this almost certainly means that they are crashing. I recommend contacting MathWorks support for help diagnosing and resolving this problem. You could also check by running the following location to see if any "matlab_crash_dump.*" files have been left behind:
jsl = parcluster().JobStorageLocation
dir(fullfile(jsl, '**', 'matlab_crash_dump.*'))
Più risposte (0)
Vedere anche
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!