RL Training Manager has progressively slower updates as training progresses

I'm training a RL agent using the train function and I'm using the Training Manager to monitor the reward evolution.

I noticed that at the beginning of the training the Training Manager is updated very quickly with the informations related to the latest episodes.

However, as the training progresses, the Training Manager updates become slower and slower. As a consequence, the training itsef slows down noticeably.

Please consider that the simulation environment is moderately complex, but the number of logged variables is absolutely minimal (a few constant values for each simulation).

What could be the reason of the progressive slow-down? Could it be due to some not negligible graphical overhead of the Training Manager? (e.g. the necessity of re-generating the whole pattern of episodes from scratch every time)

Answer 1

[Ronit]

Hi Federico,

It is possible that the slowdown you are experiencing during training could be due to the graphical overhead of the Training Manager. Also, take a look at the following potential aspects:

1. Graphical Overhead: The continuous updating of plots and charts can become resource-intensive over time, especially if the number of episodes increases significantly. Try reducing the frequency of updates in the Training Manager. It can be done by adjusting the "Plots" option in the training options to update less frequently or turn off some of the plots temporarily.
2. Logging Overhead: Even with minimal logged variables, the overhead of writing data to logs can accumulate, especially if the data is being written to disk frequently.
3. Memory Accumulation: The data accumulated from each episode could consume significant memory, slowing down the process. Ensure that unnecessary data is not being stored. Consider clearing variables that are no longer needed or reducing the amount of data logged per episode.

I recommend leveraging the "useParallel" option for parallel computing to potentially accelerate the training process. Please refer to the documentation of the "useParallel" option provided by "rlTrainingOptions":

https://www.mathworks.com/help/reinforcement-learning/ref/rl.option.rltrainingoptions.html#mw_1f5122fe-cb3a-4c27-8c80-1ce46c013bf0_sep_mw_b9de33e6-196c-47d9-8bcf-ca39f1c6bf49

I hope it helps your query!