Stop Reinforcement Learning "smoothly" when the Training Manager is disabled

I'm running a Reinforcement Learning training that requires a long time to complete.

I noticed that if I disable the Training Manager (setting the "plots" property of rlTrainingOptions to "none") the training is faster overall. However, since the Training Manager is now disabled, I do not have the possibility to manually stop the training "smoothly": the button "Stop training" is not available anymore.

Of course I could brutally stop the Matlab session, but this would kill the whole process and this is not what I want, since I have some post-processing routine that must be run after the training.

Is there the possibility to manually stop the training "smoothly" whenever desired, indeed replicating the behaviour of the "Stop training" button of the Training Manager? I thought about defining a callback function that stops the training whenever a specific keyboard button is pressed, but I'm not sure if this is feasible

Answer 1

[Sumukh]

Hi Federico,

I understand that the “train” command is being used to train one or more reinforcement learning agents and the training time is long. The speed of the training can be improved by using parallel computing. You can refer to the following documentation to configure the training options to utilize parallel computing resources:

https://www.mathworks.com/help/releases/R2024a/reinforcement-learning/ref/rl.option.rltrainingoptions.html#mw_dfab8289-c772-409f-8b7c-fdc32ba02dc3

You can also refer to the following documentation to understand more about training agents using the Parallel Computing Toolbox:

https://www.mathworks.com/help/releases/R2024a/reinforcement-learning/ug/train-agents-using-parallel-computing-and-gpu.html

Please refer to the following alternative solution that utilizes callbacks:

I am assuming that the post-processing needs to be done on the output of the “train” command “trainStats”. Since no code has been provided, I have utilized the example code for the “train” command to reproduce the issue. You can run the following line in the MATLAB command window to open the example code:

openExample('rl/TrainAReinforcementLearningAgentExample')

It is indeed true that stopping the training midway does not save the “trainStats” output of the “train” command. The output is, however, saved in the base workspace when the “Stop Training” button available on the Training Monitor is used.

Please refer to the following steps to stop the training in the example code and save the training stats without using the monitor:

The solution utilizes a custom function to stop the training. The “StopTrainingCriteria” property of the training options object “trainOpts” can be utilized to manually stop the training, along with the terminating condition of the training agent. A global variable “stopTraining” is used to stop the training from a GUI without using the Training Monitor. The modifications in the example code are as follows:

• Set the “StopTrainingCriteria” property of the “trainOpts” object to “Custom”.
• Set the “StopTrainingValue”property to the custom function handle“@myTerminationFcn”.
• Create a custom function named “myTerminationFcn”. The function’s input argument is the “trainStats” output of the “train” command. The output argument “flag” stops the training when set to “true". The stopping criteria for the training agent is specified in a “if” clause along with the “stopTraining” global variable that is used to manually stop the training. The function is as follows:

function flag = myTerminationFcn(trainingInfo)
    global stopTraining; % Flag to manually stop the training by clicking the button
    
    % The “AverageReward” in the if clause below is used to terminate the training in the example code. 
    % This termination condition can be replaced as necessary for training. 
    
    if (trainingInfo.AverageReward(end) >= 495 || stopTraining)    
        flag = true;
    else
        flag = false;
    end
end

• Create a Simple GUI with a Button to stop the training by clicking a button. This allows you to manually stop the training when required without using the Training Monitor. The GUI is created by the following function:

function createStopButton()
    global stopTraining; % Flag to manually stop the training by clicking the button
    stopTraining = false; % Initially set to false

    % Create a figure
    hFig = figure('Name', 'Stop Training', 'NumberTitle', 'off', ...
                  'MenuBar', 'none', 'ToolBar', 'none', ...
                  'Position', [100, 100, 200, 100]);

    % Create a button
    uicontrol('Style', 'pushbutton', 'String', 'Stop Training', ...
              'Position', [50, 30, 100, 40], ...
              'Callback', @stopButtonCallback);

    % Callback function for the button
    function stopButtonCallback(~, ~)
        stopTraining = true;
        disp('Training stopped.');
        close(hFig); % Close the figure
    end
end

• The above GUI can be called by running the following line in the MATLAB command window, before the training is started:

createStopButton();
% Now, start your training process

• After setting the "plots" property of “trainOpts” object to "none", the training can be started. The GUI allows the training to be stopped at any time. The “trainStats” output of the “train” command is saved to the base workspace when the training is stopped midway.

The modified example codes has been attached as a zip file with this answer. I hope this helps resolves your issue.