Reinforcement learning shows loss curves

The reinforcement learning training strategy has problems. To check the actor network's loss to determine if the model has been updated, how can I display its loss curve?

Answer 1

[Ruchika Parag]

Hi @浩文, to monitor the actor network's loss during reinforcement learning training in MATLAB, you can log and visualize the loss values using built-in tools. Here's a typical approach that works well when you're using train with an RL agent:

First, create a trainingProgressMonitor and an rlDataLogger to track training progress:

monitor = trainingProgressMonitor();
logger = rlDataLogger(monitor);

Then, define a callback function to log the actor loss. This function will be triggered after each learning step:

logger.AgentLearnFinishedFcn = @logActorLoss;

function dataToLog = logActorLoss(data)
    dataToLog.ActorLoss = data.ActorLoss;
end

When setting up your training options, include the logger:

trainOpts = rlTrainingOptions( ...
    MaxEpisodes = 1000, ...
    Logger = logger, ...
    Plots = "training-progress");

Train your agent with:

train(agent, env, trainOpts);

After training, you can visualize the logged data using the Reinforcement Learning Data Viewer:

openTrainingDataViewer(logger.LogFileName);

From the viewer, select the ActorLoss variable to view the trend over time. This gives you insight into whether the actor network is being updated effectively during training.

If you're using a custom training loop instead of the built-in train function, you can manually log the actor loss values at each training step and plot them yourself using standard MATLAB plotting functions. Hope this helps!