How to TRAIN further a previously trained agent?

Hi,

My agent was programmed to stop after reaching an average reward of X. How do I load and extend the training further?

I did enable saving of the experiences and it has created the agent file

Rajesh

Answer 1

[Rajesh Siraskar]

Hi Sourav, I figured it out after reading the documentation moer carefully!

I need to also set the ResetExperienceBufferBeforeTraining flag if I need to use previously saved experiences

This is my working code snippet. I must say this is a great feature and I really missed knowing about it!

USE_PRE_TRAINED_MODEL = true; % Set to true, to use pre-trained

% Set agent option parameter:
agentOpts.ResetExperienceBufferBeforeTraining = not(USE_PRE_TRAINED_MODEL);

if USE_PRE_TRAINED_MODEL
    % Load experiences from pre-trained agent    
    sprintf('- Continue training pre-trained model: %s', PRE_TRAINED_MODEL_FILE);   
    load(PRE_TRAINED_MODEL_FILE,'saved_agent');
    agent = saved_agent;
else
    % Create a fresh new agent
    agent = rlDDPGAgent(actor, critic, agentOpts);
end

% Train the agent
trainingStats = train(agent, env, trainOpts);

[Adrian Kaessens]

In the case of DDPG does this start the amount of noise from the beginning or will it go on from where it left of in the training process ?

[Rajesh Siraskar]

Good question Adrian: I have noticed that the noise parameters depend on the training code and parameters that you use when you restart training.

So for example lets say you had var. 0.3 and a decay rate of 1e-5. After training obviously the noise addition would have decayed, now lets say you saved this and reuse it.

When you retrain, if you settings are the same 0.3 and 1e-5, then I believe, the training resets the noise parameters so it will start afresh with this noise model parameters and decay all over again.

[Anh Tran]

Rajesh is correct. Currently the noise model resets when you train again. We are looking into how you can truly 'resume' training. As a workaround, you can set the noise variance option to a lower value than that of your previous train session.

Answer 2

[Anh Tran]

I will answer again, hopefully clear your confusion.

% Train the agent
trainingStats = train(agent, env, trainOpts);

After this line, even though the 'agent' is not returned as an output, its learnable parameters are updated. Learnable parameters, e.g. the weights and biases of the actor/critic neural networks, determines the logic behind the agent (and how it chooses action given an observation).

Now if you execute sim() or train() after this line, the 'agent' will simulate or continue training with the latest parameters.

Rajesh's workflow is very close to resume training (reuse the experiences gathered in the past, start from latest parameters). I revised the code with additional comments. Currently the noise model resets when you train again. You can consider setting the noise variance option to a lower value (still need to be > 0 because we want the agent to always explore) than that of your previous train session.

% Set to true, to resume training from a saved agent
resumeTraining = true;
% Set ResetExperienceBufferBeforeTraining to false to keep experience from the previous session
agentOpts.ResetExperienceBufferBeforeTraining = ~(resumeTraining);
if resumeTraining
    % Load the agent from the previous session
    sprintf('- Resume training of: %s', PRE_TRAINED_MODEL_FILE);   
    load(PRE_TRAINED_MODEL_FILE,'saved_agent');
    agent = saved_agent;
else
    % Create a fresh new agent
    agent = rlDDPGAgent(actor, critic, agentOpts);
end
% Train the agent
trainingStats = train(agent, env, trainOpts);

[Stav Bar-Sheshet]

Hi, this is an excellent thread!

What I'm curios about is if you continue training doest the state of the optimizer is saved and continues from the same point?

[Sayak Mukherjee]

for restarting the run with saved agent, the saved agent shaould have 'SaveExperienceBufferWithAgent' parameter set to true, right?

Answer 3

[Jonas Woeste]

Got it to work in Matlab 2022a where its a touch different:

Clue is to save the trainOpts variable after training, which then will technically be a training result object. After restoring this, increase the MaxEpisodes for further training...

% Do the agent, env stuff...
% Load pretrained agent
if isfile('trained_agent.mat') 
    load("trained_agent.mat","trainOpts")
    %   increase the max epochs to go on training
    cur_episodes = trainOpts.TrainingOptions.MaxEpisodes;
    trainOpts.TrainingOptions.MaxEpisodes = cur_episodes + num_epochs;
end
% Train
trainOpts = train(agent,env,trainOpts);
% Save
save("trained_agent.mat","trainOpts")

Please someone update the documentation about this. There its still suggesting to save the agents object...

Answer 4

[Sourav Bairagya]

In this case, you can resume your training with the previous experience buffer as a starting point.

You have to set the 'SaveExperienceBufferWithAgent' agent option to 'true'.

For some agents, such as those with large experience buffers and image-based observations, the memory required for saving their experience buffer is large. In these cases, you must ensure that there is enough memory available for the saved agents.

For more informations you can leverage this link:

https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-agents.html

[Rajesh Siraskar]

Hi Sourav,

I did go through the documentation and I did enable SaveExperienceBufferWithAgent.

But how do I next load this agent object and start the training from there on? I can use this line of code below to load the object, but how do I proceed next to begin adding to this experience?

load(RL_MODEL_FILE,'saved_agent');

Code that enabled SaveExperienceBufferWithAgent:

agentOpts = rlDDPGAgentOptions(...
    'SampleTime', Ts,...
    'TargetSmoothFactor', 1e-3,...
    'DiscountFactor', GAMMA, ...
    'MiniBatchSize', BATCH_SIZE, ...
    'ExperienceBufferLength', 1e6, 'SaveExperienceBufferWithAgent', true); 

[mr robot]

How large is "large" for an experience buffer of 1e6?

[Pei Seng Tan]

Is the option "SaveExperienceBufferWithAgent" still available for MATLAB 2022a? As no longer found it in the documentation. Will the experianced buffer with agent be saved or not to be saved since this option is removed in later documentation?

[Jonas Woeste]

Its not being saved, as the saved file is of size ~25kB regardless of trained epochs. A hint for a working practice for saving and continuing on trained agents would be nice.