getModel

Get function approximator model from actor or critic

Since R2020b

Syntax

model = getModel(fcnAppx)

Description

model = getModel(fcnAppx) returns the function approximation model used by the actor or critic function object fcnAppx.

Examples

Modify Deep Neural Networks in Reinforcement Learning Agent

Create an environment with a continuous action space and obtain its observation and action specifications. For this example, load the environment used in the example Train DDPG Agent to Control Double Integrator System.

Load the predefined environment.

env = rlPredefinedEnv("DoubleIntegrator-Continuous");

Obtain observation and action specifications.

obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create a PPO agent from the environment observation and action specifications. This agent uses default deep neural networks for its actor and critic.

agent = rlPPOAgent(obsInfo,actInfo);

To modify the deep neural networks within a reinforcement learning agent, you must first extract the actor and critic function approximators.

actor = getActor(agent);
critic = getCritic(agent);

Extract the deep neural networks from both the actor and critic function approximators.

actorNet = getModel(actor);
criticNet = getModel(critic);

The networks are dlnetwork objects. To view them using the plot function, you must convert them to layerGraph objects.

For example, view the actor network.

plot(layerGraph(actorNet))

To validate a network, use analyzeNetwork. For example, validate the critic network.

analyzeNetwork(criticNet)

You can modify the actor and critic networks and save them back to the agent. To modify the networks, you can use the Deep Network Designer app. To open the app for each network, use the following commands.

deepNetworkDesigner(layerGraph(criticNet))
deepNetworkDesigner(layerGraph(actorNet))

In Deep Network Designer, modify the networks. For example, you can add additional layers to your network. When you modify the networks, do not change the input and output layers of the networks returned by getModel. For more information on building networks, see Build Networks with Deep Network Designer.

To validate the modified network in Deep Network Designer, you must click on Analyze for dlnetwork, under the Analysis section. To export the modified network structures to the MATLAB® workspace, generate code for creating the new networks and run this code from the command line. Do not use the exporting option in Deep Network Designer. For an example that shows how to generate and run code, see Create DQN Agent Using Deep Network Designer and Train Using Image Observations.

For this example, the code for creating the modified actor and critic networks is in the createModifiedNetworks helper script.

createModifiedNetworks

Each of the modified networks includes an additional fullyConnectedLayer and reluLayer in their main common path. View the modified actor network.

plot(layerGraph(modifiedActorNet))

After exporting the networks, insert the networks into the actor and critic function approximators.

actor = setModel(actor,modifiedActorNet);
critic = setModel(critic,modifiedCriticNet);

Finally, insert the modified actor and critic function approximators into the actor and critic objects.

agent = setActor(agent,actor);
agent = setCritic(agent,critic);

Input Arguments

fcnAppx — Actor or critic function object

rlValueFunction object | rlQValueFunction object | rlVectorQValueFunction object | rlContinuousDeterministicActor object | rlDiscreteCategoricalActor object | rlContinuousGaussianActor object

Actor or critic function object, specified as one of the following:

• rlValueFunction object — Value function critic

• rlQValueFunction object — Q-value function critic

• rlVectorQValueFunction object — Multi-output Q-value function critic with a discrete action space

• rlContinuousDeterministicActor object — Deterministic policy actor with a continuous action space

• rlDiscreteCategoricalActor — Stochastic policy actor with a discrete action space

• rlContinuousGaussianActor object — Stochastic policy actor with a continuous action space

To create an actor or critic function object, use one of the following methods.

• Create a function object directly.

• Obtain the existing critic from an agent using getCritic.

• Obtain the existing actor from an agent using getActor.

Note

For agents with more than one critic, such as TD3 and SAC agents, you must call getModel for each critic representation individually, rather than calling getModel for the array returned by getCritic.

critics = getCritic(myTD3Agent);
criticNet1 = getModel(critics(1));
criticNet2 = getModel(critics(2));

Output Arguments

model — Function approximation model

dlnetwork object | rlTable object | 1-by-2 cell array

Function approximation model, returned as one of the following:

• Deep neural network defined as a dlnetwork object

• rlTable object

• 1-by-2 cell array that contains the function handle for a custom basis function and the basis function parameters

Version History

Introduced in R2020b

R2022a: getModel now uses approximator objects instead of representation objects

Using representation objects to create actors and critics for reinforcement learning agents is no longer recommended. Therefore, getModel now uses function approximator objects instead.

R2021b: getModel returns a dlnetwork object

Starting from R2021b, built-in agents use dlnetwork objects as actor and critic representations, so getModel returns a dlnetwork object.

• Due to numerical differences in the network calculations, previously trained agents might behave differently. If this happens, you can retrain your agents.

• To use Deep Learning Toolbox™ functions that do not support dlnetwork, you must convert the network to layerGraph. For example, to use deepNetworkDesigner, replace deepNetworkDesigner(network) with deepNetworkDesigner(layerGraph(network)).