rlValueFunction
Description
This object implements a value function approximator object that you can use as a
critic for a reinforcement learning agent. A value function (also known as state-value
function) is a mapping from an environment observation to the value of a policy. Specifically,
its output is a scalar that represents the expected discounted cumulative long-term reward
when an agent starts from the state corresponding to the given observation and executes
actions according to a given policy afterwards. After you create an
rlValueFunction critic, use it to create an agent such as an rlACAgent, rlPGAgent, or rlPPOAgent agent. For
an example of this workflow, see Create Actor and Critic Representations. For more information on creating
actors and critics, see Create Policies and Value Functions.
Creation
Syntax
Description
critic = rlValueFunction(net,observationInfo)critic using the deep neural
network net as approximation model, and sets the
ObservationInfo property of critic to the
observationInfo input argument. The network input layers are
automatically associated with the environment observation channels according to the
dimension specifications in observationInfo.
critic = rlValueFunction(tab,observationInfo)critic with a discrete
observation space, from the table tab, which is an
rlTable object
containing a column array with as many elements as the number of possible observations.
The function sets the ObservationInfo property of
critic to the observationInfo input
argument, which in this case must be a scalar rlFiniteSetSpec
object.
critic = rlValueFunction({basisFcn,W0},observationInfo)critic using a custom basis
function as underlying approximator. The first input argument is a two-element cell
array whose first element is the handle basisFcn to a custom basis
function and whose second element is the initial weight vector W0.
The function sets the ObservationInfo property of
critic to the observationInfo input
argument.
critic = rlValueFunction(___,Name=Value)UseDevice property using one or more name-value arguments.
Specifying the input layer names allows you explicitly associate the layers of your
network approximator with specific environment channels. For all types of approximators,
you can specify the device where computations for critic are
executed, for example UseDevice="gpu".
Input Arguments
Name-Value Arguments
Properties
Object Functions
rlACAgent | Actor-critic (AC) reinforcement learning agent |
rlPGAgent | Policy gradient (PG) reinforcement learning agent |
rlPPOAgent | Proximal policy optimization (PPO) reinforcement learning agent |
getValue | Obtain estimated value from a critic given environment observations and actions |
evaluate | Evaluate function approximator object given observation (or observation-action) input data |
getLearnableParameters | Obtain learnable parameter values from agent, function approximator, or policy object |
setLearnableParameters | Set learnable parameter values of agent, function approximator, or policy object |
setModel | Set approximation model in function approximator object |
getModel | Get approximation model from function approximator object |
Examples
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous four-dimensional space, so that there is a single observation channel that carries a column vector containing four doubles.
obsInfo = rlNumericSpec([4 1]);
A value-function critic takes the current observation as input and returns a single scalar as output (the estimated discounted cumulative long-term reward for following the policy from the state corresponding to the current observation).
To model the parametrized value function within the critic, use a neural network with one input layer (which returns the content of the observation channel, as specified by obsInfo) and one output layer (returning the scalar value). Note that prod(obsInfo.Dimension) returns the total number of dimensions of the observation space regardless of whether the observation space is a column vector, row vector, or matrix.
Define the network as an array of layer objects.
net = [
featureInputLayer(prod(obsInfo.Dimension));
fullyConnectedLayer(10);
reluLayer;
fullyConnectedLayer(1)
];Convert the network to a dlnetwork object.
dlnet = dlnetwork(net);
You can plot the network using plot and display its main characteristics, like the number of weights, using summary.
plot(dlnet)
summary(dlnet)
Initialized: true
Number of learnables: 61
Inputs:
1 'input' 4 features
Create the critic using the network and the observation specification object.
critic = rlValueFunction(dlnet,obsInfo)
critic =
rlValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {4×1 cell}
State: {0×1 cell}
To check your critic, use getValue to return the value of a random observation, using the current network weights.
v = getValue(critic,{rand(obsInfo.Dimension)})v = single
0.5196
You can now use the critic (along with an actor) to create an agent for the environment described by the given observation specification object. Examples of agents that can work with a continuous observation space, and use a value function critic, are rlACAgent, rlPGAgent, rlPPOAgent, and rlTRPOAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an actor and a critic that you can use to define a reinforcement learning agent such as an Actor-Critic (AC) agent. For this example, create actor and critic for an agent that can be trained against the cart-pole environment described in Train AC Agent to Balance Discrete Cart-Pole System.
First, create the environment. Then, extract the observation and action specifications from the environment. You need these specifications to define the agent and critic.
env = rlPredefinedEnv("CartPole-Discrete");
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);A value-function critic takes the current observation as input and returns a single scalar as output (the estimated discounted cumulative long-term reward for following the policy from the state corresponding to the current observation).
To model the parametrized value function within the critic, use a neural network with one input layer (receiving the content of the observation channel, as specified by obsInfo) and one output layer (returning the scalar value).
Define the network as an array of layer objects, and get the dimension of the observation space from the environment specification objects. Name the network input layer criticNetInput.
CriticNet = [
featureInputLayer(prod(obsInfo.Dimension));
fullyConnectedLayer(10);
reluLayer;
fullyConnectedLayer(10);
reluLayer;
fullyConnectedLayer(1)];Convert the network to a dlnetwork object.
CriticNet = dlnetwork(CriticNet);
To display the network main characteristics, use summary.
summary(CriticNet)
Initialized: true
Number of learnables: 171
Inputs:
1 'input' 4 features
Create the critic using CriticNet and the environment specification object. Set the observation name to observation, which is the name of the criticNetwork input layer.
critic = rlValueFunction(CriticNet,obsInfo)
critic =
rlValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {6×1 cell}
State: {0×1 cell}
Check your critic using getValue to return the value of a random observation, given the current network weights.
v = getValue(critic,{rand(obsInfo.Dimension)})v = single
-0.3229
AC agents use a parametrized stochastic policy, which for discrete action spaces is implemented by a discrete categorical actor.
This actor takes an observation as input and returns as output a random action sampled (among the finite number of possible actions) from a categorical probability distribution.
To model the parametrized policy within the actor, use a neural network with one input layer (which receives the content of the environment observation channel, as specified by obsInfo) and one output layer. The output layer must return a vector of probabilities for each possible action, as specified by actInfo.
You can obtain the number of actions from the actInfo specification. Name the network output actorNetOutput.
actorNet = [
featureInputLayer(prod(obsInfo.Dimension))
fullyConnectedLayer(10);
reluLayer;
fullyConnectedLayer(10);
reluLayer;
fullyConnectedLayer(numel(actInfo.Elements)) ];Convert the network to a dlnetwork object.
actorNet = dlnetwork(actorNet);
To display the network main characteristics, use summary.
summary(actorNet)
Initialized: true
Number of learnables: 182
Inputs:
1 'input' 4 features
Create the actor using rlDiscreteCategoricalActor together with the observation and action specifications.
actor = rlDiscreteCategoricalActor(actorNet,obsInfo,actInfo)
actor =
rlDiscreteCategoricalActor with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {6×1 cell}
State: {0×1 cell}
To check your actor, use getAction to return a random action from a given observation, using the current network weights.
a = getAction(actor,{rand(obsInfo.Dimension)})a = 1×1 cell array
{[-10]}
To return the probability distribution of the possible actions as a function of a random observation and given the current network weights, use evaluate.
prb = evaluate(actor,{rand(obsInfo.Dimension)})prb = 1×1 cell array
{2×1 single}
prb{1}ans = 2×1 single column vector
0.5917
0.4083
Specify the optimization options for the actor and the critic using rlOptimizerOptions. These options control the learning of the network parameters. For both networks, set the gradient threshold to 1. For this example, set the learning rate to 0.01. For the actor network, set the learning rate to 0.05.
criticOpts = rlOptimizerOptions( ... LearnRate=1e-2,... GradientThreshold=1); actorOpts = rlOptimizerOptions( ... LearnRate=5e-2,... GradientThreshold=1);
Specify agent options, including the objects previously created for both actor and critic.
agentOpts = rlACAgentOptions(... NumStepsToLookAhead=32,... DiscountFactor=0.99,... CriticOptimizerOptions=criticOpts,... ActorOptimizerOptions=actorOpts);
Create an AC agent using the actor, the critic and the agent options object.
agent = rlACAgent(actor,critic,agentOpts)
agent =
rlACAgent with properties:
AgentOptions: [1×1 rl.option.rlACAgentOptions]
UseExplorationPolicy: 1
ObservationInfo: [1×1 rl.util.rlNumericSpec]
ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
SampleTime: 1
To check your agent, use getAction to return a random action from a given observation, using the current actor and critic network weights.
act = getAction(agent,{rand(obsInfo.Dimension)})act = 1×1 cell array
{[-10]}
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
For additional examples showing how to create actors and critics for different agent types, see Compare DDPG Agent to LQR Controller and Train DQN Agent to Balance Discrete Cart-Pole System.
Create a finite set observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment with a discrete observation space). For this example, define the observation space as a finite set consisting of four possible values 1, 3, 5 and 7.
obsInfo = rlFiniteSetSpec([1 3 5 7]);
A value-function critic takes the current observation as input and returns a single scalar value as output (the estimated discounted cumulative long-term reward for following the policy from the state corresponding to the current observation).
Since both observation and action spaces are discrete and low-dimensional, use a table to model the value function within the critic. rlTable creates a value table object from the observation and action specifications objects.
vTable = rlTable(obsInfo);
The table is a column vector in which each entry stores the value of the corresponding observation, under the given policy. You can access the table using the Table property of the vTable object. The initial value of each element is zero.
vTable.Table
ans = 4×1
0
0
0
0
You can also initialize the table to any value, in this case, an array containing all the integers from 1 to 4.
vTable.Table = reshape(1:4,4,1)
vTable =
rlTable with properties:
Table: [4×1 double]
Create the critic using the table and the observation specification object.
critic = rlValueFunction(vTable,obsInfo)
critic =
rlValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlFiniteSetSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {[4×1 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation, using the current table entries.
v = getValue(critic,{7})v = 4
Obtain values for a random batch of 8 observations.
v = getValue(critic,{[1 3 5 7 7 5 3 1]})v = 1×8
1 2 3 4 4 3 2 1
Get the seventh value in the batch.
v(7)
ans = 2
You can now use the critic (along with an actor) to create an agent for the environment described by the given observation specification object. Examples of agents that can work with discrete observation spaces, and use a value function critic, are rlACAgent, rlPGAgent, rlPPOAgent. rlTRPOAgent does not support actors or critics that use tables.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous four-dimensional space, so that there is a single observation channel that carries a column vector containing four doubles.
obsInfo = rlNumericSpec([4 1]);
A value-function critic takes a batch of observations as input and returns a corresponding batch of scalars as output (each element in the batch is the estimated discounted cumulative long-term reward for following the policy from the state corresponding to the observation).
To model the parametrized value function within the critic, use a custom basis function. Create a custom function that returns a vector of three elements, given an observation as input. Here, the third dimension is the batch dimension. For each element of the batch dimension, the output of the basis function is a vector of three elements.
myBasisFcn = @(myobs) [
myobs(2,1,:).^2;
myobs(3,1,:)+myobs(1,1,:);
abs(myobs(4,1,:))
]myBasisFcn = function_handle with value:
@(myobs)[myobs(2,1,:).^2;myobs(3,1,:)+myobs(1,1,:);abs(myobs(4,1,:))]
The output of the critic is the scalar W'*myBasisFcn(myobs), which represents the estimated value of the observation under the given policy. Here W is a weight column vector which must have the same size as the custom basis function output. The elements of W are the learnable parameters.
Define an initial parameter vector.
W0 = [3;5;2];
Create the critic. The first argument is a two-element cell containing both the handle to the custom function and the initial weight vector. The second argument is the observation specification object.
critic = rlValueFunction({myBasisFcn,W0},obsInfo)critic =
rlValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {[1×3 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation, using the current parameter vector.
v = getValue(critic,{[2 4 6 8]'})v = 104
Obtain values for a random batch of 10 observations.
v = getValue(critic,{rand(4,1,10)});Get the seventh value in the batch.
v(7)
ans = 6.9592
You can now use the critic (along with an actor) to create an agent for the environment described by the given observation specification object. Examples of agents that can work with continuous observation spaces, and use a value function critic, are rlACAgent, rlPGAgent, rlPPOAgent. rlTRPOAgent does not support actors or critics that use custom basis functions.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an environment and obtain observation and action information.
env = rlPredefinedEnv("CartPole-Discrete");
obsInfo = getObservationInfo(env);A value-function critic takes the current observation as input and returns a single scalar value as output (the estimated discounted cumulative long-term reward for following the policy from the state corresponding to the current observation).
To model the parametrized value function within the critic, use a recurrent neural network with one input layer (receiving the content of the observation channel, as specified by obsInfo) and one output layer (returning the scalar value).
Define the network as an array of layer objects. To create a recurrent network, use a sequenceInputLayer as the input layer (with size equal to the number of dimensions of the observation channel) and include at least one lstmLayer.
myNet = [
sequenceInputLayer(obsInfo.Dimension(1))
fullyConnectedLayer(8)
reluLayer
lstmLayer(8)
fullyConnectedLayer(1)
];Convert the network to a dlnetwork object.
dlNet = dlnetwork(myNet);
Display a summary of network characteristics.
summary(dlNet)
Initialized: true
Number of learnables: 593
Inputs:
1 'sequenceinput' Sequence input with 4 dimensions
Create a value function representation object for the critic.
critic = rlValueFunction(dlNet,obsInfo)
critic =
rlValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
Normalization: "none"
UseDevice: "cpu"
Learnables: {7×1 cell}
State: {2×1 cell}
To check your critic, use getValue to return the value of a random observation, using the current network weights.
v = getValue(critic,{rand(obsInfo.Dimension)})v = single
0.0017
You can use dot notation to extract and set the current state of the recurrent neural network in the critic.
critic.State
ans=2×1 cell array
{8×1 dlarray}
{8×1 dlarray}
critic.State = {
-0.1*dlarray(rand(8,1))
0.1*dlarray(rand(8,1))
};To evaluate the critic using sequential observations, use the sequence length (time) dimension. For example, obtain actions for 5 independent sequences each one consisting of 9 sequential observations.
[value,state] = getValue(critic, ...
{rand([obsInfo.Dimension 5 9])});Display the value corresponding to the seventh element of the observation sequence in the fourth sequence.
value(1,4,7)
ans = single
0.0769
Display the updated state of the recurrent neural network.
state
state=2×1 cell array
{8×5 single}
{8×5 single}
You can now use the critic (along with an actor) to create an agent for the environment described by the given observation specification object. Examples of agents that can work with continuous observation spaces, and use a value function critic, are rlACAgent, rlPGAgent, rlPPOAgent. rlTRPOAgent does not support actors or critics that use recurrent networks.
For more information on input and output format for recurrent neural networks, see the Algorithms section of lstmLayer. For more information on creating approximator objects such as critics and critics, see Create Policies and Value Functions.
Create a finite-set observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as an hybrid (that is mixed discrete-continuous) space with the discrete channel carrying a single observation labeled 7, 5, 3, or 1, and the second one being a vector over a continuous three-dimensional space.
obsInfo = [rlFiniteSetSpec([7 5 3 1]) rlNumericSpec([3 1])];
A value-function critic takes a batch of observations as input and returns as output a corresponding batch of scalars, each representing the estimated discounted cumulative long-term reward (the value) obtained by following the policy from the state corresponding to the given observation.
To model the parametrized value function within the critic, use a custom basis function. Create a custom function that returns a vector of four elements, given the content of the two observation channels as input. Note that the first channel carries a scalar (one row and one column) but the respective myBasisFcn input has also the batch dimension. Similarly, the second channel carries a vector with three elements, but it has the same batch dimension as the first channel. The sequence dimension is not supported for stateless approximators.
myBasisFcn = @(obsDisc,obsCont) [
obsDisc(1,1,:) + obsCont(1,1,:);
obsDisc(1,1,:) - obsCont(2,1,:);
obsDisc(1,1,:).^2 + obsCont(3,1,:);
obsDisc(1,1,:).^2 - obsCont(3,1,:)
];The output of the critic is the scalar W'*myBasisFcn(observation), which represents the estimated value of the observation under the given policy. Here W is a weight column vector which must have the same size as the custom basis function output. The elements of W are the learnable parameters.
Define an initial parameter vector.
W0 = ones(4,1);
Create the critic. The first argument is a two-element cell containing both the handle to the custom function and the initial weight vector. The second argument is the observation specification object.
critic = rlValueFunction({myBasisFcn,W0},obsInfo)critic =
rlValueFunction with properties:
ObservationInfo: [2×1 rl.util.RLDataSpec]
Normalization: ["none" "none"]
UseDevice: "cpu"
Learnables: {[1×4 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation, using the current parameter vector.
v = getValue(critic,{5,[0.1 0.1 0.1]'})v = 60
Note that the critic does not enforce the set constraint for the discrete set element.
v = getValue(critic,{-3,[0.1 0.1 0.1]'})v = 12
Obtain values for a random batch of 5 observations.
getValue(critic,{ ...
rand([obsInfo(1).Dimension 5]), ...
rand([obsInfo(2).Dimension 5]) ...
})ans = 1×5
2.8718 0.6859 0.4322 1.5246 2.9352
You can now use the critic (along with an actor) to create an agent for the environment described by the given observation specification object. Examples of agents that can work with mixed observation spaces, and use a value function critic, are rlACAgent, rlPGAgent, rlPPOAgent. rlTRPOAgent does not support actors or critics that use custom basis functions.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Version History
Introduced in R2022a
