Policy

Reinforcement learning policy

Since R2022b

• 

Libraries:
Reinforcement Learning Toolbox

Description

Use the Policy block to simulate a reinforcement learning policy in Simulink^® and to generate code (using Simulink Coder™) for deployment purposes. This block takes an observation as input and returns an action as output. You associate the block with a MAT file that contains the information needed to fully characterize the policy. The MAT file can be generated by the generatePolicyFunction or generatePolicyBlock function.

Examples

Generate Policy Block for Deployment

Generate a policy block to deploy a trained policy.

• Open Live Script

Ports

Input

observation — Environment observations

scalar | vector | nonvirtual bus

This port receives observation signals from the environment. Observation signals represent measurements or other instantaneous system data. If you have multiple observations, you can use a Mux block to combine them into a vector signal. To use a nonvirtual bus signal, use the bus2RLSpec function.

update parameters — Enable update of policy parameters

scalar

When the received scalar signal has a value greater than zero, the parameters of the Policy block update with the values received from the parameters port.

Dependencies

To enable this port, select the Policy parameter inputs parameter.

parameters — Input

bus

Use this port to supply the updated values of the policy parameters. The port must receive a bus signal with the structure and data type of the bus object that the block creates when you enable the Policy parameter inputs parameter. The parameters of the Policy block update with the values received from this port when the update parameters port receives a value greater than zero.

Note

This port can accept the structure generated by the policyParameters function.

Dependencies

To enable this port, select the Policy parameter inputs parameter.

Output

action — Policy action

scalar | vector | nonvirtual bus

Action computed by the policy based on the observation input. Connect this port to the inputs of your system. To use a nonvirtual bus signal, use the bus2RLSpec function.

Note

Policy blocks generated from a continuous action-space rlStochasticActorPolicy object or a continuous action-space rlACAgent, rlPGAgent, or rlPPOAgent object do not enforce the constraints set by the action specification. For these blocks you must enforce action-space constraints within the environment.

Parameters

Policy block data MAT file — Policy block data MAT file

blockAgentData.mat (default) | file name

Enter the name of the MAT file containing the information needed to fully characterize the policy. This file is generated by generatePolicyFunction or generatePolicyBlock. When you generate the block using generatePolicyBlock and specify a non-default dataFileName argument, then the generated block has this parameter set to the specified filename, so that the block is associated with the generated data file.

To use a Policy block within a conditionally-executed subsystem, such as a Triggered Subsystem (Simulink) or a Function-Call Subsystem (Simulink), you must generate its data file from an agent or policy object that has its SampleTime property set to -1. Doing so allows the block to inherit the sample time of its parent subsystem.

Programmatic Use

Block Parameter: MATFile

Type: string, character vector

Default: "blockAgentData.mat"

Policy parameter inputs — Add policy parameters input ports and create bus object and parameter structure

off (default) | on

Select this parameter to add the update parameters and parameters input ports to the block. When you select this parameter and click Apply, the block creates both a bus object and a structure in the global workspace of the model (typically the MATLAB^® workspace).

• The bus object specifies the dimensions and data types of the bus that you must connect to the parameters port. You can specify the name of the bus object using the Policy parameter bus name parameter.

• The structure is consistent with the data returned by the policyParameters function, and contains values taken from the MAT file specified in Policy block data MAT file.

Click Generate IC blocks to automatically generate initial condition blocks referencing the initial policy parameters.

Programmatic Use

Block Parameter: PolicyParamsAsInput

Type: string, character vector

Values: "off", "on"

Default: "off"

Policy parameter bus name — Name of policy parameters bus object

policy_param_bus (default) | file name

Enter the name of the bus object that the block creates when you select Policy parameter inputs and click Apply. The block creates the bus object in the global workspace of the model, which is typically the MATLAB workspace.

Programmatic Use

Block Parameter: PolicyParamsBusName

Type: string, character vector

Default: "policy_param_bus"

Generate IC blocks — Generate blocks containing initial conditions for the policy parameters

button

Click this button to generate Constant blocks, containing initial conditions, that connect to the update parameters and parameters ports. The block connected to the update parameters port returns as output a false logical value. The block connected to the parameters port returns as output the structure containing the policy parameters created in the workspace when you click the Apply button.

Tips

When using Embedded Coder^® to generate parallel code, enable the Generate parallel for loops optimization parameter to improve the performance when the data being processed is large in size. However, if the network and the data is small, the overhead of initializing the threads for parallelization significantly reduces the performance. In this case, disable Generate parallel for loops. See Generate parallel for-loops (Embedded Coder) and coder.MexCodeConfig (MATLAB Coder) for more information.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.

Version History

Introduced in R2022b