PPO强化学习时动作空间超出设定的数据范围

在运用PPO方法进行强化学习训练时，我的动作空间设定如下：

numAct = 1;

actInfo = rlNumericSpec([numAct 1],"LowerLimit",-1,"UpperLimit",1);

actInfo.Name = "kvalue";

但是我在训练过程中运用SCOP发现动作输出严重超出了这个范围，这个原因是什么呢？

Answer 1

[Aiswarya]

请注意，我将用英语回答这个问题。

(Please note that I will be answering the question in English.)

I understand that you are using the PPO agent and trying to set the LowerLimit and UpperLimit of data space as ActInfo using rlNumericSpec . But the action output is clearly not in the range specified by you.

The action bounds depend on the type of agent. On-policy agents like PPO don’t enforce constraints set in the action specification (specified using rlNumericSpec) for continuous action spaces . If you want to enforce these limits you have to do it explicitly on the environment side.

This is because these agents use Gaussian distributions (mean and standard deviation are outputs from the actor-network) to sample exploration actions. Mean is bounded by tanh and scaling layers, but we sample actions from the unbounded Gaussian distribution. Thus actions can be outside of the limits during training. One option is to set the agent.UseExplorationPolicy = false after training, so the agents can use only mean, and the actions are always within limits.

The rlPPOAgent documentation also mentions the fact that action bounds need to be set by the user within the environment:

Proximal policy optimization (PPO) reinforcement learning agent - MATLAB (mathworks.com)

Also note that this is not the case with agents like SAC, for which the action bounds can be enforced with `rlNumericSpec` .

I hope this helps.

Regards,

Aiswarya

[郭欣]

I got it ! thanks !!!