i have unstable second order system and trying to make it follow square wave using reinforcement learning but the agent doesn't converge

i have unstable second oreder system with 2 poles at 2 ,3 and i have square wave as refrence and trying to make the system follow the ref. the observation are x1,x2 and error and the reward function is -rms(error) and the error signal is (ref-x2) as x2 is the output

the A matrix [0 -6;1 5] B=[10;1] C[0 1] D[0]

[farouk]

@Emmanouil Tzorakoleftherakis any help

[Emmanouil Tzorakoleftherakis]

I would start by looking at how the problem is set up in similar examples like the ones below. Take a look at the reward and hyperparameters used:

https://www.mathworks.com/help/reinforcement-learning/ug/train-ddpg-agent-to-swing-up-and-balance-pendulum.html

https://www.mathworks.com/help/reinforcement-learning/ug/water-tank-reinforcement-learning-environment-model.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-ddpg-agent-to-balance-double-integrator-system.html

[farouk]

@Emmanouil Tzorakoleftherakis at the above examples according to what we select the (action limits, step time, number of neurons)?

[Sam Chak]

@farouk and @farouk,

Does applying action (in the unrestricted solution space) from -Inf to Inf to the unstable system help RL to converge?

Is the default number of neurons, 256 not sufficient?

Is training the RL Agent to stabilize this simple unstable 2nd-order system really challenging? Just curious...

[farouk]

i have difficulty to stablizie second order system have poles higher than 0.5

[Sam Chak]

Hi @farouk, @farouk, Thinking from the mathematical perspective, what are the manipulated variables in RL that you think can be tuned (maybe with GA)?

Does the difficulty have something to do with Learning Rate of RL (which I read about in some documentation)?

What knowledge and theorem are required so that we can deterministically set the values to make the agent converge?

Answer 1

[Sam Chak]

@farouk

I don't know how many iterations your RL will take to stabilize the system and track the square wave. But you can probably at least use the generated data for performance comparison purposes.

% 2nd-order system
A   = [0 -6; 1 5]; B = [10; 1]; C = [0 1]; D = [0]; 
sys = ss(A,B,C,D);
Gs  = tf(sys)

Gs =
 
     s + 10
  -------------
  s^2 - 5 s + 6
 
Continuous-time transfer function.

% Manipulated Variable (MV)
s   = tf('s');
Gv  = 20*(s^2 - 5*s + 6)/(s*(s + 10))

Gv =
 
  20 s^2 - 100 s + 120
  --------------------
       s^2 + 10 s
 
Continuous-time transfer function.

% Closed-loop system
Gcl = minreal(feedback(Gv*Gs, 1))

Gcl =
 
    20
  ------
  s + 20
 
Continuous-time transfer function.

% Tracking a square wave reference signal
tau = 2;
[u, t] = gensig("square", tau, 4);
lsim(Gcl, u, t)
ylim([-1 2]), grid on

% Output signal of MV
Gu = minreal(feedback(Gv, Gs))

Gu =
 
  20 s^4 - 200 s^3 + 740 s^2 - 1200 s + 720
  -----------------------------------------
    s^4 + 25 s^3 + 56 s^2 - 820 s + 1200
 
Continuous-time transfer function.

lsim(Gu, u, t), grid on

The system output can track the square wave reference input. It takes approximately 0.25 s to converge to the reference signal without overshoot.