Run reinforcement learning Agent on raspberry

Hi,

my question is, how to run and train an reinforcement learning Agent on a raspberry pi.

I am having a Simulink model which uses an DDPG-agent from the reinforcement learning toolbox to learn a Sine-wave (on a raspberry and locally). I upload the Simulink model (sine_learning.slx. Raspberry_USB.slx).

The Raspberry reads a Sine-wave and outputs an action. The learning process should cause an convergence of the action with the sine-wave.

I have upladed the files used to train the model (with and without raspberry connection).

Only 1 file is run to train the net:

createSineAgent.m to train an agent locally

and

createUSBSineAgent.m to train on the raspberry.

The referenced Simulink models are sine_learning.slx and Raspberry_USB.slx. Two other files define the net-form and other training parameters.

The model is trained on a computer by exchanging the Serial-Read with a sine-wave-Simulink-block . During this the RL-agent is saved as a ".mat" file.

Now i want to execute this model on the raspberry. Running "create_USBAgent.m" directly does not seem to work. The training process is no initialized.

I heard i need to deploy the RL-Agent .

See:

https://www.mathworks.com/help/reinforcement-learning/ug/deploy-trained-reinforcement-learning-agents.html

I followed this instruction and generated an evaluatePolicy_mex.mexw64 file (according to - "C/C++ code for table, deep neural network, or linear basis function policies using MATLAB® Coder™" - see link

My question is, how can this Mex-file replace the network on the raspberry? Also is it possible to deploy an RLAgent on the board which continues to learn and improve its policy?

Alternatively, are there other ways to achieve my goal (running and training an RLAgent on a raspberry)?

Thank you for any tip or hint!!

Best regards,

V

Answer 1

[Sean de Wolski]

You won't want to compile the MEX file to run on the Pi because it requires MATLAB. Instead generate code and recompile it for the Pi. Or bake that into the Simulink model (though a MATLAB function or system) and then when you generate code for the model, that part will come along (recommended workflow).

You can't generate code for training on the Pi, inference/prediction only.

[Victor Bayer]

"Or bake that into the Simulink model (though a MATLAB function or system) and then when you generate code for the model, that part will come along (recommended workflow)"

This is how i would like to do it. Call the Mex. File in a Matlab function block.

However i can't find any documentation on how that is possible.

Do you have any tips on that?

[Victor Bayer]

If i understand correctly, you are proposing generating C-code for the RLAgent and calling it (not the mex file) in Simulink.

Similar to:

https://www.mathworks.com/matlabcentral/answers/757154-how-to-implement-reinforcement-learning-using-code-generation/?s_tid=ans_lp_feed_leaf

[Walter Roberson]

There are two things referred to as mex files.

1. files that end in .mex* file extensions. These are code that has been compiled to shared libraries (such as .dll or .so) for the architecture of the machine they are compiled on. The only cross-compiling that was ever supported was compiling on 64 bit windows for use on 32 bit windows. I might have missed some development for MacOS but as far as I know, compiling to ARM is not supported on any system. Note that these files require MATLAB desktop or at least MCR to run, and MCR is not available for ARM.
2. sometimes mex files are used to refer to C or C++ (or Fortran) source code that has been written using MATLAB headers and using MATLAB library calls. In theory you could compile those to ARM. However they require an implementation library for the target system, and that is not available for ARM.

All of this is distinct from using a Simulink project with Simulink Coder technology to generate C / C++ targeting Raspberry or Arduino, or using MATLAB Coder to generate C++ targeting Raspberry (Arduino not supported). These routes do not require MCR... for the library functions they support.