References to multi-agent reinforcement learning schemes in the reinforcement learning toolbox

Can somebody provide several references on multi-agent reinforcement learning schemes in reinforcement learning toolbox？

[Umar]

Hi Lin,

Plese refer to following links regarding multi-agent reinforcement learning,

https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-play-turn-based-game.html https://www.mathworks.com/help/releases/R2023b/reinforcement-learning/ref/rl.env.rlmultiagentfunctionenv.html

Please let me know if you have any further questions.

[Lin]

Thank you for you answer!Unfortunately, I could not find any reference even after your advices.I'm looking for references & informations related to the "LearningStrategy:centralized&decentralized" property of object "rlMultiAgentTrainingOptions".Please let me know if you have further tips.Thanks again!

Answer 1

[Emmanouil Tzorakoleftherakis]

The following examples are all on multi-agent reinforcement learning (centralized or decentralized):

https://www.mathworks.com/help/reinforcement-learning/ug/train-agents-for-path-following.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-3-agents-for-area-coverage.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-2-agents-to-collaborate.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-play-turn-based-game.html

[Lin]

Thank you for you answer!Unfortunately, I could not find any reference even after your advices.I'm looking for references & informations related to the "LearningStrategy:centralized&decentralized" property of object "rlMultiAgentTrainingOptions".Please let me know if you have further tips.Thanks again!

[Umar]

Hi @Lin,

The centralized critic provides feedback to all agents based on the collective experience of the entire team. This approach enables agents to coordinate their actions effectively and learn from a global perspective. Here is an example of how you can set the "LearningStrategy" property to "centralized" in MATLAB:

options = rlMultiAgentTrainingOptions('LearningStrategy', 'centralized');

Conversely, in a decentralized learning strategy, each agent learns independently based on its local observations and rewards. There is no centralized critic providing feedback to all agents. Each agent makes decisions autonomously without considering the global state of the environment. This approach is useful when agents have limited communication or when scalability is a concern.To set the "LearningStrategy" property to "decentralized" in MATLAB, you can use the following code snippet:

options = rlMultiAgentTrainingOptions('LearningStrategy', 'decentralized');

To showcase the "LearningStrategy" property of the "rlMultiAgentTrainingOptions" object in Matlab, I will create a simple example that involves setting up a multi-agent training scenario with both centralized and decentralized learning strategies and then visualize the training progress using plots. So, first setting up the environment by defining the environment and agents for our multi-agent system. I will create a simple environment with two agents.

% Define the environment

env = rlPredefinedEnv("SimpleMultiAgentEnvironment");

% Define the agents

agent1 = rlQAgent(env);

agent2 = rlQAgent(env);

Note: rlPredefinedEnv requires Reinforcement Learning Toolbox.

Then, creating the multi-agent training options by creating the "rlMultiAgentTrainingOptions" object and set the learning strategy to both centralized and decentralized.

% Create multi-agent training options

multiAgentOpts = rlMultiAgentTrainingOptions;

multiAgentOpts.LearningStrategy = ["centralized", "decentralized"];

Now, training the agents using the defined options and visualize the training progress.

% Train the agents

trainingStats = trainMultiAgent(agent1, agent2, env, multiAgentOpts);

% Plot the training progress

figure;

subplot(2,1,1);

plot(trainingStats.CumulativeReward);

title('Cumulative Reward');

subplot(2,1,2);

plot(trainingStats.ActorLoss);

hold on;

plot(trainingStats.CriticLoss);

legend('Actor Loss', 'Critic Loss');

title('Actor and Critic Loss');

Finally, observing the results after running the code, you will see two plots showing the cumulative reward and the actor/critic losses for both centralized and decentralized learning strategies. This detailed implementation demonstrates how to utilize the "LearningStrategy" property of the "rlMultiAgentTrainingOptions" object in Matlab to train agents with different strategies and visualize their training progress effectively. Feel free to customize the environment, agents, and training options to explore more complex multi-agent scenarios and further enhance your understanding of centralized and decentralized learning strategies in reinforcement learning.By understanding the nuances of the "LearningStrategy" property and its implications for multi-agent training, you can tailor your approach to suit the specific requirements of your reinforcement learning scenario. Please let me know if you have any further questions.

[Lin]

Hello @Umar,

Thank you so much for your assistance.Your instructions are very helpful to me.“The centralized critic provides feedback to all agents based on the collective experience of the entire team.”I want to get this sentence related to the paper.It would be perfect if you could provide relevant papers.Thanks again!

[Emmanouil Tzorakoleftherakis]

Adding a visual of the explanationm above. There are many papers on scholar if you search for centralized/decentralized MARL and I am sure there will be slight variations from paper to paper on the mechanics of it. What we provide in Reinforcement Learning Toolbox is one potential option.

[Umar]

Hi @Lin,

It sounds like Emmanouil Tzorakoleftherakis provided his feedback to your recent posted comments. If you still have any further questions for us, please let us know, we will be more happy to assist you.

[Lin]

@Umar@Emmanouil Tzorakoleftherakis

Thank you very much for your advice, your help has been very helpful to me!

[Umar]

@Lin,Thank you for your kind words and feedback. I am glad to hear that our advice provided was helpful to you. If you have any more questions or need further assistance, please do not hesitate to reach out.

[Lin]

@Emmanouil Tzorakoleftherakis

Can you tell me which documentation this picture is from?Thank you very much.

[Emmanouil Tzorakoleftherakis]

It's from my own notes, not publicly available

[Umar]

Hi @Lin,

To delve deeper into the topic and explore relevant research papers, you may consider the following scholarly articles, hope they will provide information relevant to sentence, “ The centralized critic provides feedback to all agents based on the collective experience of the entire team ”.

Title:” Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning”

https://arxiv.org/abs/2102.04402

Title: "Enhancing Multi-Agent Coordination through Common Operating Picture Integration”

https://arxiv.org/abs/2311.04740

These papers delve into the theoretical foundations and practical implications of utilizing a centralized critic in multi-agent systems, shedding light on the benefits and challenges associated with this approach. By studying these resources, you can gain a comprehensive understanding of how centralized critics operate and their impact on the performance of multi-agent systems. Please let me know if you have any further questions.

[Lin]

Hello @Umar@Emmanouil Tzorakoleftherakis,

I am very grateful for your help. Your answers have resolved my doubts and filled the gap of lacking references on multi-agent reinforcement learning in Matlab. Thank you again.

[Umar]

@Lin,

Glad to help out again, please feel free to ask any questions if you still need any help.