Just Curious, when I run parallel computing with k cores, why there are k matlab background processes in the task manager on my machine?

I'm recently running some simulations using matlab parallel computing with 100 cores(distributed across 10 nodes) on a computing cluster of my university. However, I received warnings about consuming too much CPUs on the node where I ran my file(after logging in). I checked the task manager and found that there are 100 matlabs launched as background processes, each one consuming an equal amount of CPU on this node(that's why total CPU consumption quickly blow up). Once I shut down 'parpool', those background processes disappeared. Just curious about what is going on there, why does matlab start a background process for each parallel on my machine? What are these background processes for?

Answer 1

[Matt J]

That's how parallel labs in the Parallel Computing Toolbox work. Essentially, they each run their own separate MATLAB session.

[Tmat]

Ah, I see. Thank you so much, Matt! Is there a way to do parallel computing without triggering those background processes(so that I won't use too much CPUs on one node)?

[Matt J]

I think the 100 background processes are inevitable, since they are running your 100 parallel jobs. The question is whether your entire parpool is running on your host node. You probably want to distribute some workers to other nodes. I suspect you need to dig into cluster profiles to do that,

https://www.mathworks.com/help/distcomp/discover-clusters-and-use-cluster-profiles.html#btmmlnd