GPU temperature limit control

I'm running extensive CUDA code on 3 GPUs and eventually found out that one GPU stops working due to reaching the temperature limit (95C). Is there a way that I can adapt/decrease the GPU temperature limit programmatically?

Answer 1

[Jason Ross]

You can change some GPU settings via the nvidia-smi utility. One of them is the power cap. Perhaps this can act as a proxy to reducing the temperature? There are a number of ways to monitor the temperature using the utility, but on a quick scan I don't see a way to set a lower temperature limit directly. You might be able to do this via the desktop settings app they ship. On Windows it's in Control Panel and on Linux, it's nvidia-settings.

I'd also suggest seeing if you can get more cooling in your machine. There might be some free/open fan headers, and fans are pretty inexpensive to buy.

The relevant nvidia-smi entry is here:

 SW Power Cap SW Power Scaling algorithm is reducing the clocks below
 requested clocks because the GPU is consuming too much
 power. E.g. SW power cap limit can be changed with

 nvidia-smi --power-limit=

Full nvidia-smi documentation is here: http://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf