The CUDA driver must recompile the GPU libraries

Hi everyone, I'm now using MATLAB 2020a, when I tried to train the CNN, I met a warning: The CUDA driver must recompile the GPU librariesbecause your device is more recent than the libraries. Recompiling can take several minutes.

My GPU is RTX 3090, I have checked my CUDA driver and reinstalled the CUDA10.1 driver, I also tried MATLAB 2020b version, but the warning still exist.

Can anyone help me? Thanks a lot!

Answer 1

[Joss Knight]

There isn't really much to say. This 'forward compatibility' is explained in the GPU Computing Requirements documentation. As it says, just allow the NVIDIA driver to compile, and when it's done, that should be it. You may need to increase the CUDA_CACHE_MAXSIZE environment variable, if you find this compilation is happening repeatedly.

The warning will continue to be displayed once per session, even if you experience no compilation. You can just suppress that warning if you don't want to see it.

[Jianyu Zhao]

Thanks mate, I saw the release document, so acutally cuda11 is not supported by MATLAB, the only way is to wait supported version:)or just wait the cuda recompile the GPU libraries.

[Joss Knight]

FWIW once you've waited for the compilation to finish (which might take over an hour) I've found that most things work. And once the compiled binaries are cached it shouldn't happen again. There are some problems with cuDNN though; in my estimation Deep Learning should not be risked unless you can confirm it's giving you right answers.

[Jianyu Zhao]

I just tested my GPU with an example from MATLAB Documentation, I think the RTX3090 may have some problems when training the network. It is exactly the same code, but my 3090 GPU could generate a completely wrong result.

The example's name is Transfer Learning Using AlexNet, you can find it in MATLAB Documentation.

The first image is from MATLAB documentation, it's the right training result.

The second one is from my training, it's the wrong training result from my RTX3090.

[Jianyu Zhao]

And I also found that if I trained exactly the same network with the same data, but using CPU or GPU, their results are also different, do you have any ideas?

[Walter Roberson]

Deep Learning for the Ampere GPUs is not correct in R2020b, due to bugs in the NVIDIA libraries.

(I do not have any information about when it is expected to be fixed.)

[Joss Knight]

Sorry, I don't really have any more information for you than I've already given. Ampere is not supported, so wrong answers are not a surprise.

[Jianyu Zhao]

Okay Thanks Joss Knight, I understand. I will try other ways to do my research and wait for the version which could support Ampere. Thank you for you reply!