GPU Coder used, but got error: Error generated while running CUDA-enabled program: [700,cudaErrorIllegalAddress]
8 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
I'm using GPU coder to accelerate N-D interpolation function interpn().
Everything worked fine when my problem scale was small, i.e. interpolating in 1-D or 2-D dimensions. But when I throw a larger problem, of 5-D interpolation to it, MATLAB threw the following error message:
Error generated while running CUDA-enabled program: [700,cudaErrorIllegalAddress] an
illegal memory access was encountered. To generate additional debugging information,
enable the 'SafeBuild' GPU code generation option.
I followed the error message suggestion above and added the 'SafeBuild' GPU code generation option, and got the following error instead:
Error generated while running CUDA-enabled
program: [46,cudaErrorDevicesUnavailable] all
CUDA-capable devices are busy or unavailable
I know that when I wasn't using GPU Coder, the code still worked but kind of slow. But now when GPU coder is enabled, it is not working. I have NVIDIA RTX 2080ti with 11G memory. When I was using CPU to do the calculation, the problem took 1 hour and peak RAM use was around 40GB out of 64GB RAM. Could this error be because I ran out of GPU memory by establishing my problem too large? How can I tell if this is the case?
OK update: I rebooted my Windows PC, and ran the same program again, the following error showed instead:
Error generated while running CUDA-enabled
program: [2,cudaErrorMemoryAllocation] out of
memory
I think this confirms that indeed my problem is causing GPU to run out of memory just to start the problem. Now I know the problem, is there any suggestions how to split the problem for interpn? Thanks!
3 Commenti
Tong Zhao
il 2 Ago 2021
Hi Hari,
Thank you for your attention. I got answer to this problem with some help. Indeed my problem was so big that it created arrays with size larger than 4GB (double arrays with size 2^32) to be interpolated. There's a limit to how large the indices of an array could be (int32), I think this might be why. I'm still curious though as to how one could expand the size limit to beyond (int32).
First
il 7 Dic 2021
hi may i know the solution ? i met same error of cuda out of memory but its quite weird. Same code can run cpu1+gtx980 but error above when i run on cpu2+gtx980 and cpu2 has more much more cores and memory than cpu1.
Risposte (1)
Gautam
il 19 Gen 2023
Modificato: Gautam
il 19 Gen 2023
Hi,
The error is most likely that a matrix is growing beyond it's intended size. Take a careful look at your code to analyze which matrix is growing beyond the intended value.
Growing size matrices are silently handled by MATLAB, but the coder generates CUDA C which does not handle growing sizes so it is mostly seg-faulting when you run the generated code.
Try running your code in MATLAB and check if every matrix at the end of its computation is having the intended size.
Also, when you run into a "cudaErrorIllegalAddress" you have to restart MATLAB and run your code. This is because, when the coder runs the generated code for the first time, it creates a CUDA context and it reuses the same context in the subsequent runs till the MATLAB application is closed. Now, when you encounter a seg-fault when running the generated code, the context gets corrupted, MATLAB doesn't know about it and so it tries to run the same context in the next run, which will fail. So the solution is to restart MATLAB.
Gautam
0 Commenti
Vedere anche
Categorie
Scopri di più su GPU Computing in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!