MATLAB Answers

Computer crashed when calculating matrix (mixed with sparse and full) multiplication and summation

4 views (last 30 days)
Paula
Paula on 15 Jun 2021
Answered: James Tursa on 16 Jun 2021
In a Matlab script, I have to multiply a large sparse matrix with a full matrix, as follows:
A1 = B * C.
B is a full matrix, its size is 5000-by-4800 double. C is a sparse matrix, its size is 4800-by-3E5 sparse double, and its nnz/numel = 2.08E-4, so it’s very sparse.
My desktop computer crashed at this line of code. I thought it was because matrix C is too large, so I “broke down” matrix C along its columns, and let them multiply matrix B one by one, such as follows:
for j=1:3E5
Aj=B*C(:,j);
A1(:,j)=Aj;
end
The computer did not crash this time, it was able to calculate the multiplication and provide matrix A1. However, it was slow (elapsed time was about 10 seconds), and the “Physical Memory Usage” in the Task Manager was alarmingly approaching the maximum.
Immediately following this “for” loop, I have one more line of code, which is a simple summation of two matrices:
A = A1 + A2.
Matrix A1 (a full matrix) is obtained from above. Matrix A2 is a sparse matrix of the same size (5000-by-3E5), it is very sparse.
As I mentioned, the computer did not crash calculating the “for” loop. But it crashed at this summation line of code afterwards. Could it be because too much “Physical Memory Usage” was used up already prior to the summation?
In conclusion, the whole thing is a mess, and I appreciate any help to be able to calculate these large matrix multiplication and then summation. Or could it be that the matrix C is simply too large and there’s no way to solve it?
Thank you!
-------------------------
Update: I just tried something else: Without running any code, I entered in the Command window: A1 = zeros(5000, 3E5). This simply creates a full matrix of zeros, and I see that the “Physical Memory Usage” in the Task Manager immediately goes up to about 14GB, approaching the maximum (16GB).
Does this mean a full matrix that has 3E5 number of columns is just too big for the computer to handle, and there's nothing I can do to solve it?
  6 Comments
Paula
Paula on 16 Jun 2021
Understood. Thank you very much for the help and suggestions!
I would like to "accept" your answer to end this question, but I don't know how to. I think it's because you answered as a "Comment", not as an "Answer"? But I'm not sure, sorry.
If you don't mind clicking on "Answer this question", and copy pasting one of your answers to there, I would gladly accept it.

Sign in to comment.

Accepted Answer

James Tursa
James Tursa on 16 Jun 2021
Remember that each element of the B*C result is simply the dot product of a row of B with a column of C. If that column of C has any non-zeros in it (even if it is only one non-zero), then in general you would expect the dot product to be non-zero, the only exception being if you had very fortunate cancellation. The fact that sum(any( C )) = 3E5 for your particular case means that all of the columns of C have at least one non-zero, so the result of B*C would be expected to be essentially a full matrix. Hence your runtime and memory problems ...
You will need to chunk out your calculations to avoid the memory issue and simply live with the calculation runtimes, or figure out another way to solve your problem. We don't have any insight into what that problem actually is, but maybe you could solve a smaller problem that approximates your actual problem within acceptable error, etc.

More Answers (1)

Steven Lord
Steven Lord on 15 Jun 2021
What release of MATLAB are you using?
What operating system? I'm suspect you're going to say Linux. If I'm right, the OOM Killer may have struck again. The page linked as the first footnote says how to check if this was the case.
  3 Comments
Paula
Paula on 15 Jun 2021
What I meant by "crash" was, the "Physical Memory Usage” in the Task Manager hit the maximum. Matlab keeps calculating, but I'm not able to stop it, there's no response if I click anywhere in the Matlab window. I have to click "End Task" in the Task Manager to force Matlab to shut down.
As for the computer, there's no blue screen, nor does it shut down. I think I should be able to use other applications on the computer, even when Matlab "crashed".
However, I did not find a corresponding crash log file, sorry. I entered the command "ls...", and I did find a list of crash files in the temporary directory, but all of them are of previous times when Matlab crashed.
I have a new "finding" to report please:
I just tried to run my script again:
...
A1 = B * C;
A = A1 + A2;
I put breakpoints so that these two lines of code were run individually. This time, the problem is different:
Matlab did not crash at the line "A1 = B * C" this time. But the “Physical Memory Usage” in the Task Manager approaches the maximum.
Then when I click "Continue" and let the line "A = A1 + A2" to be calculated, Matlab stopped and an error message showed: "Out of memory."
At the time when I posted my original question, I had the exact same code and the same matrices, and the problem was that Matlab "crashed", I had to end it from the Task Manager. I don't know why it's a different problem and a different error message when I ran it this time.
I will keep looking for crash log files.
Thank you!

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by