Out of memory for Blkdiag

I have two big matrices ( 5000X329176 and 5000X328658 ) and I want to combine them into one matrix which is in a form of these two in diagonal and all the other elements 0. basically I want blkdiag(A,B).

The problem is even though I made both these guys in int8 variable type and sparse and they take a small size but for blkdiag function, it needs to make a bigger matrix first ,zeros (5000+5000 X 329176+328658) and this guy is by default made in double which takes a huge amount of Ram.

I know e.g. zeros(100000,650000) needs 484 GB memory and sparse(zeros(100000,650000)) because of execution order needs the same amount ! however zeros(100000,650000, 'int8') needs 60GB memory which is available.

Unfortunately, blkdiag does not have this option. So, any idea about solving the problem?

Answer 1

[Matt J]

If the matrices are sparse, why store them as int8? Why not store them as sparse doubles? If you pre-convert A and B to type sparse, then blkdiag will assemble them without creating non-sparse type matrices, e.g.,

>> A=sprand( 5000,329176,.0001 );
>> B=sprand( 5000,328658,.0001 );
>> C=blkdiag(A,B);

>> Whos C

Name   Size             Kilobytes     Class    Attributes

C      10000x657834         10279     double   sparse

and sparse(zeros(100000,650000)) because of execution order needs the same amount !

You would never create a sparse matrix this way. You always use one of the following syntaxes, which avoid allocating a full sized matrix,

   S=sparse(m,n);
   S=sparse(I,J,S);

[Matt J]

Vahid Hematian Dehkordi commented:

Thank you so much. I thought sparse logical or sparse int8 should have the minimum size but it seems sparse double is the right way.