Out of memory issue

Hello My name is Kim from SKKU, South Korea.

I want to import my data which has dm4 format.

The dimension of data is 512*512*100*100. (21 GB at double 8 bytes)

My computer has 48GB RAM and i tried this at the computer which has 64GB RAM but computer still says 'out of memory' for my 21 GB file.

Could you tell me the way that I can import my data? I tried to crop my data, but It can be done when importing is done.

[Jan]

The posted information does not allow to estimate, if there are a lot of large arrays store in the RAM. The code shown in the error message shows, that the data are store as singles.

Use the debugger to find out, how large the junk is, which exhausts the memory:

dbstop if error

Then check the size of the typecasted vector.

Without seeing the code it is hard to guess further details. Why does GetTagType does not have any inputs? Is it a script? Do functions provide the data as global variables?

Are you using this function: https://www.mathworks.com/matlabcentral/fileexchange/43005-read-dm3-and-dm4-image-files ?

[Seon Je Kim]

Exactly. I am using ReadDMFile function.

However I modified it little bit because it cannot import 4D image data.

I am not sure this function works well but MATLAB says out of memory so I am trying to solve this error first.(if you know another function which can treat the 4dimensional DM format file well, please tell me.)

P.S. the GetTagType does not has input because it has variables with shared scope i guess.

[KSSV]

In what format your file is? You can load matrix by matrix, do the process to avoid memory issue. I don't think there is need to load the entire data.

[Seon Je Kim]

My file format is .dm4 the private format of digital micrograph program. I tried to crop the file but It is impossible with the program. It is possible after importing to MATLAB.

Answer 1

[Rupesh]

Hi Seon je Kim,

After going through the question, I understand that you want to handle your large nD dataset without overloading your computer's memory, to do so you can try two solutions; one is Tall Arrays and other is memory mapping.

Tall Arrays in MATLAB

Tall arrays in MATLAB are designed to handle large datasets that are too big to fit into memory. They allow you to work with data that can be as large as the available disk space on your system. When you use tall arrays, MATLAB performs operations in chunks, processing only a portion of the data at a time, which prevents memory overflow errors. To use tall arrays with your nD dataset in the dm4 format, you'll first need to convert the data into a format that MATLAB can efficiently process in chunks, such as a collection of CSV files or a MAT-file. Once the data is in a compatible format, you can create a tall array using the `tall` function. MATLAB will manage the memory and disk usage for you, allowing you to perform operations on the dataset as if it were entirely in memory. For example, you can calculate statistics, filter data, and even perform machine learning tasks without loading the entire dataset into RAM at once.

Memory Mapping in MATLAB

Memory mapping is another technique for handling large datasets that don't fit into RAM. It allows you to map a portion of a file directly into your address space and access it as if it were loaded in memory. This can be done in MATLAB using the “memmapfile” class, which creates a memory map to a file containing binary data. This is particularly useful for large dm4 files, as you can access and process parts of the data without reading the entire file into memory. With memory mapping, you can specify the format and dimensions of the data to interpret the binary file correctly. You can then access the data using indexing operations, which MATLAB translates into file I/O operations behind the scenes. This method is efficient because it only reads the parts of the file you need into memory, keeping the memory footprint low and preventing 'out of memory' errors while allowing you to work with data slices that fit within your available RAM.

You can refer to following documents for more information regarding Memory mapping Operations and can also find related MATLAB Answers for better understanding.

https://www.mathworks.com/matlabcentral/answers/2083268-find-data-from-files-that-are-too-large-to-read-in

https://www.mathworks.com/help/matlab/memory-mapping.html

Hope this helps!