save command in matlab parfor loop not going through all iterations

I'm trying to save a matrix within a parfor loop. I know matlab doesn't allow this and one needs to create a separate program. A bit of googling suggest creating a separate function, say parsave as follows:

      function parsave(fname, x,y)

        save(fname, 'x', 'y')

       end

The problem I'm encountering is that the program seems to be halting without any error before going through all of the iterations (i.e., it is saving about half of the files). I've got quite a lot of small files (around 1M). Could this be the problem?

Thanks in advance

Answer 1

[Jason Ross]

You don't provide a lot of detail as to where you are saving the files, but in my experience, when you get above a few thousand files in a directory (dependent on operating system and filesystem type), things start to slow down tremendously. If you are dumping one million files in a single directory, I'd say that you need to re-think that design as the time to do anything with that directory (even something as trivial as a file listing) is going to take forever, even if your OS/filesystem "theoretically" allows it. Your program will be bottlenecked on file I/O, and will be a pain to maintain and troubleshoot over time.

Look into making bigger files or developing a directory structure. Since you are also doing this in parallel and you don't know where the files are going to end up, you need to be very, very careful about duplicate names as well.

Keep in mind that your system is also limited in the number of file handles it can have open, as well, and you could be hitting a limit there. As well as the ability of your hardware to process the file creation requests as the number of files grows.