system command in parfor loop

I am invoking a C++ executable from the Matlab command prompt.

Interactively, it goes like this:

>> [status,cmdout] = system('multi.exe','-echo')

Input unique file identifier. Ex: 2~04-13-013939 - 2~11-30-091444

And this works fine.

However, I need to invoke the C++ executable multiple times. So I created a parfor loop, like so:

parpool
tic
parfor i=1:nfiles
    [status,cmdout] = system('multi.exe','-echo')
    str3{i,1}
end

where str3 is a series of inputs:

str3 =

14×1 cell array

{'1~11-24-054646'}

{'1~11-24-095017'}

{'1~11-29-060908'}

{'1~11-29-101743'}

{'1~11-30-091225'}

{'2~11-24-055554'}

{'2~11-24-100903'}

{'2~11-29-060541'}

{'2~11-29-101814'}

{'2~11-30-091444'}

{'3~11-24-055933'}

{'3~11-29-101847'}

{'4~11-24-060505'}

{'4~11-29-101935'}

This ran for over 54 hours, but the task never completed, and I eventually hit Ctrl-C.

Any thoughts/ feedback?

[Mario Malic]

Check this out. Parfor worked fine for me using Sysem.Diagnostics.Process.

https://www.mathworks.com/matlabcentral/answers/583955-get-the-status-using-system-command-when-program-has-been-closed#answer_485480

In order to relay the messages from workers in parpool, there are functions which I don't remember the name exactly. Check out this example https://www.mathworks.com/help/parallel-computing/parallel.pool.dataqueue.html

[jessupj]

just to play devil's advocate, i've approached a similar problem in the past by calling parallel instances of matlab from the shell. it was by no means the most I/O-efficient method and wasted a lot of time opening and closing matlab & pooling. the individual system jobs were geophysical models that took O(days) so the data importing and pool opening overhead was comparatively insignificant.

Answer 1

[Walter Roberson]

parpool
tic
parfor i=1:nfiles
    cmd = sprintf('echo "%s" | multi.exe', str3{i,1});
    [status,cmdout] = system(cmd, '-echo')
end

[Sujatha]

Sorry that didn't work. Please see attachment.

[Sujatha]

This may be pertinent (please see enclosure). Thanks!

[Sujatha]

I think I know what the issue is: In interactive mode, I type:

>> [status,cmdout] = system('multi.exe','-echo')

Matlab responds with:

Input unique file identifier. Ex: 2~04-13-013939 -

I then type:

  2~11-30-091444 <Return> (Keyboard Return)

In the parfor loop, there is no <Return> (Keyboard Return) after the str3{i,1} input to the parallel processes!

[Walter Roberson]

Perhaps you need

parpool
tic
parfor i=1:nfiles
    cmd = sprintf('(echo "%s" & echo.) | multi.exe', str3{i,1});
    [status,cmdout] = system(cmd, '-echo')
end

The period after the echo is important, and must be immediately after the echo with no space.

[Sujatha]

No joy!

>> parfor i=1:4

cmd = sprintf('(echo "%s" & echo.) | multi.exe', str3{i,1});

[status,cmdout] = system(cmd, '-echo')

end

'multi.exe' is not recognized as an internal or external command,

operable program or batch file.

'multi.exe' is not recognized as an internal or external command,

operable program or batch file.

'multi.exe' is not recognized as an internal or external command,

operable program or batch file.

'multi.exe' is not recognized as an internal or external command,

operable program or batch file.

[Sujatha]

What does:

(echo "4~11-24-060505" & echo.)

do?

[Walter Roberson]

At the windows COMMAND level,

(echo "4~11-24-060505" & echo.) |

should create an output stream that contains

4~11-24-060505<NEWLINE><NEWLINE>

and make it available as input to the next program in sequence (which here would be multi.exe)

Could you confirm for me that if you hard-code

system('multi.exe', '-echo')

that the program multi runs, but that if you hard-code

system('(echo "4~11-24-060505" & echo.) | multi.exe', '-echo')

that it fails saying it does not know what multi.exe is ?

[Sujatha]

Yes, that is true.

The program runs with:

system('multi.exe', '-echo')

The program fails with:

system('(echo "4~11-24-060505" & echo.) | multi.exe', '-echo')

and does not recognize "multi.exe".

[Walter Roberson]

Well, there is another possibility.

parpool
tic
parfor i=1:nfiles
    tname{i} = tempfile();
    [fid, msg] = fopen(tname{i});
    if fid < 0
      error('iteration %d failed to create temporary file "%s" because "%s"', i, tname{i}, msg);
    end
    fprintf(fid, '%s\n\n', str3{i,1});
    fclose(fid)
    cmd = sprintf('multi.exe < "%s"', tname{i});
    [status,cmdout] = system(cmd, '-echo')
end

Then once you are sure that all of the executables have finished, you can delete all of the files named in tname .

[Sujatha]

An UndefinedFunction error was thrown on the workers for 'tempfile'. This might be because the

file containing 'tempfile' is not accessible on the workers. Use addAttachedFiles(pool, files)

to specify the required files to be attached. For more information, see the documentation for

'parallel.Pool/addAttachedFiles'.

Caused by:

Unrecognized function or variable 'tempfile'.

[Sujatha]

addAttachedFiles(ans, tempfile), didn't help.

[Walter Roberson]

parpool
tic
parfor i=1:nfiles
    tname{i} = tempname();
    [fid, msg] = fopen(tname{i});
    if fid < 0
      error('iteration %d failed to create temporary file "%s" because "%s"', i, tname{i}, msg);
    end
    fprintf(fid, '%s\n\n', str3{i,1});
    fclose(fid)
    cmd = sprintf('multi.exe < "%s"', tname{i});
    [status,cmdout] = system(cmd, '-echo')
end