saving a job.parallel object

I'm running in batch and I would like to save the variable job (CJSIndependentJob or CJSTask object).

is it possible?

[Steven Lord]

What do you hope/plan to do with the saved job object? Do you hope to use it to somehow pause the job and resume it later on, to be able to rerun the job, etc.?

[Sandra Martinez]

Hi Steven,

I would like to save it in a file, to eventually be able to download data from it such as: errors, Date Time, execution time, etc. I already saw that I can do this by using [pending,running,completed] = findTask(job); and retrieve the duration time from there using the FinishDateTime and StartDateTime methods and saving that to a txt. But it seems to me that the simplest thing would be to directly save the work and eventualy do a load if I want to access that data later. Maybe it is too heavy to save the entire job object, but in principle I would like to know how to do it.

Answer 1

[Edric Ellis]

It is not supported to save job / task objects themselves - the expected workflow is that you use parcluster to get back your job / task when you restart MATLAB. In other words:

% First MATLAB session - initiate batch job
clus = parcluster("myProfile");
job = batch(clus, @someFcn, numOutputs, {in1, in2});

% Later, restart MATLAB
clus = parcluster("myProfile")
clus.Jobs % Look at existing jobs
job = clus.Jobs(3) % or whatever

[Sam Marshalik]

To add to Edric's response, after running the job, you can fetchOutputs or load(job) to get all the outputs or variables from the job. Alternatively, you can use something like ValueStore to store all of your results in an itemized fashion.

[Sandra Martinez]

Thank Sam and Edric.

Now I understand better where those variables are stored (.....\MATLAB\local_cluster_jobs\R2021a\Job86) and the number ID is: job(1).ID).

Now I need to save some varibles (when the whole process finished) but in may case the variable job is an array:

for k=1:30
    job(k)=batch(@myFUN,1,{matfiles,k,Parameters,base_folder,saveflag,Plot_Figures},'CaptureDiary',true);
end

therefore I can't use the command after,

wait(job)

since this error message appears

Error using wait

Expected input number 1, job, to be a scalar.

to finally do,

[pending,running,completed] = findTask(job);

So the question is: How can I make a code that runs in batch in several workers, and when it finishes it saves some running data in a file. Or, there exists a function wait for an array job?

Matlab 2021a

[Edric Ellis]

The batch command always creates only a single job. One possibility is to write a parfor loop either inside myFun, or perhaps in a separate script, and then use the 'Pool' option to batch to run the script with a parallel pool available. Maybe something like this:

% Script: runMyFUN.m
parfor k = 1:30
    out{k} = myFUN(. . .);
end

% Launch script to run in parallel
job = batch("runMyFUN", Pool=6)

[Sandra Martinez]

Edric,

Why do you say that: "The batch command always creates only a single job"

In fact, if I type job it returns me the following:

job = 

1x4 Job array:

ID           Type        State              FinishDateTime  Username  Tasks
-----------------------------------------------------------------------------
1  5234    independent      running                                  User      1
2  5235    independent      running                                  User      1
3  5236    independent      running                                  User      1
4  5237    independent      running                                  User      1

And they all started at the same time

>>job(1).Tasks.StartDateTime
12-Feb-2025 11:08:30
>> job(2).Tasks.StartDateTime
12-Feb-2025 11:08:30
>> job(3).Tasks.StartDateTime
12-Feb-2025 11:08:30
>> job(4).Tasks.StartDateTime
12-Feb-2025 11:08:30

so, in fact works. But I don't know how to use the wait function in this case.

Maybe the neatest way is to do as you suggest.

[Edric Ellis]

What I meant was that each individual invocation of batch returns a scalar parallel.Job instance. If you want to wait for all jobs, you could just call wait in a loop, or do:

arrayfun(@wait, job)

[Sandra Martinez]

excellent! that was what I needed!