Efficient parallel unit test execution

I have the following issue with the unit testing framework: In my usecase, the test class setup functions take quite a bit of time to execute, the tests themselves are fast in comparison.

classdef TestExample < matlab.unittest.TestCase
    properties (ClassSetupParameter)
        dataToProcess = {'DataA','DataB','DataC','DataD','DataE'};
    end
    
    properties(Access=private)
        res1;
        res2;
        res3;
    end
    
    methods(TestClassSetup)
        function setup(obj,dataToProcess)
            pause(2);
            disp(dataToProcess);
            obj.res1=1;
            obj.res2=2;
            obj.res3=3;
        end
    end
    
    methods(Test)
        function test1(obj)
            obj.verifyEqual(obj.res1,1);
        end
        
        function test2(obj)
            obj.verifyEqual(obj.res2,2);
        end

        function test3(obj)
            obj.verifyEqual(obj.res3,3);
        end
    end
end

I'd like to run the tests for data A, B and C (and for a multitude of such TestClasses) in parallel such that

• Thread 1: DataA → test1();test2();test3();
• Thread 2: DataB → test1();test2();test3();
• Thread 3: DataC → test1();test2();test3();

However calling

clc;
tic;
ts=matlab.unittest.TestSuite.fromClass(?TestExample);
r=matlab.unittest.TestRunner.withTextOutput();
r.runInParallel(ts);
toc;

splits the tests in random groups running DataA, B and C on different threads, repeating the heavy workload in setup() to the point where I get a slowdown):

>> Split tests into 15 groups and running them on 6 workers.
>> Elapsed time is 6.477652 seconds.

Is there a convenient way to profit from parallelization in case the class setup methods are computationally expensive? Do fixtures or the test runner somehow support grouping the tests manually?

Side-note: I'd refrain from migrating the setup logic to blown-up test functions such as:

methods(Test)
    function testData(obj,dataToProcess)
        pause(2);
        disp(dataToProcess);
        obj.res1=1;
        obj.res2=2;
        obj.res3=3;
        obj.verifyEqual(obj.res1,1);
        obj.verifyEqual(obj.res2,2);
        obj.verifyEqual(obj.res3,3);
    end
end

to keep the console output meaningful, especially in case of an error.

Answer 1

[Sujay]

Hi DAX,

You can create Shared test fixture class that loads dataToProcess and runs the TestExample suite.

Using fixtures is more efficient in my experience. You can see more on creating shared test fixtures here: Write Tests Using Shared Fixtures

Alternatively, if you want to manually group and monitor how data is being consumed by tests running in parallel here is a quick workaround you can try:

Simplify and move dataToProcess part outside of Test Example

classdef TestExample < matlab.unittest.TestCase

properties(Access=private)

res1;

res2;

res3;

end

methods(TestClassSetup)

function setup(obj)

obj.res1=1;

obj.res2=2;

obj.res3=3;

end

end

methods(Test)

function test1(obj)

disp('test1');

obj.verifyEqual(obj.res1,1);

end

function test2(obj)

disp('test2');

obj.verifyEqual(obj.res2,2);

end

function test3(obj)

disp('test3');

obj.verifyEqual(obj.res3,3);

end

end

end

Have a ManuallyGroupedTests class that runs tests from TestExample

classdef ManuallyGroupedTests < matlab.unittest.TestCase

properties(ClassSetupParameter)

dataToProcess = {'DataA','DataB','DataC','DataD','DataE'};

end

methods(TestClassSetup)

function setup(~, dataToProcess)

disp(dataToProcess);

end

end

methods(Test)

function test(~)

r = matlab.unittest.TestRunner.withTextOutput;

ts = matlab.unittest.TestSuite.fromFile('TestExample.m');

r.run(ts);

end

end

end

To run tests in parallel

clc;

tic;

ts=matlab.unittest.TestSuite.fromClass(?ManuallyGroupedTests);

r=matlab.unittest.TestRunner.withTextOutput();

r.runInParallel(ts);

toc;

Regards,

Sujay