Main Content

testrunner

Create test runner

Since R2021a

collapse all in page

Syntax

runner = testrunner
runner = testrunner('minimal')
runner = testrunner('textoutput')

Description

example

runner = testrunner creates a default test runner, which is similar to the runner that the testing framework configures by default when you call the runtests function.

The testrunner function returns a matlab.unittest.TestRunner object. You can call the methods on the returned object to run and operate on your test suite and to customize running tests. For example, to run a suite of tests, use run(runner,suite).

example

runner = testrunner('minimal') creates a minimal runner with no plugins installed. The returned test runner is the simplest runner possible and produces no text output. Use this syntax when you want to have complete control over which plugins to add to the runner.

example

runner = testrunner('textoutput') creates a runner that is configured for text output. The output produced includes test progress as well as diagnostics in the event of test failures.

This syntax creates runners that tend to run tests more quickly, because the testing framework does not record diagnostics on test results produced by a nondefault runner. For more information, see Programmatically Access Test Diagnostics.

Examples

collapse all

Open Script

Run a suite of tests with a default runner and access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,"Hello World!","llo")
end

Create a test suite from the tests in sampleTest.m. Then, create a default runner and run the tests.

suite = testsuite('SampleTest');
runner = testrunner;
results = run(runner,suite);

Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\work\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)

ans = 

  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.3962
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed, 0 Incomplete.
   0.39619 seconds testing time.

When you run tests with a default runner, the testing framework uses a DiagnosticsRecordingPlugin instance to record diagnostics on test results. Access the recorded diagnostics for the second test using the DiagnosticRecord field in the Details property on the TestResult object.

records = results(2).Details.DiagnosticRecord

records = 

  QualificationDiagnosticRecord with properties:

                          Event: 'VerificationFailed'
                     EventScope: TestMethod
                  EventLocation: 'sampleTest/testB'
          TestDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
     FrameworkDiagnosticResults: [1×1 matlab.automation.diagnostics.DiagnosticResult]
    AdditionalDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
                          Stack: [1×1 struct]
                         Report: 'Verification failed in sampleTest/testB.↵    ---------------------↵    Framework Diagnostic:↵    ---------------------↵    verifyGreaterThan failed.↵    --> The value must be greater than the minimum value.↵    ↵    Actual Value:↵        13↵    Minimum Value (Exclusive):↵        42↵    ------------------↵    Stack Information:↵    ------------------↵    In C:\work\sampleTest.m (testB) at 10'
Open Live Script

Generate JUnit-style test results by creating a minimal runner and then adding an XMLPlugin instance to the runner.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a test runner with no plugins. This code creates a silent runner that produces no output. You can now install whatever plugins you like.

runner = testrunner('minimal');

Create an XMLPlugin instance that writes JUnit-style XML output to the file myTestResults.xml.

import matlab.unittest.plugins.XMLPlugin
xmlFile = 'myTestResults.xml';
p = XMLPlugin.producingJUnitFormat(xmlFile);

Add the plugin to the test runner and run the tests.

addPlugin(runner,p)
results = run(runner,suite);

Display the results from the second test.

results(2)
ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.0723
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   0.0723 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field. 

records = results(2).Details
records = struct with no fields.

Now, view the contents of the generated artifact.

disp(fileread(xmlFile))
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<testsuites>

  <testsuite errors="0" failures="1" name="sampleTest" skipped="0" tests="3" time="0.083731">
    <testcase classname="sampleTest" name="testA" time="0.0085045"/>
    <testcase classname="sampleTest" name="testB" time="0.0723">
      <failure type="VerificationFailure">Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --&gt; The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex97531283\sampleTest.m (testB) at 10</failure>
    </testcase>
    <testcase classname="sampleTest" name="testC" time="0.0029273"/>
  </testsuite>

</testsuites>
Open Live Script

Run a suite of tests with a runner that is configured for text output, and then access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a runner that produces text output and use it to run the tests.

runner = testrunner('textoutput');
results = run(runner,suite);
Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex48684143\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)
ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 1.9894
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   1.9894 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field. 

records = results(2).Details
records = struct with no fields.

Tips

Version History

Introduced in R2021a

See Also

| | | | |

Topics