Machine-Readable Results for Tests Written Using Polyspace Test API

You can write C/C++ unit tests using the Polyspace^® Test™ xUnit API. When you run these tests along with other commands, you can print the test results in a machine-readable format, and extract the test output easily from the output of the other commands.

Prerequisites

This topic describes test authoring using the Polyspace Test xUnit API. To compile these tests, you are required to know some file paths in advance. For your convenience, you can define environment variables to stand for the file paths, or otherwise include the file paths in your build. For more information, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

How to Print in Machine-Readable Format

By default, test results are printed on the console in a human-readable tabular format. To see test results in a machine-readable format, use the option -format mrf or -format m with the test runner executable after building your sources and tests. Alternatively, you can define the following configuration macro:

#define PST_OUTPUT PST_OUTPUT_MRF

For more information on configuration macros, see Configuration Macros in Polyspace Test API for C/C++ Code.

For more details on test results formats, see Results Format of Tests Written Using Polyspace Test API.

Result Format Example

The following sections show examples of test results in a machine-readable format.

Example Files

To see the test results format described in this topic:

Copy the source files in the folder polyspaceroot\polyspace\examples\doc_pstest\test_results_format to a writable location. Here, polyspaceroot is the Polyspace installation folder, for example, C:\Program Files\Polyspace\R2026a.
Build and run the tests as usual.
For example, the file test_all_pass.c contains tests for functions defined in the file example.c. To build and run these tests with a GCC compiler in Windows^®, enter the following at the command line:
```
gcc example.c test_all_pass.c <PSTUNIT_SOURCE> -I <PSTUNIT_INCLUDE> -o testrunner
```
To run the tests, enter:
```
testrunner.exe -format mrf
```
For more information on the file paths <PSTUNIT_SOURCE> and <PSTUNIT_INCLUDE>, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

The folder contains two test files:

test_all_pass.c contains two test suites, suiteGlobalSumTestsAllPass and suiteStatusTestsAllPass. Each suite contains two tests, both passing.
test_some_fail.c contains two test suites, suiteGlobalSumTestsSomeFail and suiteStatusTestsSomeFail. Each suite contains a failing test.

All Tests Pass

When all tests pass, you see the following results:

##[begin]
##[begin_run]
##[begin_suite_config;suiteGlobalSumTestsAllPass]
##[end_suite_config]
##[begin_suite;suiteGlobalSumTestsAllPass]
##[begin_test;globalSumTestOne]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[begin_test;globalSumTestTwo]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[end_suite;pass]
##[begin_suite_config;suiteStatusTestsAllPass]
##[end_suite_config]
##[begin_suite;suiteStatusTestsAllPass]
##[begin_test;statusTestOne]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[begin_test;statusTestTwo]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[end_suite;pass]
##[end_run;pass;requested_suites:2;passed_suites:2;failed_suites:0;incomplete_suites:0;requested_tests:4;passed_tests:4;failed_tests:0;incomplete_tests:0]
##[end]

Some Tests Fail

When one of the tests fails in both test suites (globalSumTestTwo in suite suiteGlobalSumTestsSomeFail and statusTestTwo in suite suiteStatusTestsSomeFail), you see the following results:

##[begin]
##[begin_run]
##[begin_suite_config;suiteGlobalSumTestsSomeFail]
##[end_suite_config]
##[begin_suite;suiteGlobalSumTestsSomeFail]
##[begin_test;globalSumTestOne]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[begin_test;globalSumTestTwo]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[assessment;verify;0;expr:(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum);file:test_some_fail.c;line:42;message:Saturating sum did not return expected value.;diag:lhs="4294967295" rhs="2147483648"]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;fail,verify_fail]
##[end_suite;fail]
##[begin_suite_config;suiteStatusTestsSomeFail]
##[end_suite_config]
##[begin_suite;suiteStatusTestsSomeFail]
##[begin_test;statusTestOne]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;pass]
##[begin_test;statusTestTwo]
##[begin_suite_test_setup]
##[end_suite_test_setup]
##[assessment;verify;0;expr:(pst_long_long_t)(status2) == (pst_long_long_t)(expected_status);file:test_some_fail.c;line:72;message:Saturating sum did not return expected value.;diag:lhs="1" rhs="0"]
##[begin_suite_test_teardown]
##[end_suite_test_teardown]
##[end_test;fail,verify_fail]
##[end_suite;fail]
##[end_run;fail;requested_suites:2;passed_suites:0;failed_suites:2;incomplete_suites:0;requested_tests:4;passed_tests:2;failed_tests:2;incomplete_tests:0]
##[end]

Result Format Reference

The results in the machine-readable format consist of several parts, each beginning with a ##[ and ending with a ]. By default, the results are printed in a verbose format with one part of the result per line. If you use the option -format noverbose:

The entire result is printed in one block without new lines.
Each part of the message uses acronyms, for example ##[b]...##[B] instead of:
```
##[begin]
...
##[end]
```

Parts of a Result Message

The following table shows the various parts of a machine-readable test result (both the verbose and nonverbose formats).

Message Description

Message	Description
##[begin] ... ##[end] OR `##[b]...##[B]`	Beginning and end of test execution. When you run tests written using the Polyspace Test API along with other commands, you can look for a `##[begin]` to identify the beginning of test results.
##[begin_run] ... ##[end_run;run_results] OR `##[r]...##[R;run_results]`	Beginning and end of test runs. In the `end_run` message, you see details of how many suites/tests passed or failed. For more details, see Test Execution Summary.
##[begin_suite_config;suiteName] ##[end_suite_config] OR `##[sc;suiteName]##[SC]`	Beginning and end of suite configuration. You configure a suite using the `PST_SUITE_CONFIG` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.
##[begin_suite;suiteName] ... ##[end_suite;status] OR `##[s;suiteName]...##[S;status]`	Beginning and end of suite execution. A suite consists of one or more tests with an optional common configuration. In the `end_suite` message, you see whether the suite passed, failed or is incomplete. The `status` states: `pass` if all tests in the suite pass. `fail` or `incomplete` if at least one test fails or does not complete execution. For more information, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. The suite execution includes suite setup and teardown functions. For more details, see Additions to Result Message When Setup and Teardown Functions Are Executed.
##[begin_test_config;suiteName;testName] ##[end_test_config] OR `##[tc;suiteName;testName]##[TC]`	Beginning and end of test configuration. You configure a suite using the `PST_TEST_CONFIG` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.
##[begin_test;testName] ... ##[end_test;status] OR `##[t;testName]...##[T;status]`	Beginning and end of test execution. In the `end_test` message, you see whether the test passed, failed or is incomplete. If the test fails, in the verbose format, you also see additional test status information: `assert_fail`: An `ASSERT` statement failed. `verify_fail`: A `VERIFY` statement failed. For more information on the different between `ASSERT` and `VERIFY` statements, see Assessment Macros in Polyspace Test API for C/C++ Code. `assume_fail`: An `ASSUME` statement failed. `expect_fail`: An expected failure occurred. For more information on expected failures, see Write C/C++ Tests with Known Errors or Failures to Support Test-Driven Development. The test execution includes test setup and teardown functions. For more details, see Additions to Result Message When Setup and Teardown Functions Are Executed.

##[begin]
...
##[end]

##[b]...##[B]

Beginning and end of test execution.

When you run tests written using the Polyspace Test API along with other commands, you can look for a ##[begin] to identify the beginning of test results.

##[begin_run]
...
##[end_run;run_results]

##[r]...##[R;run_results]

Beginning and end of test runs.

In the end_run message, you see details of how many suites/tests passed or failed. For more details, see Test Execution Summary.

##[begin_suite_config;suiteName]
##[end_suite_config]

##[sc;suiteName]##[SC]

Beginning and end of suite configuration. You configure a suite using the PST_SUITE_CONFIG macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

##[begin_suite;suiteName]
...
##[end_suite;status]

##[s;suiteName]...##[S;status]

Beginning and end of suite execution. A suite consists of one or more tests with an optional common configuration.

In the end_suite message, you see whether the suite passed, failed or is incomplete. The status states:

pass if all tests in the suite pass.
fail or incomplete if at least one test fails or does not complete execution.

For more information, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

The suite execution includes suite setup and teardown functions. For more details, see Additions to Result Message When Setup and Teardown Functions Are Executed.

##[begin_test_config;suiteName;testName]
##[end_test_config]

##[tc;suiteName;testName]##[TC]

Beginning and end of test configuration. You configure a suite using the PST_TEST_CONFIG macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

##[begin_test;testName]
...
##[end_test;status]

##[t;testName]...##[T;status]

Beginning and end of test execution.

In the end_test message, you see whether the test passed, failed or is incomplete. If the test fails, in the verbose format, you also see additional test status information:

assert_fail: An ASSERT statement failed.
verify_fail: A VERIFY statement failed.
For more information on the different between ASSERT and VERIFY statements, see Assessment Macros in Polyspace Test API for C/C++ Code.
assume_fail: An ASSUME statement failed.
expect_fail: An expected failure occurred.
For more information on expected failures, see Write C/C++ Tests with Known Errors or Failures to Support Test-Driven Development.

The test execution includes test setup and teardown functions. For more details, see Additions to Result Message When Setup and Teardown Functions Are Executed.

Test Execution Summary

At the end of a run, you see the following message:

##[end_run;status;requested_suites:n_suites_requested;passed_suites:n_suites_passed;failed_suites:n_suites_failed;incomplete_suites:n_suites_incomplete;requested_tests:n_tests_requested;passed_tests:n_tests_passed;failed_tests:n_tests_failed;incomplete_tests:n_tests_incomplete]

Here:

status is pass if all tests passed and fail or incomplete if at least one test failed or did not complete execution.
n_suites_requested is the number of suites requested for execution.
n_suites_passed is the number of suites that passed.
n_suites_failed is the number of suites that failed.
n_suites_incomplete is the number of suites that did not complete execution.
n_tests_requested is the number of tests requested for execution.
n_tests_passed is the number of tests that passed.
n_tests_failed is the number of tests that failed.
n_tests_incomplete is the number of tests that did not complete execution.

For example, the message:

##[end_run;fail;requested_suites:2;passed_suites:0;failed_suites:2;incomplete_suites:0;requested_tests:4;passed_tests:2;failed_tests:2;incomplete_tests:0]

indicates that:

The overall status of the run is fail (at least one test failed execution).
Of the 2 suites requested, both failed execution.
Of the 4 tests requested, 2 passed, 2 failed and none of the tests were incomplete.

If you print the results in a nonverbose format, the same message looks like this:

##[R;status;n_suites_requested;n_suites_passed;n_suites_failed;n_suites_incomplete;n_tests_requested;n_tests_passed;n_tests_failed;n_tests_incomplete]

For example, the message in the preceding example reduces to this:

##[R;f;2;0;2;0;4;2;2;0]

Additions to Result Message When Setup and Teardown Functions Are Executed

In your test and suite configurations, you can specify setup functions that execute before tests and teardown functions that execute afterward. The setup and teardown functions are of two types:

Setup functions that run before all tests in a suite, and teardown functions that run after all tests in a suite.
Setup functions that run before a specific test (or each test in a suite), and teardown functions that run after a specific test (or each test in a suite).

For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

If you specify setup and teardown functions in your configurations, the result message contains additional parts related to the execution of these functions. The following table describes the parts of a result message related to setup and teardown functions (both the verbose and nonverbose formats).

Message Description

Message	Description
##[begin_suite_setup] ... ##[end_suite_setup;status] OR `##[ss]...##[SS;status]`	Beginning and end of execution of suite setup functions. These functions run before all tests in a suite and are specified using the `PST_SUITE_SETUP` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. The `status` is `fail` if the suite-setup function throws an exception or contains a failed assessment. Otherwise, the `status` is `pass`.
##[begin_suite_teardown] ... ##[end_suite_teardown;status] OR `##[st]...##[ST;status]`	Beginning and end of execution of suite teardown functions. These functions run after all tests in a suite and are specified using the `PST_SUITE_TEARDOWN` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. The `status` is `fail` if the suite-teardown function throws an exception or contains a failed assessment. Otherwise, the `status` is `pass`.
##[begin_suite_test_setup;] ... ##[end_suite_test_setup] OR `##[cs]...##[CS]`	Beginning and end of execution of suite-specific test setup functions. These functions run before each test in a suite and are specified using the `PST_SUITE_TEST_SETUP` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. There is no status message associated with test setup functions. If a test setup function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the `end_test` message.
##[begin_suite_test_teardown;] ... ##[end_suite_test_teardown] OR `##[ct]...##[CT]`	Beginning and end of execution of suite-specific test teardown functions. These functions run after each test in a suite and are specified using the `PST_SUITE_TEST_TEARDOWN` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. There is no status message associated with test teardown functions. If a test teardown function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the `end_test` message.
##[begin_test_setup] ... ##[end_test_setup] OR `##[ts]...##[TS]`	Beginning and end of execution of test setup functions. These functions run before a specific test in a suite and are specified using the `PST_SETUP` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. There is no status message associated with test setup functions. If a test setup function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the `end_test` message.
##[begin_test_teardown] ... ##[end_test_teardown] OR `##[tt]...##[TT]`	Beginning and end of execution of test teardown functions. These functions run after a specific test in a suite and are specified using the `PST_TEARDOWN` macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code. There is no status message associated with test teardown functions. If a test teardown function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the `end_test` message.

##[begin_suite_setup]
...
##[end_suite_setup;status]

##[ss]...##[SS;status]

Beginning and end of execution of suite setup functions. These functions run before all tests in a suite and are specified using the PST_SUITE_SETUP macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

The status is fail if the suite-setup function throws an exception or contains a failed assessment. Otherwise, the status is pass.

##[begin_suite_teardown]
...
##[end_suite_teardown;status]

##[st]...##[ST;status]

Beginning and end of execution of suite teardown functions. These functions run after all tests in a suite and are specified using the PST_SUITE_TEARDOWN macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

The status is fail if the suite-teardown function throws an exception or contains a failed assessment. Otherwise, the status is pass.

##[begin_suite_test_setup;]
...
##[end_suite_test_setup]

##[cs]...##[CS]

Beginning and end of execution of suite-specific test setup functions. These functions run before each test in a suite and are specified using the PST_SUITE_TEST_SETUP macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

There is no status message associated with test setup functions. If a test setup function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the end_test message.

##[begin_suite_test_teardown;]
...
##[end_suite_test_teardown]

##[ct]...##[CT]

Beginning and end of execution of suite-specific test teardown functions. These functions run after each test in a suite and are specified using the PST_SUITE_TEST_TEARDOWN macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

There is no status message associated with test teardown functions. If a test teardown function throws an exception or contains a failed assessment, the overall test execution fails. The failed status is reflected in the overall status shown in the end_test message.

##[begin_test_setup]
...
##[end_test_setup]

##[ts]...##[TS]

Beginning and end of execution of test setup functions. These functions run before a specific test in a suite and are specified using the PST_SETUP macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.

##[begin_test_teardown]
...
##[end_test_teardown]

##[tt]...##[TT]

Beginning and end of execution of test teardown functions. These functions run after a specific test in a suite and are specified using the PST_TEARDOWN macro. For more details, see Group C/C++ Tests into Suites with Common Setup and Teardown Code.