Limitations of Automatic C/C++ Test Generation

Using Polyspace® Test™, you can automatically generate C/C++ unit tests with specific objectives instead of manually authoring tests. For instance, you can automatically generate tests that call functions with boundary values for inputs, or tests that cover true and false outcomes of all conditions in a function (full condition coverage).

This topic lists some of the limitations of automatic test generation. For more information on the steps for automatic test generation, see:

Test Generation Requires Test Authoring Support

If a function is not supported for graphical test authoring, automatically generating graphical tests to cover the function is not supported. However, you can write a script that calls the function and generate scripted tests from the script. For more information, see Generate C/C++ Tests for Scripts Calling Functions

In addition to functions not supported for graphical test authoring, other types of functions are not supported for automatic test generation. The following sections list situations where a function is supported for graphical test authoring but might not be fully supported for automatic test generation.

Certain Data Types Not Supported for Test Generation

If a function parameter or return value has one of the following types, it is not supported for automatic test generation:

  • Union type

  • An aggregate type (struct or class) with a pointer member or an array of pointers

  • C++ class type

  • C++ reference type

Functions with Certain Scopes Not Supported for Test Generation

The following types of functions are not supported for test generation:

  • Nonpublic class methods (C++)

  • Static functions

Functions with Certain Callees Not Supported for Test Generation

A function is not fully supported for automatic test generation if it contains a condition with a call to:

  • An external function (whose definition is not available at the time of test generation)

  • A function with a non-deterministic output.

For instance, in the following example, the second condition in func() cannot be covered by automatic test generation because the rand() function has non-deterministic output.

#include <stdlib.h>
void func(int x) {
   srand(time(NULL));
   if (rand() && x < 0) {
   
   }
   if (x > 15) {
   }
}
To enable full support for automatic test generation, you must create a mock for the function with nondeterministic output. For more information, see Override Callee Definitions When Testing Functions in Polyspace Platform User Interface.

Generated Tests Can Be Suboptimal

Automatically generated tests are designed to achieve the specified code coverage objectives, but in some cases, they are not the most optimal possible. For instance, consider the incrementGlobal() function:

#define MAX 10
int glob;

int incrementGlobal(void) {
    glob++;
    if (glob == MAX) {
        glob = 0;
        return 0;
    }
    return 1;
}
To test the true value of the condition in if(glob == MAX), you can write tests that use the least or the greatest number of steps:

  • A single-step test that initializes glob to MAX - 1 and invokes incrementGlobal().

  • A multi-step test with MAX steps that initializes glob to 0 in the first step and invokes incrementGlobal() in each step.

Automatic test generation can produce either of these tests (or something in between).

Test Generation Fails When Code Has Runtime Errors

If a function call causes a run-time error, test generation for the function can lead to incorrect results. For instance, all coverage objectives for test generation might not be satisfied.

To avoid this issue (or to diagnose unexpected test generation results), check for the absence of run-time errors in your project. To prove absence of certain types of run-time errors, you can run Polyspace Code Prover™ on your project. See Code Prover Analysis in Polyspace Platform User Interface (Polyspace Code Prover).

Test Generation Can Fail with Large Number of Inputs

If test generation requires values to be generated for a large number of input elements, the process can take very long and fail because of an inbuilt timeout. In most typical situations, the number of function parameters themselves would not be large enough to run into this issue. You would typically encounter this situation if the parameters somehow contain a large array requiring a large number of values to be generated. For instance:

C++ Tests Generated in Mixed Language Mode

If you generate tests using a project configuration that has the source language set to C-CPP, the tests are generated as C++ files and compiled using C++ compilation rules.

In particular, if your source code contains identifier names that are reserved keywords in C++, even if the source file is a C file, the generated test in C-CPP mode is in a C++ file and might fail compilation.

For more information on setting source code language in project configuration, see Source code language (-lang).

See Also

Topics