7 Essential Types of Software Tests Every Engineer Needs to Know | Software Testing, Part 2

Software testing has become an invaluable tool for anyone who writes or maintains code and software. However, there are many types of software tests. And it can be difficult to know where to start. In this video, we'll cover seven essential types of software tests every engineer needs to know.

We'll start with what is probably the most common type of software testing, called unit testing. Unit testing focuses on verifying the functionality of individual components or units of code in isolation. An example unit test would take an individual function or method, like this Fibonacci function, and provide its sample inputs. Then it compares the actual output with the expected output, given those inputs.

This particular unit test would also be a form of regression testing, which we'll talk about a little later. Unit tests help isolate and identify bugs at an early stage and can identify issues without the complexity of interactions with other parts of the software. It is relatively fast and easy to write and run unit tests. And unit testing can begin from the project's initial stages. The main drawback is that not all problems can be found by looking at single units in isolation.

So in addition to unit testing, the second type of software test we can leverage is integration testing. Integration testing verifies the interfaces between the code units to ensure they interact as expected. Integration testing is needed for detecting issues, such as data format inconsistencies, code interface incompatibilities, or incorrect assumptions about component interactions. An example would be ensuring different components of a rocket's control software work together correctly.

The Ariane 5 rocket disaster was fundamentally due to this type of integration error. One piece of the rocket's software was outputting a 64-bit data type to another piece that expected a 16-bit data type as input. When the data value grew larger than can be represented with a 16-bit data type, there was an integer overflow. This overflow ultimately resulted in the rocket exploding during its ascent.

The benefit of integration testing is that, it can find problems you'd never see if you only looked at units in isolation. However, integration tests are slower and more expensive to write and run and must necessarily be done later in the development process, as they require each integrated component to be ready and available for testing.

We can perform integration testing at various levels of granularity, from just two small components, all the way up to integrating the entire system, which is known as system testing, our third type of software test. System testing involves testing the entire integrated system to ensure it meets the specified requirements. It covers all the combined parts of the software to validate the full functionality. This level of testing simulates real-world scenarios to verify that the software behaves correctly under expected conditions.

An example would be testing a self-driving car's ability to navigate scenarios in a virtual world. These scenarios require all the various software components to be working together correctly. System testing has the benefit of directly testing the full behavior that users expect. And some problems will only be identified at this level. However, as it requires the full system to be available, it must be conducted later in the development process and is the most expensive type of software testing.

Unit testing, integration testing, and system testing make up the software testing pyramid. The pyramid represents the granularity and frequency of execution of the tests. Tests lower on the pyramid are more granular and are run more frequently. They are faster, easier, and cheaper to write and run. They can also be done earlier in the development process.

Tests higher on the pyramid are more time-consuming, expensive, and must be done later in development. The pyramid emphasizes having a larger number of low-level tests and fewer high-level tests. Generally, a project should aim to have approximately 70% unit tests, 20% integration tests, and 10% system tests by its completion.

The pyramid represents just one dimension to software testing. In addition to the granularity and frequency of testing, there is another orthogonal dimension to software testing, which is, what is the goal of that particular test? Functional tests focus on the correct answers and behaviors of the software. There are many types of functional tests.

The fourth and fifth type of software tests we'll cover are the two most common functional tests-- smoke testing and regression testing. Smoke tests are a fast and easy way to check the basic functionality of software. It verifies that the most critical features work, and that the software is ready for additional testing. The term smoke testing comes from electronics, where an easy way to test for major problems is to power on a device and see if it starts smoking.

In software development, a smoke test is the most basic method to execute the core function of your software. If it's a script, run the script. If it's a function, call it with basic inputs. If it's an app, launch the app. If that doesn't error, the smoke test passes.

Let's look at a quick example using MATLAB. We have an existing project whose main component is a script. As a first step, we can add a smoke test for this script. MATLAB can automatically generate a test template for us, which, for a script, will be a smoke test. This test will simply run the script, and if it doesn't error, the test will pass. That's all a smoke test is.

Smoke tests are fast and easy to write. So they are often the first tests to be developed and used. If the smoke test doesn't pass, there isn't much point in running the other tests until we fix it. So when running a collection of test, smoke tests are typically the first test run. And if they don't pass, then the test process is often aborted early to save time. However, smoke tests are only testing if major components are completely broken. They don't consider if the behavior has changed or is correct.

Regression testing, the fifth type of software testing, checks if changes to our code cause any unexpected changes in behavior. To put it another way, the answer shouldn't change unless we intended it to change. A regression test will run the current version of the software and compare its output to an answer that was either manually computed or computed by a previous version of the same software. If the answer doesn't match, the test will fail.

A common example is when we rewrite or refactor our code. Maybe because it needs to be faster or more efficient, or use a new version of some software dependency. A regression test will check that the new version produces the same answer as the old version. It's important to note that a regression test is testing for sameness, not correctness. It's up to the person writing the test to ensure that the sameness is, in fact, correct. While functional testing focuses on getting the right answer and behavior, non-functional testing focuses on other aspects of the software, such as its usability, security, and stability, among many others.

Today, we'll focus on the most common type of non-functional testing, performance testing, which is our sixth type of software test. As your project grows in size and complexity, the performance or execution speed of its various features may slow down. Performance testing evaluates your software's performance under various conditions. By running simulations with different settings and amounts of data and users, performance tests seek to measure criteria, like response times and resource usage, to identify bottlenecks and optimize performance.

No matter how you're testing your software components, in real-world applications, those components rarely live in isolation. They are almost always connected to and used with other components, such as data sources, user interfaces, or other internal components built by other teams. We could do an integration or system test, as discussed earlier, but those tests can be slow, expensive, and dependent on having the other component to be available and ready in our test environment.

Instead, we can use our final technique, which is mocking. Mocking is not a type of testing on its own. Instead, it's a mechanism to more easily test your component in isolation, despite your component requiring other dependencies. Mocking creates a simple simulation of the other components we need to interact with. An example of mocking could involve interacting with hardware, such as a data logger, when the real hardware isn't available.

Prior to the test, we would create a software object that receives commands and returns representative data in a similar way to a real data logger. Then, if the real data logger isn't available, we can still run tests that need to interact with it. Note that we don't need to build some high-fidelity simulation of the other component. We only need enough functionality to test our interactions with it.

Testing with mock components helps isolate and test specific parts of the system without relying on the actual components, thus making errors easier to isolate. It's important to note that testing with mocks is not a replacement for integration testing.

Lastly, it's important to remember the types of software testing we discussed today can be combined. You could have a unit test that uses mocking, or an integration test that's a regression test, or a system performance test. And there are more types of tests beyond what we've dived into today.

But if you're ready to learn more about software testing, you can view all the videos in this series here. And to see an example of implementing unit testing in MATLAB, see this video. Thanks for watching, and we'll see you in the next one.