Basics of Automated Testing

This section describes the fundamentals of automated testing.

Unit, Functional, and Integration testing

Total Test supports developers to create, execute and automate unit, functional and integration tests. To make it very simple and understandable, from a technical point of view, the difference between unit and functional test is that a unit test virtualizes all external calls while a functional test runs live.

Unit test – executing a test case of a program in isolation where all external calls (Files, Db2, sub programs CICS/IMS APIs) are virtualized / stubbed out by the test case.
Functional test – executing a test case of a program in a live environment with live data.
Integration test – executing a test case that tests collaboration between two or more programs in a live environment with live data.

The test types are illustrated in the pyramid below. The pyramid illustrates that unit tests should be the major part of test cases. On top of unit tests we then have functional tests and integration tests, and in the end, we have the User Acceptance Testing (UAT), which should be a minor part compared to unit, functional and integration testing.

There are no officially agreed upon clear definitions of different test types and each vendor and customer seems to have their own definitions. As a start let us define what each test type means in relation to Total Test and elaborate a bit more on the differences compared to the technical difference above.

Type	Description
Unit Test	This tests the smallest piece of executable code in an application. For COBOL this is a program. It is created by a developer that understands the program code and logic and knows how to, for example, get into a specific IF statement to test this part of the code. The developer is in control and can stub out all external calls from the program to other systems such as IO, Db2, sub programs and CICS/IMS. The developer can decide what an external call from the program should return in a specific test case. A unit test can be described as a white box test where the developer has deep knowledge about the source code, and the test is executed in an isolated environment without requirements for access to external systems and data.
Functional Test	A functional test tests a function in an application. A function is typically implemented as a COBOL program. The functional test tests that the program implements its specification correctly and is typically created by a developer or tester/QA role. The tester understands what the program is supposed to do and sets up the functional test case by providing input to the program and defining expected output. The functional test is a black box test where the tester does not need to know about the internal source code of the program, and the program is executed in a live environment with live data and live systems.
Integration Test	An integration test tests the collaboration between two or more functions/programs and the data they use. This could, for example, be one program creating an entry in a table and another program reading the entry. The integration test will verify that the collaboration between the programs and their data works as expected. It can be made by a tester or developer. Like for functional testing, integration tests are executed in a live environment with live data and systems.

Shifting left and test-driven development

Shifting left means that testing should be done earlier in the development process, whereas test-driven development means that you have test cases for all software requirements.

DevOps pipelines—automated tools and processes that enable developers and operations professionals to collaborate to build and deploy code to a production environment.
Repositories—a place to store code, or software projects. A repository hosting service is an organizational management tool which offers a transparent view into the traditionally opaque workflow process of software development and production.
Test-driven development—a software development process that relies on software requirements which are converted into test cases before the software is completely developed, and tracking software developments by repeatedly testing the software against all test cases.

In DevOps we often talk about shifting left. This means that testing should be done earlier in the development process. Traditionally, testing on the mainframe is done in the later stages of a project, typically as part of end user acceptance testing. This is very late in the process for finding issues. Shifting left means that bugs should be found as early as possible. The earliest they can be found is at the same time that they are introduced into the source code by the developer. This is where unit testing comes in. A developer should write unit tests that test the changed code and in this way the bug is found as soon as it is introduced into the code. This is the fastest way to detect and correct bugs and has the largest benefit for increasing velocity, quality, and efficiency.

Functional test is typically done in a test environment with live data and is performed later in a sprint where a developer has moved the compiled load modules from a development environment to a test environment. This means that functional tests find bugs later, and hence the feedback to development is slower and more costly.

Integration testing typically comes after functional testing which means that it is more expensive to detect bugs compared to both unit and functional testing. The following illustration shows how unit, functional, and integration tests can be used by developers and testers in different environments during an agile sprint.

Types of testing

For unit and functional tests, there are variations in the purpose of a specific test case. All of the test cases should validate that a program behaves as required under testing, some tests validate that a change to the program yields the desired (new) outcome, while others ensure that the same change did not yield any undesired outcomes in parts of the program that were not supposed to change. Other tests make sure that the program behaves as required under rare conditions such as error handling or specific date/time handling.

A common problem when adopting unit and functional testing is that the developers and testers only create test cases for successful outcomes. The failed scenarios or the rare cases are often not tested, which leads to hidden errors or bugs. Therefore, negative testing as well as boundary condition testing are important to implement. Refer to the following table for information about the types of testing:

Type	Description
Positive Testing	The first test case you can consider creating, and also the test most start with, is the scenario where your program gets valid input and returns valid output—the normal successful execution. Many create few test cases of programs which have a positive results, where everything is as expected. This is a good start to create tests, but not sufficient to get good test coverage.
Boundary Testing	Unit and functional test cases should test values that have high and low boundaries. For unit tests you should know what the valid range of data is for every field and variable in your program. Verifying the maximum range of values is a great way to validate that calculations work at the edge of valid values. This includes testing values from input to the program as well as values from any external sources such as data sets, sub programs, Db2, and CIC/IMS API calls. For functional test you should know the significant input values where a program will behave differently and cover these in the test cases. Generally, this also tests the format of the expected data. If the format of the data is not correct, it should cause the test case to fail. You will often need several tests to cover all high and low boundary values and different variations. It is good practice to include out-of-bounds testing to verify that data validation and error-handling code is working correctly.
Negative Testing	Negative test cases are for the failure pathways in your program. For example, you simulate an out-of-memory condition that you know will cause your program to fail. You write these test cases to validate that your program error-handling code is working correctly. You need to write these types of test cases to achieve a high code coverage percentage. If only normal execution is tested, none of the error-handling code will be executed and validated and added into the code coverage percentage. Most well-written, robust programs have a fair amount of error-handling code in them and can account for a substantial percentage of the size of a program. Total Test Db2 stubs enables you to edit the SQLCODE and SQL Result and program stubs allow you to edit the data returned by a sub program, and the return code it exists with to simulate an error and test the error-handling code in you program. Negative testing also includes test cases where you send invalid data to your program to verify if it abends correctly or return a specific error code. Negative testing can be made with both unit and functional testing where the return code can be asserted. Functional testing can also verify that a program abends in a negative test.
Date/Time Testing	Many programs have special code that gets executed at specific times. Many financial applications have end-of-day, end-of-month, end-of-quarter, and end-of-year processing. To test those pieces of code that only get processed at those dates and times, you need a way to simulate the system date to be the specifically required date. Changing the system clock is not only difficult on the mainframe, but also not advisable. Unforeseen, negative consequences including deletion/archiving of data that suddenly has become outdated, expiring of software licenses, and so on. Even though there are shops that have set up their own special time travel LPARs, most shops use tools targeted at simulating dates/times for individual program runs or applications, without impacting the execution of other applications on the same system. These tools should also be used when you need to run unit tests for date- or time-dependent functionality. When recording a unit test with Total Test, data/time interactions are recorded and stubbed out as well. This means that the unit test will be executed and data asserted for the same date/time as when it was recorded. BMC AMI DevX Xchange tool allows a program to execute with a specific user-selected time. This approach only requires a change to the Total Test runner JCL to include Xchange and to set the desired date and time. It is a good idea to indicate in the JCL file name that you have a date and include a description in the JCL related to what date is set and why the specific date was selected.
Regression Testing	Regression tests ensure previously developed and tested software continue to work in the same way after code changes have been made or after it has interfaced with other software. Changes may include bug fixes, feature enhancements, configuration changes or running on new systems. New bugs that are uncovered are called software regressions. Regression testing simply means, run all relevant test cases again and ensure no bugs have been introduced into parts of the program that previously worked. One approach to regression testing is to run automated tests after each time code is built. You need to identify problems introduced into the software as quickly as possible and doing this in an automated fashion instead of depending on testers remembering to manually execute them is preferred. If code gets checked in and built, you should run your regression tests to check if new problems have been introduced with those changes. Some customers want to run tests after every code check-in or code promotion. BMC's source code management tool, Code Pipeline, integrates with Total Test via webhooks to kick off unit and functional tests after a code promotion.
Baseline Testing and Re-baseline test cases	With baseline we mean that you create test cases for a program that works. These tests are the baseline tests for the program. Baseline tests are run before making code changes to ensure the current system is working as expected. Otherwise, you may discover after making a change and testing new changes that the code fails, not because of the new changes but because of problems that existed before the changes. When you have finished code changes and updated test cases for a program, these test cases now make up the new baseline test suite of the program. A good collection of baseline tests should cover the different kinds of testing described above – both testing boundary conditions, negative testing and date/time testing where relevant. Baseline tests should be part of your regression test suite. When a program is changed, it might make baseline test cases fail, either because requirements to the program has changed or – for unit tests – that the internal logic has changed, and internal assertions fail. The process of changing the failed test case to match the updated program is called re-baselining. This involves changing expected values in the test case to new expected values, and when data structures have changed, for example, fields changed, added or removed, which means updating the test case and related stubs to conform to the new data structures.

Automated testing concepts in Total Test

Virtualized testing	Unit Test—tests the smallest piece of executable code in an application. For COBOL, this is a program. It is created by a developer that understands the program code and logic and knows how to, for example, get into a specific IF statement to test this part of the code. The developer is in control and can stub out all external calls from the program to other systems such as IO, Db2, sub programs, and CICS/IMS. The developer can decide what an external call from the program should return in a specific test case. A unit test can be described as a white box test, where the developer has deep knowledge about the source code, and the test is executed in an isolated environment without requirements for access to external systems and data.
Non-virtualized testing	Functional Test—tests a function in an application. A function is typically implemented as a COBOL program. A tester/QA or developer creates a functional test, which verifies that the program implements the specification correctly. The tester understands what the program is supposed to do and sets up the functional test case by providing input to the program and defining expected output. The functional test is a black box test where the tester does not need to know the internal source code of the program, and the program is executed in a live environment, with live data, and live systems. Integration Test—tests the collaboration between two or more functions/programs and the data they use. This could, for example, be one program creating an entry in a table and another program reading the entry. The integration test will verify that the collaboration between the programs and tests if their data works as expected. This can be created by a tester or developer. Integration tests are also executed in a live environment, with live data, and live systems.
Test cases	Focus on both, what to test and how to test. Test cases contain clearly defined steps for testing a specific feature in an application. They also contain high-level documentation describing an end-to-end functionality that needs to be tested. Each time an interface is called during the test run, it represents a test case. The simplest variant of a test scenario contains just one test case, which is referred to as a simple test scenario. In terms of creating a Total Test Virtualized Test, you need to provide the following information in order to create a test case for a test scenario: Which interface is being called? What is the name of the target that you want to test (program or load module)? Which input data needs to be passed? Which check conditions must be met if the target functions as expected?
Test scenarios	Test scenarios are used to define and describe what to test, which input to use, and which output to expect in order to consider the test successful. A test scenario is created to be a reproducible test of one or more specific requirements implemented by an interface. Check conditions are used and are essential to successful testing. These tests are required to properly document the test scenario, and are also crucial for the re-use of the test scenario. Check conditions are needed to: Automate the tests Re-use the scenario for regression testing.
Test suites	Test suites enable you to execute multiple test scenarios in one test run. This is useful for larger test runs in integration or regression tests. Test suites are intended to just loosely bundle test scenarios. The test scenarios contained in a test suite should not have any dependencies to each other. This condition is met if the test scenarios in a test suite can be run in any order and always lead to the same results. A test suite is a loose bundle of test scenarios. The test scenarios in a test suite are not related to each other and can be executed in any order without interfering with each other. The test scenarios are bundled in a test suite mostly for practical reasons. For example, you may have a collection of test scenarios that you want to run regularly as a regression test.
Total Test projects	Use Eclipse projects and contain all of the files needed for testing. It groups them into buildable and reusable units. An Eclipse project can have natures assigned to it which describe the purpose of this project. For example, the Java nature defines a project as Java project. Projects can have multiple natures combined to model different technical aspects.