Functional / Integration Testing Best Practices
A user can create and execute functional tests from the web and the Total Test client. The web client is mainly meant for non-developers testing called programs (having a linkage section) by providing input and expected output values. For more advanced test cases, the Total Test client is recommended. We recommend you use Total Test to share test cases with a SCM such as Git. The built-in version control in the Total Test web repository is very limited and does not allow collaboration on the same test case. This section explains the best practice of creating functional and integration tests for specific use cases or program types.
Testing called programs
To test a called program in Batch and CICS, you should define a component in the repository and create test cases by providing input and expected output values. When you are working in Total Test, consider using the Microsoft Excel integration that makes management of test data much easier when you want to test your program with multiple sets of input/output data. We recommend performing tests with test boundary values in addition to negative testing. With functional test you can assert on the return code and also that the program abends. If you want to test the same program in different load libraries, you must not create multiple component definitions. Instead, you can have just one test scenario (xaunit file) and multiple execution context files (xactx). When you execute the test scenario, you have the possibility to override the runtime values of the component definition, and you can for example, define a specific list of load libraries to use. The runtime parameters are saved to the xactx file and will be used for subsequent executions.
If the called program uses input/output files, and they can be different from test to test, then you should define the DD statements in the functional test case, in the test scenario instead of using the default DD statements setup in the component definition.
Testing main batch programs
If the program you want to test is a Batch program without a linkage section, there is no need to use a component definition. You can use a Job element in a test scenario to submit a complete JCL specification. The Job element gives some more freedom to have multiple steps to create temporary data sets used by the program. After executing the main batch program, you should verify if the data produced by the program is as expected. You can use the SQLSelect element to select data in Db2 and verify with the Verify element, and you can use the elements VerifyRemoteFile or FileAIDCompare to compare actual and expected data sets in the mainframe. You can also extract a spool file from the job witl SpoolFile and verify the content with the Verify element.
It is best practice to create self-contained test cases, and before running the job, you can use the SQLUpdate element to update Db2, and you can use the WriteRemoteFile to create input and expected output data sets.
Keeping test scenarios reasonable in size
A test scenario has the flexibility to have hundreds or thousands of test cases defined. However, if you make a test scenario too large, it can take a long time to complete and you will get delayed feedback. You might also run into memory issues, where data cannot be contained in memory. We recommend to design test scenarios of a reasonable size, with not more than 100 test cases in one test scenario. However, there might be scenarios requiring more test cases, such as the case for called programs with small linkage sections.
Prefer test cases in a SCM over the web repository
The web client enables users to create test cases of called programs that are saved in the Total Test repository and that are executable from the web client and from a REST API. If you use this functionality to create and save test cases, we recommend to import the test cases into Total Test and share them in a real SCM. You can maintain the test cases in Total Test and upload them back to the repository if you want to execute them here. You can import test scenarios from the repository directly in Total Test from the Repository view.
Prefer automation with the CLI and SCM over REST APIs
You can use test cases in the repository for test automation by executing these through a REST API. This is typically done from, for example, Jenkins. However, we recommend to setup a CI/CD pipeline with integration to an SCM and use the Total Test CLI instead. This is a more mature solution that will also allow you to get additional benefits such as access to Junit and Sonar Qube reports directly in the pipeline.
Do not use the scheduling mechanism available in the web client
The original development of functional testing focused on a complete product for DevOps test automation without the need for additional products. This means that it is possible to define test cases in the web based repository and execute these through either REST based APIs or through a built-in scheduling mechanism that can execute test cases and suites based on a time interval and send email notifications. We do not recommend these options in the product – the scheduling mechanism and the possibility to send emails. Its two features that we will deprecate from the product because there are better alternatives – both free and commercial. It is recommended to use pipeline technology such as Jenkins to solve the test automation and email sending possibilities.