Best Practices for Test Automation using Total Test

This section gives an introduction to test automation. 

Automate Testing to Shift Left

Whether you’re a COBOL developer creating unit tests for code, or an application test domain expert validating that your programs meet business needs through functional or integration tests,  you can use this guide to get started with automated testing. This document will guide you through best practices for using BMC’s automated testing tool: BMC AMI DevX Total Test (Total Test). It covers topics that guide both, organizations and individuals in their efforts to shift left. Additionally, there are tutorials on how to organize test cases and minimize the work of creating, executing, and maintaining unit, functional and integration tests using automation.

There is a lot more to successfully automating unit testing on the mainframe than implementing a new tool, which is, of course, critical to enabling automation.

Frequently used terms and concepts described in this document are explained in Glossary.

Summary of best practices

This document contains information and best practices of using Total Test and getting started with Test Automation. Therefore, let us summarize the most important tips in this best practices list:

Start with:

1. Develop an organizational approach: as an organization, start with one dedicated team, get experience and success, and then expand the adoption
2. Adopt Total Test as individuals: As a user or team, pick simple programs first, learn the tool and its limitations, and then expand to more complex programs and test scenarios.

Then be sure to:

1. Do different kind of testing – the successful path, boundary value testing and negative testing.
2. Use Code Coverage to understand the quality of the test cases.
3. Share Test Cases with a Source Control Management (SCM) System: use an SCM to share test cases.
4. Integrate into CI/CD pipelines after getting success with manual execution. CI/CD pipelines are not critical to get most success with test automation. The execution can still be manual.

Develop an organizational approach

When you, as an organization, want to adopt Total Test in your development organization, we recommend the same principles of adoption as for individual users.

1. Start small and simple
2. Get experience
3. Get successful results
4. Expand the adoption

Adopting Total Test is more than just starting to use a new tool. It is about changing developers habits. This can be very hard for many people, especially for very experienced persons that have been working the same way for decades. There will be resistance to change, and there will be complaints that it's too difficult or impossible to do test automation.

These complaints are not specific to Total Test, this is generally true for adopting tools that change work practices and habits. The same experiences were seen 30 years ago in another industry; mechanical engineers and designers getting CAD (Computer Aided Design) systems to make drawings of designs digitally instead of on paper. It took many years to adopt the new technology, and today it is unthinkable to not use CAD software in engineering design processes.

We believe the same is true for test automation on the mainframe. This is process is new to developers and the adoption has to be gradual. Big bang implementations where everybody must start writing and executing tests from the same starting day will not give the expected results. Instead, since there is no experience in house and no success stories to tell, it will cause a lot of frustration and resistance and there is a risk that developers will simply refuse to use the tools. Again, this is not specific to testing, the same result has been seen in organizations adopting a modern IDE to replace ISPF interface. This also causes challenges.

Organizational adoption of Total Test

These principles can be applied to adoption other tools, specifics here will apply to Total Test.

Pick a small team

The team should be motivated to get started using Total Test. It will be good to have a mix of younger developers and senior developers. What we often see is that the younger developers have a better understanding of the need for test automation and DevOps, they have learned this in college, and they are often more motivated and have easier to learn new methods and tools. Their minds are still fresh and not fixated by old habits. The senior developers might have resistance in adopting tools – but when they see the results produced by the younger developers, they often get motivated. They see how test automation can actually help them be more productive and increase quality. They get more time to do what they find interesting, and that is to develop business functionality.

Benchmark time spent in manual testing: If possible, have the team measure how much time they use on manual testing when they implement changes and deploy these.

Measure how many bugs make it to production: estimate how many bugs that are leaked into production.

These numbers are interesting since they can later be compared to similar numbers after adopting test automation and Total Test. The comparison can be used to calculate ROI seen from a pure cost/quality perspective.

Carefully choose programs

The team should choose a number of programs to get started with. This document later describes how to pick these programs. Start simple to learn the tool and get success, and then increase complexity. Programs that are frequently changed or are business critical are best candidates for test automation since this is where you get most benefit of automation. If the team finds programs that cannot be tested due to various reasons, then focus on other programs that can be tested.

Encourage developers to use the steps in "How to get started as a user." The team should approach learning and adopting Total Test by following the steps suggestion in “How to get started with Total Test as a user”, and understand the best practices in this document.

You don’t need all processes in place from the start, and you don’t need 100% test coverage. Most important is to get started building up a regression test suite and get experience in the organization. Processes and automated execution in pipelines are secondary and can come later.

After creating test cases for a selected number of programs, the team should evaluate the use of testing. How much time will be saved going forward, now that automated testing is in place? Time used on creating the initial tests should not be taken into account. There will always be an initial cost of creating test cases. The benefits come when using the tests in a regression test suite the next times programs change. Also what has the impact of leaked bugs to production been?

After the initial experiences and success with the first team, you will have ambassadors internally that can help spread the message about the benefits using Total Test and test automation. This will make the continued adoption much easier since other teams can see and hear the value from coworkers.

Establish best practices

Use the rest of this document as inspiration. Work closely with one team on describing, establishing and adopting the processes. For instance, we recommend to adopt a Test Driven Development/Maintenance approach where you always start to reproduce a bug by creating a new test case that fails, then you fix the bug and then rerun the test case until it is successful. Use of a SCM to share test cases is another important best practice that should be adopted. No developers should have test cases only on their machines. The use of code coverage is also a practice that should be encouraged. Actively using code coverage will make sure that a program is tested sufficient.

Establish a “Test Automation Knowledge Center” or what you want to call it. You should have at least one expert user of Total Test and who has understanding test automation and DevOps that can help coworkers getting started, educate team in understanding best approach for success, and help solve issues. The expert user should also be the one responsible for having contact with BMC Support to create and update support cases. Having expertise in house to do first level support of internal users learning to use the tool is the fastest way to improve adoption.

Document processes an develop naming standards

Document the following: 

• Test Scenarios and test cases
• What a test case is testing
• The testing schedule
• The number of programs being tested

As the number of test cases for each program grow, it becomes necessary to document what each repository, project, test scenario and test case covers so when getting back to these it is easy to understand what the specific resources are meant for.

It is very important to document what a test case is testing. There are test repositories that contain hundreds of undocumented test cases. When many test cases start to fail, it is extremely difficult to know where to begin fixing them. In these situations, test cases are often thrown away because no one knows what they were attempting to test. What a tremendous waste!

The documentation for a test case can be very straight forward. If you don't know what the purpose of the test case was, you don't know why it failed and you can't fix the test or the program when it fails. Documentation should include:

• A short description (a sentence or two) describing what the test was supposed to verify
• Any additional details about any setup required to execute the test
• Any limitations where the test won't work

It is possible to write documentation directly into test scenarios and test cases, and for functional test, each of the palette elements used in a test scenario can also be documented. Currently it is not possible to write documentation to a test project, and a solution to this could be to add a readme.txt file to the project and write the documentation of the project here.

Use Intelligent Naming Conventions

It might seem unimportant but providing a good name for your test case is important. Use intelligent naming standards for your unit test cases. The entire development team for an application should adhere to using the same naming convention.

A name should provide some indication of what the test is doing and what is being tested. A name that immediately helps people understand what the test is for makes it easier every time someone new has to work with the test or every time a test case fails. A name should provide immediate clues about where to start diagnosing the test failure. We recommend including the name of the:

• Program/subprogram being tested
• Feature being tested
• Business rule being tested

Adopt a Naming Standard

It is a very good idea to define a naming standard for repositories, projects, test scenarios, test cases, and test suites. The naming standard will of course depend on how you use these artifacts to divide your testing artifacts. If we follow our general recommendation for using a repository for an application and a project for testing one program, and test scenarios for testing different scenarios of the program, then this is a suggestion.

Use Descriptive Messages in the assertions

When creating the unit test and authoring the assertion messages, think about what you would want to know when the test fails. If this assertion fails, what should you look for? Is it a consistent error that doesn't need investigation? If you decide to keep your assertion messages short, include a comment in the test case providing more information about why the test case could fail.

Total Test generates a default assertion message, which just indicates what field values don’t match the expected values. You can edit the assertion message with more information to better describe a test failure. It is best to focus on failures that you have already encountered rather than creating messages for everything. If the failure needs more description, it can be added into the Test Scenario Properties tab in the description box.

Organize your tests

This section describes the best practices to organize test cases. 

Use Sub Folders for Test Cases

When the number of test cases and programs being tested grow, it is helpful to organize test scenarios so it is easy to understand what specific test scenarios are actually testing. Documentation is important, but structuring test scenarios into a folder structure can help a lot to get an overview and to navigate to a concrete test scenario. Folders can be used in both unit and functional testing, and you can place them in any folder underneath the project. For unit tests, the complete unit test folder must be copied to a sub folder. One example of creating a folder structure is to break down an application into functional areas and sub areas and create a similar folder structure. For each area or sub area you can then add unit and functional test scenarios.

Use Test Case Categories

Organize your test cases by category to ensure the tests for test cases are similar. It is best to know what the test categories are before you start generating test cases. Categorization can also help you organize test cases into test suites. Furthermore, in writing descriptions of each category, you will help people understand which categories should hold their test cases. Possible categories might be:

• The application a program is part of
• The feature(s) a program implements
• How fast or slow a program executes, which might help you decide if you should run unit tests only in full regression tests (longer-running tests) or if you could run them also during smoke tests (quick-running tests)
• Resources needed; for example, if a Db2 database is required

Whatever you decide, categories should remain consistent. You want these categories to stand the test of time, otherwise you will be creating more maintenance work than benefits with categorization.

You can put the category name in the Test Scenario or Test Case Description box on the Properties Tab in Total Test. You can use the Search | Files… feature of Total Test/Eclipse to find the files using the “Containing text” entry box with your category name.

Group Test Scenarios into Test Suites

A test suite is a way to logically group-test scenarios. When you create a test suite, you should decide what the organizing theme of the test suite should be. There are many reasons to organize test scenarios into test suites. For example, you may want to regression-test an entire application or a program, or you may want a quick smoke test to determine if a build was successful before running the entire regression test suite. Use a descriptive naming convention for test suites and document the goal or reason for why a test suite exists:

• Determine the goal of the test suite
• Determine which test cases help meet the goal of the test suite
• Create a naming convention for each type of test suite (Smoke Test Suite, Component Test Suite, Application Test Suite, Regression Test Suite, etc.)
• Document the purpose of the test suite on the Properties Tab in the description box of Total Test.

Include Code Coverage

A test all by itself does not guarantee the quality of your code, if the test does not execute or test the functionality of your code that requires testing. This might be a crucial (potentially complex) function, or a part of code that you modified in your previous code change.

Measuring the code coverage of your tests, i.e. determining those code lines that were executed by your tests, and setting them in relation to all code lines, is one of the most common means of measuring and improving the quality of your tests. Typically, code coverage can be determined per test (or test scenario), executed, and be accumulated across executions of several tests (or test scenarios).

Identifying parts of code that have not been executed help you determine missing test cases. Spending time and effort to create these test cases will ensure that more and more parts and functions of your code are tested before the code is moved to production.

On the other hand, especially considering existing mainframe code and the size and complexity of many mainframe programs, trying to reach full code coverage will often not be a realistic goal. Still, the target should always be to test all areas of code that have been modified during any specific change. Over time, this will help increasing code coverage almost automatically.

Use Pipeline Technology for Test Automation

First step in test automation is to have test cases that can be reused again and again without the need for manual setup. With this first step, a developer or tester initiates the execution of the test scenarios and evaluates if the execution was successful or not. This brings a lot of value since these regression test the applications and ensure that no bugs have been introduced. The problem with this first step is that someone must remember to find, execute, and evaluate the test scenarios. This is where pipelines come in. A DevOps pipeline is used to automatically test, build and/or deploy changes. They are typically called Continuous Integration/ Continuous Delivery/Deployment (CI/CD) pipelines since they either automatically build or deploy code when needed. Total Test fits very well into pipeline technologies such as Jenkins, Electric Cloud and XebiaLabs XL Release where we have dedicated plugins. It can also fit into other pipeline technologies as well through the use of the Command Line Interface (CLI).

Within a pipeline, you can automatically checkout test scenarios and test suites from a SCM like Git, then find and execute the test scenarios and test suites, and produce output reports in Junit output and Sonar Qube format. Hence you can view the test results in Jenkins and Sonar Qube in case a test failed, and then go back to your IDE and reproduce and fix the problem.

Adoption of CI/CD pipelines typically happen after an organization has adopted unit/functional testing tools and wants to improve their testing practices. It is the next natural step in a DevOps journey.

Having pipelines responsible for the execution and evaluation of the tests, ensures that changes are tested and hence lowers the time to make a change, at the same time, leaked bugs are reduced.

Once the development or maintenance of the test cases is complete and the cases are running successfully in their test suites, you are ready to add test steps into your new or existing pipeline scripts. The pipeline scripts can be scheduled or executed on demand. We recommend an initial short test suite being first in the pipeline script to verify that tests are running correctly, a smoke test to verify that the configuration and security authorization are set up. After the initial smoke test scenario, you can add the more detailed and longer-running test suites.

The combination of using a SCM for sharing test cases and a pipeline technology like Jenkins is strong. When it is first setup, developers don’t need to worry about automation. When they add or update a test scenario and commit the changes to the SCM, the pipeline can automatically checkout the new or modified test cases and execute them without any additional work.

Adopt Total Test as an individual

This section describes the best practices to adopt Total Test as an individual user. 

Setting up your first test

Start simple, with unit testing. When you are new to Total Test and test automation, there will be a learning curve to use the tool, to adopt new development and testing processes and getting started with test automation. Therefore, we recommend starting simple with the first test cases to simply learn how Total Test works before moving into more complex program testing. The simplest program to get started with is a called BATCH module which does not have any external dependencies or just uses input/output files.

The following is a suggested list of activities to get started with 

Total Test

:

1. Install and setup required software and install Workbench for Eclipse
2. Find a simple Batch program for the first test case
3. Record test case from Code Debug launch configuration
   If you don’t have Code Debug you cannot record a test case and you will have to create it manually- This is easiest to do with functional testing where you don’t have program and data stubs
4. Execute test case and see result
5. Modify test case with different expected values, execute and look at assertions failures in result
6. Do the above with a more complex program
7. Get experience with Code Coverage and use this actively when creating and executing test cases.

When you have become familiar with recording and executing test cases with Total Test it is time to move on in the learning journey of DevOps and test automation. Next step is to start using a SCM to share test cases. We recommend using a Git repository, but other repository technologies work fine as well.

Planning for testing

First steps in the process of identifying and prioritizing programs that you should cover with unit and functional tests include:

• Create a list of applications
• Identify the importance and risk of these applications
• Create a list of programs for each application

Your source code repository is a good place to start identifying applications, components that make up applications and programs that make up components.

Identifying What Programs Should Be tested

Use information from your source code management (SCM) system to identify programs. Prioritize building and executing test cases for those that are modified most frequently.

Initially, look at programs that had changes in the last 90 days and sort by programs with the most changes. Then look at programs that changed in the prior year and see how many programs are found in both sets. The intersection of those two lists probably provides a good indication of programs that continue to evolve and that will provide good payback for creating unit and functional tests.

Prioritize tests for High-maintenance Programs

Programs don’t have equal importance and don’t have equal maintenance requirements. We recommend selecting a set of programs that regularly require maintenance as the place to initially start the process of building test cases. You will get the most benefit from automated testing on programs that require frequent re-testing.

In a mainframe environment, you generally have large number of programs that aren't automatically tested. When planning for testing, think about the return on your test automation investment. The following should be high on your list of priorities for automated testing:

• Complicated programs that change frequently
• Programs executing complex and complicated financial or statistical computations
• Programs that bear a large commercial or financial risk if incorrectly coded
• Programs that require constant maintenance because of regulatory requirements or other changing requirements
• Programs that have many test permutations that are difficult to do manually
• Programs that are impossible to test manually because of the data the program uses
• Programs that people are afraid to change
• Programs with high McCabe complexity calculations

Determine Quick Wins

From those programs high on your priority list, go for some quick wins first. Immediately trying to tackle the monoliths in your environment is likely to fail and result in frustration. The faster developers see first successful results, the faster they will gain the necessary experience, confidence and conviction to try unit testing more daunting programs.

One sign of a program qualifying as a “quick win” is how easy it is to isolate the program from its environment. As we explain later in this document, programs being tested, as well as their unit tests, should not access live databases or other external data sources, nor should they depend upon called sub-programs to be executable.

Total Test allows you to define stubs and other means of “simulating” a working environment for the program in question. However, the high variance of possible data sources and data-access functions on the mainframe makes it difficult for unit testing tools to cover every possible constellation.

Although vendors do their best to provide full support over time, there are gaps in what is possible to stub out. Therefore, when in doubt and you have the choice between two programs, go for the one with a more complete set of interactions that can be simulated.

Finally, it may have been a decade since some mainframe software was completely rebuilt. Infrequently changed programs should be low on the priority list for getting unit tests.

Create a Test Plan

Think about the risk and cost of being wrong versus the cost of writing some tests. Look around at what you already have. You may not be doing automated testing, but you probably have been doing manual testing.

You may be able to use your manual testing plan as a blueprint for creating your automated tests. A good first step would be to run the tests manually while running code coverage to get a sense of how complete the manual test cases are.

Create Tests: Unit, Functional or Integration

With Total Test you can create unit, functional and integration tests of your program. So, what kind of testing should you choose when you create your test plan? The answer is that you probably want them all. In general, you should start focusing on unit testing. A developer creates the unit tests and should be able to locate bugs as soon as they are introduced. With unit testing it is also possible to simulate error scenarios to test error routines – testing that can be very difficult to do with functional testing that is executed at live systems with live data.

When you have unit tests for a program you would probably also want to verify that your program executes “live”, and this is where functional testing comes in. It can be the developer that creates the functional test, but it can also be a QA person that only knows what the program is expected to do. The QA person sets up input data for the program and also sets up expected output of the program. Input and output data in a functional test can be data in DB2, datasets, and through the linkage section. Total Test functional test supports all of these.

Verify test cases

This section describes the best practices to verify test cases. 

Verify Test Cases before making code changes

Before you apply any changes to a piece of code, you should make sure you have a set of tests that capture the current behavior of your code. Use code coverage to make sure these tests cover a sufficiently large percentage of your code, and make sure these tests still run without failures. These tests will serve a two-fold purpose:

• Regression tests for those parts of your code that don’t and should not undergo any changes
• Baselines for testing your changes if they already execute the parts of your code that will undergo changes

In the latter case, these tests will have to be re-baselined, either before or after you change your code. Re-baselining is the process of updating your test cases to work with newly changed program code.

When you first create a test case, verify it is correct. This test case is considered a baseline test. As programs change, you may have to update the test case to validate the updated program. We call the updating of a test case to work with an updated program re-baselining the test. For example, you may have to update assertion values if program calculations change or add new assertions if new data values are used.

If you find that your current set of test cases do not achieve sufficient code coverage (or if you don’t have any tests at all), be sure to define missing tests before you change your code, thus capturing its current behavior. You want to validate that unexpected changes have not been introduced into the program.

Fixing bugs and Test Driven Development

It is a best practices in test driven development to not fix a bug before you have a test case that produces the bug. Therefore, if you find a bug in a program, start with creating a test case that reproduces the bug, and defines how the program is intended to work. Then start to fix the bug and rerun the test case until it is successful. Finally add the test case to your regression test suite. Following this process will ensure you that the bug has been fixed and that it will never appear again. The regression test suite will make sure of it.

Run Unit Tests After Code Changes

This recommendation may seem obvious, but we’ve encountered customers who were so concerned about the cost of running their unit tests that they tried to limit the number of times they executed them. The cost of requiring people to relearn the code to fix bugs is higher than the cost of running a test frequently. The cost of letting bugs move into the next phase of testing is higher than the cost of rerunning tests. If you’re concerned about the cost of running tests, make sure you don't have duplicate tests, and remove tests if no one knows what they test.

Managing test code

Managing test cases for a small number of programs is easy – but what happens when we have thousands of programs to be tested in multiple different applications? How should we organize our test cases with such huge number of tests in so many different contexts? How many SCM repositories do we need? How many test projects do we create? How many test cases can we have in one project? How many different programs should we test in one project?

There is not one definite answer to all these questions, and the way you choose to structure test cases into folders, projects and repositories depend on the complexity of your organization, applications, internal processes etc. Here we will try to give some overall recommendations that can hopefully help in determining your strategy in organizing tests

Repository – One per Application

Most SCMs operate with a repository that has the collection of folders that you share. With a repository you can create branches so you can work on multiple versions and features concurrently. Therefore a repository should be used to contain only the folders and files that share the same life cycle and that belong to each other. Otherwise you will have to merge changes with developers that work on test cases not related to your programs/application. The simplest way of using repositories is probably to have one repository per Application. If you have a very large application maintained by multiple teams you might want to also use multiple repositories for one application. One approach could be to have one repository per application team.

Project – only unit test one program in a project

An Application typically contains multiple programs. To make it simple and clear we suggest to only test one program in a test project when its related to unit testing. You can have multiple test cases of the same program in the same project. If you have a lot of test cases for one program you might want to create multiple test projects, each one focusing on a specific area of testing. It is only a suggestion to use one project for one program. For some use cases if might be more pragmatic to have tests of multiple programs that relate closely to each other in the same project. In that case it is suggested to only test one program per test scenario. Again, this is also just a suggestion to simplify the orchestration of tests, and you have the flexibility to orchestrate and organize your test cases into test scenarios, suites folders and projects as you find best.

For a project containing only functional tests this “Test one program per project” is not as strong. With functional testing there are more freedom to place test scenarios in a folder structure which makes it easier to separate test scenarios and different program testing from each other. If you choose to have functional tests of multiple programs in one test project, then we recommend to use a folder structure to separate test scenarios for different programs, e.g. by having one folder for each program and under each program folder organize test scenarios for that program into sub folders.

Folders – Organize Test Scenarios into Feature Testing

In a test project you can organize test scenarios into a folder structure. This makes it easier to understand what specific test scenarios test, and it makes it easier to see where all the generated test stubs belong to. In this way – if you have generated 20 test cases, you would have 20 sub folders describing the scenarios. Organizing tests into folders also makes it easier to work concurrently. (Using subfolders is not yet available for unit tests (19.6.1)) With functional testing its possible to place test scenarios in any folder making it easy to organize the tests.

Test scenario – Only test one program per scenario

With a test scenarios – both for unit and functional testing it is possible to add test cases of different programs. We recommend to only test one program within a test scenario. It will make it clearer to understand the purpose of the test scenario and it makes it easier to only test one program what that has changed. A test scenario that is made for integration testing will of course contain test of multiple programs since this is the definition of an integration test.

Use Branching for Multiple Versions

If a program exists in multiple concurrent versions, test cases must also be available in multiple versions. This is both the case if structures (linkage section, copybooks etc.) change or if program logic is changed. When structures change, test cases and data must be updated to reflect the new structures, and when program logic changes, it must be ensured that there are test cases that cover the logic and ensure that its bug-free. In both cases it might be necessary to have test cases for the new and old version of the program and be able to switch back and forth. The technology to use for this is “branching”. Most SCMs have the possibility to create branches, and this means work on a “copy” of the code/test cases and have the original version intact. With branching its always possible to go back to the old version, and its possible to concurrently work on the old and the new version – and later merge the changes together. Its only necessary to create branches if you want to work on two different versions. You can always check out an older version of the test cases and use for the older version. Its only when you want to change test cases for the older version and you or someone else already have started to work on test cases for the newer version that you need to create a new branch.

Manage Concurrent Test Case Development

If you have more than one developer creating test cases for the same program at the same time, you will have to coordinate a bit between developers, or setup some naming standards. Lets see what kind of problem you can get into with concurrent development:

Share Test Cases with an SCM

Test cases should be viewed as first class citizens – they are as important as the source code of the programs they are testing since they ensure the quality of delivered applications. Test cases should therefore be shared in a Source Control Management system (SCM) as their counterpart source code is. Hence, developers and testers can collaborate on them and manage test cases for different versions of the programs they are testing. Testing of programs should never depend on test cases saved on individual developers or testers workstations.

Save Test Cases in an SCM

Once a test case is working, save it in your source code management (SCM) solution so it can be tracked. Provide some way of linking the version of program code to the correct version of the test case. You can use branching in the SCM that correspond to the different versions of the source code of your applications and programs, and you can use tagging to tag specific commits in the SCM and describe what source code version the state of the testcases at that point corresponds to.

Choose an SCM

With Total Test, unit, functional and integration test cases are all XML based files and can be shared in any SCM system that supports decentral files. This can be Git, Subversion, MS Team concert, IBM RTC and so on. We recommend using Git since this is an open, very strong and very popular SCM that is used by multiple open source project and business and has multiple free and commercial offerings. And test cases shared with Git can easily be integrated into all modern IDEs and pipeline technologies.

Version Control for Test Cases

Do not use the Total Test web repository to version Test Cases. Total Test Functional testing also provides the possibility to create test cases through a web browser. These test cases are saved in a local repository at the Total Test server and can be executed multiple times – even through a REST API. Using a web browser and a built-in repository to create and execute functional tests of a load module is very easy and is a good way of getting started with test automation. However, sharing test cases in the server repository is not recommended practice. They cannot be maintained and its harder to integrate into DevOps pipelines. Instead we recommend importing these test cases into Total Test and where they can be maintained, shared and automated with a real SCM.

Reuse existing Test Cases

Before you create a new test case, take a quick look in your test case repository to see if there is already a test case available to test your code. There may not be an exact-match test, but you might find one that can be easily modified that will save you the work of building one from scratch.

Remember, you don't want to have duplicate test cases. If you created a new test case and it’s working reliably, put it into the repository as soon as possible. If you wait, someone else might duplicate your work—or worse, you might lose track of the test case, losing the benefit of getting it executed with the rest of the regression tests. Also, keep the JCL to run the test case with it in the test repository. Note that functional tests through Total Test can, with limits, automatically generate JCL for test cases for different environments

Setting up a test repository using a GitHub repository

Git is one of the most common SCMs that allow you to manage files stored in an Eclipse project, and GitHub is one of the most common server implementations of Git. Therefore, it is important to discuss the use of GitHub to implement a test repository.

Following the recommendations mentioned in the preceding sections, you will need to create a GitHub repository for all tests for one application. The repository will start out empty.

In your Eclipse IDE, you must create/record Total Test scenarios. Again following the recommendations mentioned in the preceding sections, you will need to create one project for each program, containing sub-folders for each test scenario.

By using standard Git commands, you will first create a local Git repository, and second, add your test projects to that local repository. Alternatively, there is a Git Plugin for Eclipse that simplifies the use of Git without having to know individual Git commands to perform the required tasks. Once the local repository had been setup, you connect it to the GitHub repository and push your local repository to the GitHub repository, allowing other developers and tools to make use of the tests you have created.

Setting up an automated pipeline

Once your tests are shared in a centralized SCM repository, automation tools like Jenkins will be able to access these test, and you can setup automated CI build pipelines for your mainframe code by using your tests.

In this section

• • This section gives an introduction to test automation. 
  • Automate Testing to Shift Left
  • Summary of best practices
  • Develop an organizational approach
  • Organizational adoption of Total Test
  • Organize your tests
  • Include Code Coverage
  • Use Pipeline Technology for Test Automation
  • Adopt Total Test as an individual
  • Verify test cases
  • Managing test code
  • Use Branching for Multiple Versions
  • Manage Concurrent Test Case Development
  • Share Test Cases with an SCM
  • Setting up a test repository using a GitHub repository
  • Setting up an automated pipeline