Jenkins Pipeline Script

After the pipeline has been triggered, the pipeline will execute the following code implementing the previously described steps.

Related topics

Basic-pipeline-example

Pipelines

Global variables

The first part initializes global variables that are specific to the environment the pipeline executes in and will neither differ between executions of any given job, nor between different jobs using the same script.

Name

Description

Git_URL

URL to the GitHub project containing the repository storing the TTT projects

Git_TTT_Repo    

Name of the repository

Git_Branch

GitHub branch to use for accessing the TTT scenarios

SQ_Scanner_Name

Name of the Sonar Scanner as defined in Jenkins Manage Jenkins->Global Tools Configuration->SonarQube Scanner

SQ_Server_Name

Name of the SonarQube server configuration in Jenkins Manage Jenkins->Configure System->SonarQube servers

SQ_Project

Name of the SonarQube project to use

MF_Source

Folder that the Code Pipeline download plugin will download the sources to

XLR_Template

Name of the XLRelease release template to use when triggering the release

XLR_User

Jenkins credentials token to use to connect to XLRelease

TTT_Folder

The code will download the TTT projects to this sub folder of the Jenkins workspace

TTT_Vt_Environment

ID of environment defined in the Total Test repository to be used for execution of tests

TTT_Sonar_Results_File

Name of the SonarQube results file generated by the Total Test CLI

CES_Url

URL for CES

ISPW_Runtime

Code Pipeline runtime configuration to use

ISPW_Changed_Programs_File

Name for the .json file created by Code Pipeline for Intelligent Test Case Execution

mailRecipientMap

Groovy Map to store Code Pipeline Owner Ids (TSO user id) and email addresses

 String Git_URL                     = "https://github.com/${Git_Project}"
 String Git_Ttt_Repo                = "${ISPW_Stream}_${ISPW_Application}_Total_Tests.git"
 String Git_Branch                  = "master"
 String SQ_Scanner_Name             = "scanner"
 String SQ_Server_Name              = "localhost"  
 String SQ_Project                  = "${JOB_NAME}"
 String MF_Source                   = "MF_Source"
 String XLR_Template                = "A Release from Jenkins"
 String XLR_User                    = "admin"                           
 String TTT_Base_Folder             = "Tests"
 String TTT_Vt_Folder               = "Virtualized_Tests"
 String TTT_Vt_Environment          = '123456789123456789123456'  
 String TTT_Sonar_Results_File      = './TTTSonar/generated.cli.suite.sonar.xml'
 String CES_Url                     = "http://ces.server:2020"
 String ISPW_Runtime                = "ispw"
 String ISPW_Changed_Programs_File  = 'changedPrograms.json'

 Map    mailRecipientMap            = ["ABC1234":"name@company.com"]

Node and stages

The node statement tells Jenkins which node (agents in a Jenkins network) to use. It may be used to distribute work, run jobs in parallel, or run steps in a job in parallel. This pipeline will use one node with several stages. All variables defined within the node are local to the node and available to all stages therein.

node{

Initialization

The example application uses three parallel paths (DEV1, DEV2, DEV3). In order to use the correct STEPLIB concatenation in the Total Test runner.jcl there are three versions of the JCL file in each Total Test project used. To determine the correct JCL file to use, the script determines the current path from the Code Pipeline level being passed to the pipeline. Using the path number, the name for the runner JCL to be used is being built and the next level in the path being determined.

def PathNum             = ISPW_Src_Level.charAt(ISPW_Src_Level.length() - 1)
    def ISPW_Target_Level   = "QA" + PathNum
    def CC_DDIO_Override    = "SALESSUP.${ISPW_Application}.${ISPW_Target_Level}.LOAD.SSD"    

Get the email address of the owner of the promotion set from the map of mail recipients.

   def mailRecipient       = mailRecipientMap[(ISPW_Owner.toUpperCase())] 

Stage 1 - Clear out the workspace

This stage cleans out the workspace from any previous runs of the pipeline. The call to dir(...) sets the location to the root of the workspace. Any code within the code block is executed in relation to this location. Therefore, the deleteDir() will delete the current directory including all sub folders.

 stage("clean previously downloaded source")
   {
        dir("./")
       {
            deleteDir()
       }
   }  

Stage 2 - Download sources from Code Pipeline

This stage downloads all COBOL sources and COBOL copybooks from Code Pipeline (the mainframe) that are part of the set triggering this specific pipeline execution, using the Code Pipeline Container downloader. The code has been generated by the Jenkins Syntax Generator using "Sample step": Checkout, "SCM": Code Pipeline Container.

Important

  • The containerType parameter tells the plugin which type container is passed via the containerName parameter. Valid values are:
    • 0 - Assignment
    • 1 - Release
    • 2 - Set
  • ispwDownloadIncl: true ensures that any copybook that is referenced by any of the components will be downloaded alongside, even if they are not part of the assignment. This ensures that SonarQube gets all required files and does not flag false negative issues.
 stage("Retrieve Code From ISPW")
    {
            checkout(
                [
                   $class:             'IspwContainerConfiguration',
                    connectionId:       "${HCI_Conn_ID}",
                    credentialsId:      "${HCI_Token}",
                    componentType:      '',
                    containerName:      ISPW_Assignment,
                    containerType:      '0',
                    ispwDownloadAll:    false,
                    ispwDownloadIncl:   true,
                    serverConfig:       '',
                    serverLevel:        ISPW_Target_Level
                ]
            )
    }

Stage 3 - Download Total Test assets from GitHub

This stage uses a Git clone to download the Total Test assets for the Code Pipeline stream and application. The convention used in the example is that the name of the repository is made up from the stream name and application name, e.g. FTSDEMO_RXN3_Total_Tests. The code has been generated by the Jenkins Syntax Generator using "Sample step": Checkout, "SCM": Git.

Important

Using the relativeTargetDir parameter results in the content of the repository to be cloned into a dedicated sub folder of the Jenkins workspace. This simplifies execution of the test scenarios.

  stage("Retrieve Tests")
    {
        Git_URL = "${Git_URL}/${Git_Ttt_Repo}"

        checkout(
            changelog:  false,
            poll:       false,
            scm:        [
               $class:                             'GitSCM',
                branches:                           [[
                    name: "*/${Git_Branch}"
                    ]],
                doGenerateSubmoduleConfigurations:  false,
                extensions:                         [[
                   $class:             'RelativeTargetDirectory',
                    relativeTargetDir:  "${TTT_Base_Folder}"
                ]],
                submoduleCfg:                       [],
                userRemoteConfigs:                  [[
                    credentialsId:  "${Git_Credentials}",
                    name:           'origin',
                    url:            "${Git_URL}"
                ]]
            ]
        )
    }

Execute test scenarios

Using the single Total Test CLI, we can use a single call to the Total Test plugin to execute all test scenarios for the desired environment (environmentId) and list of affected programs (jsonFile). The code for the latter has been generated by the Jenkins Syntax Generator using "Sample step": totaltest.

Important

 stage("Execute related Unit Tests")
   {

        totaltest(
            serverUrl:                          CES_Url,
            serverCredentialsId:                HCI_Token,
            credentialsId:                      HCI_Token,
            environmentId:                      TTT_Vt_Environment,
            localConfig:                        false,
            folderPath:                         TTT_Base_Folder + '/' + TTT_Vt_Folder,
           recursive:                          true,
            selectProgramsOption:               true,
            jsonFile:                           ISPW_Changed_Programs_File,
            haltPipelineOnFailure:              false,                 
            stopIfTestFailsOrThresholdReached:  false,
            createJUnitReport:                  true,
            createReport:                       true,
            createResult:                       true,
            createSonarReport:                  true,
            contextVariables:                   '"ispw_app=' + ISPW_Application + ',ispw_level=' + ISPW_Target_Level + '"',
            collectCodeCoverage:                true,
            collectCCRepository:                CC_Repository,
            collectCCSystem:                    ISPW_Application,
            collectCCTestID:                    BUILD_NUMBER,
            clearCodeCoverage:                  false,
            logLevel:                           'INFO'
        )

After execution of the test scenarios, the results are being passed to the JUnit plugin. This will generate a diagram showing development of the test results over time. Also, it allows you to drill into the test results to determine which test cases and assertions failed.

 junit(
            allowEmptyResults:  true,
            keepLongStdio:      true,
            testResults:        "TTTUnit/*.xml"
        )
   }

Stage 4 - Download Code Coverage results after test execution

This stage will download the code coverage results stored in the Code Coverage repository on the mainframe to the Jenkins workspace. The results will be placed in the Coverage subfolder. During download the plugin matches the downloaded source for each program it finds coverage data for against the corresponding DDIO file. The listing in the DDIO file must be preprocessed when code coverage is to be shown in SonarQube. This way it makes sure that code coverage information ending up in SonarQube actually flags the right statements as not being executed or being executed. Therefore the parameter cc.sources is required to point to the folder containing the downloaded sources. The properties defined by the cc. parameters need to be one parameter per line, therefore \r is being used.

The code for downloading the code coverage results has been generated by the Jenkins Syntax Generator using "Sample step": step, "Build step": Retrieve Code Coverage Statistics.

 stage("Collect Coverage Metrics")
    {
            def ccSources       = "${ISPW_Application}/${MF_Source}"
            def ccProperties    = 'cc.sources=' + ccSources +
                '\rcc.repos='           + CC_Repository     +
                '\rcc.system='          + ISPW_Application  +
                '\rcc.test='            + BUILD_NUMBER      +
                '\rcc.ddio.overrides='  + CC_DDIO_Override

            step(
                [
                   $class:                 'CodeCoverageBuilder',                    
                    connectionId:           HCI_Conn_ID,
                    credentialsId:          HCI_Token,
                    analysisProperties:     ccProperties
                ]
            )
    }

Stage 5 - Pass data to SonarQube using SonarScanner

This stage will pass the downloaded COBOL sources, the results of the unit tests, and code coverage metrics to SonarQube using the Sonar Scanner.

The tool method returns the installation path of to the SonarScanner. withSonarQubeEnv retrieves information about the SonarQube server, which then does not need to be specified via parameters for the SonarScanner. Since the Total Test plugin creates one result file per test scenario, the location and name of all of these files need to be passed to the SonarScanner via the sonar.testExecutionReportPaths parameter. The value will have to be a comma separated list of path and file names which is built in the initial stage.

stage("Check SonarQube Quality Gate")
    {
        def scannerHome = tool SQ_Scanner_Name

        withSonarQubeEnv(SQ_Server_Name)
        {
            bat "${scannerHome}/bin/sonar-scanner "                                                 +
                " -Dsonar.tests=${TTT_Base_Folder}"                                                 +
                " -Dsonar.testExecutionReportPaths=${TTT_Sonar_Results_File}"                       +
                " -Dsonar.coverageReportPaths=Coverage/CodeCoverage.xml"                            +
                " -Dsonar.projectKey=${JOB_NAME}"                                                   +
                " -Dsonar.projectName=${JOB_NAME}"                                                  +
                " -Dsonar.projectVersion=1.0"                                                       +
                " -Dsonar.sources=${ISPW_Application}/${MF_Source}"                                 +
                " -Dsonar.cobol.copy.directories=${ISPW_Application}/${MF_Source}"                  +
                " -Dsonar.cobol.file.suffixes=cbl,testsuite,testscenario,stub,result,scenario"     +
                " -Dsonar.cobol.copy.suffixes=cpy"                                                  +
                " -Dsonar.sourceEncoding=UTF-8"
        }

After the results have been passed to SonarQube, query the resulting Sonar quality gate, by registering a Sonar Webhook call back. timeout will wait up to the specified time for the results of the quality gate to be returned.

timeout(time: 2, unit: 'MINUTES') {
            def qg = waitForQualityGate()

The results can be queried by the status property. If the status is not okay, i.e. the quality gate failed, the assignments corresponding to the set will be regressed using the ISPW operation plugin, an email is sent to the owner of the set, and the pipeline being aborted using the error method.

The code to regress the assignment has been generated by the Jenkins Syntax Generator using "Sample step": ispwOperation.

Important

The currentBuild.result = "FAILURE" sets the result of the pipeline to "FAILURE" without interrupting execution. This allows the emailext to pick up the correct status when sending an email.

Only after sending the email do we abort the pipeline, using error.

if (qg.status != 'OK')
            {
               echo "Sonar quality gate failure: ${qg.status}"
               echo "Pipeline will be aborted and ISPW Assignment will be regressed"
               echo "Regress Assignment ${ISPW_Assignment}, Level ${ISPW_Target_Level}"

               ispwOperation(
                   connectionId:           HCI_Conn_ID,
                   credentialsId:          Jenkins_CES_Token,
                   consoleLogResponseBody: true,  
                   ispwAction:             'RegressAssignment',
                   ispwRequestBody:        """
                       runtimeConfiguration=${ISPW_Runtime}
                       assignmentId=${ISPW_Assignment}
                       level=${ISPW_Target_Level}
                       """
               )

               currentBuild.result = "FAILURE"

               emailext(
                   subject:    '$DEFAULT_SUBJECT',
                   body:       '$DEFAULT_CONTENT',
                   replyTo:    '$DEFAULT_REPLYTO',
                   to:         "${mailRecipient}"
               )
               
               error "Exiting Pipeline"
            }
        }   
    }

Stage 6 - Trigger XLRelease release

If the quality gate passes, i.e. the pipeline does not get aborted, an XL Release template will be triggered - using the XL Release plugin - to execute CD stages beyond the Jenkins pipeline. The code has been generated by the Jenkins Syntax Generator using "Sample step": xlrCreateRelease.

Important

The propertyName, propertyValue pairs reflect the parameters that are required by the XL Release template.

stage("Start release in XL Release")
    {
       xlrCreateRelease(
           releaseTitle:       'A Release for $BUILD_TAG',
           serverCredentials:  "${XLR_User}",
           startRelease:       true,
           template:           "${XLR_Template}",
           variables:          [
                [propertyName: 'ISPW_Dev_level',    propertyValue: "${ISPW_Target_Level}"],
                [propertyName: 'ISPW_RELEASE_ID',   propertyValue: "${ISPW_Release}"],     
                [propertyName: 'CES_Token',         propertyValue: "${CES_Token}"]
            ]
       )
    }
}


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*