Analyzing the performance profile for z/OS UNIX system services
Once your measurement request is complete and you have created the Strobe Performance Profile, you are ready to analyze and identify performance improvement opportunities from the Strobe Performance Profile.
This chapter is a guide for analyzing the Strobe Performance Profile for an application executing in a z/OS UNIX System Services address space. Refer to the Using-the-Strobe-Application-Performance-Measurement-System for a complete description of the Strobe Performance Profile.
About the Application
The example in this chapter refers to a Performance Profile produced for an application that had been recently ported to a z/OS UNIX System Services environment. The application uses information from a variety of investment files to calculate the present value of a customer’s investment portfolio and create a report detailing the investments. The application was originally written and executed in a client/server UNIX environment and was used by a single customer representative in a small branch office.
The application was ported so that a customer service representative could run the program. However, they soon detected performance problems in the application, which prompted the system administrator to use Strobe to analyze the application and look for performance improvement opportunities.
Verifying the Measurement Session Information
The starting point for analyzing any Performance Profile is the Measurement Session Data report. This report describes the computing environment during a measurement session and provides measurement statistics for should the measurement session.
Several report fields help you determine if the Performance Profile is valid and which reports in the Performance Profile you should analyze. To verify the validity of the Performance Profile, examine the following fields:
- In the JOB ENVIRONMENT column, ensure that the PROGRAM MEASURED, DATE OF SESSION, and TIME OF SESSION match your measurement request.
In the MEASUREMENT PARAMETERS column, ensure that the appropriate version of z/OS UNIX System Services appears in the SUBSYSTEM field.
- In the MEASUREMENT STATISTICS column, examine the RUN MARGIN OF ERROR PCT and the CPU MARGIN OF ERROR PCT fields. A run-time margin of error of less than 2% reflects a reliable measurement. A high CPU margin of error does not mean the Performance Profile is invalid. Rather, it indicates that you should focus your analysis on the reports that detail run time.
Measurement Session Data Report
After you verify that the Performance Profile is valid, the next step is to determine which reports in the Performance Profile to analyze.
Choosing Between CPU and Wait Reports
You typically do not need to examine all of the reports in the Performance Profile. To determine whether to examine the CPU execution or wait reports in the Performance Profile, check the MEASUREMENT STATISTICS column and the values reported for EXEC TIME PERCENT and the WAIT TIME PERCENT fields.
In Measurement Session Data Report, the EXEC TIME PERCENT value of 43.32% suggests that performance might be improved by reducing the CPU usage. CPU time is the run time during which the central processing unit was in use by application tasks executing within the measured job step. The WAIT TIME PERCENT value of 56.68% indicates that trying to curtail wait time might also improve performance.
In the traditional MVS environment, excessive I/O usually results in high wait; however, in the z/OS UNIX System Services environment, excessive I/O can cause high CPU usage due to I/O processing taking place in z/OS UNIX System Services address spaces. To determine whether the high CPU use is associated with data sets or CPU resources, first examine the Resource Demand Distribution report. If the resource is CPU and the service by CPU is high, then review the execution reports described in this chapter to determine which programs are responsible for the high CPU use. If the high CPU use is associated with data sets, then review the Data Set Characteristics report and the Data Set Characteristics Supplement report as described in Additional Reports To Examine and in the Using-the-Strobe-Application-Performance-Measurement-System.
In this section, the example focuses on reducing CPU usage associated with program activity. First we analyze the Resource Demand Distribution report to verify that program activity is responsible for the high CPU use. Next, we examine the following CPU execution reports:
- Program Section Usage Summary
- Program Section Usage By Procedure
- Attribution of CPU Execution Time.
Identifying High CPU Use in a z/OS UNIX System Services Environment
The first step in finding significant activity in a z/OS UNIX System Services address space is to examine the Resource Demand Distribution report.
The Resource Demand Distribution report lists the usage associated with specific tasks. In this example, more than 42% of the CPU use is associated with the program module /u/s-ort. The module /u/s-ort in the DDNAME or TASK column represents the initiating module. The percentage of 42% represents activity by the module itself and other modules that it called.
Because more than 42% of the CPU use is associated with the program module /u/s-ort, this report indicates that program activity is responsible for high CPU use. The following example walks you through the CPU execution reports for a task causing high CPU usage. A detailed description of the Performance Profile is found in Using-the-Strobe-Application-Performance-Measurement-System.
Resource Demand Distribution Report
The Program Section Usage Summary Report
Once you have identified that program activity is responsible for high CPU usage, the next step is to identify the modules and control sections that are the greatest consumers of CPU by reviewing the Program Section Usage Summary report. To investigate a module’s CPU consumption, first look at the TOTAL field under the % CPU TIME column and the histogram spike. In this example, the histogram spike indicates a CPU use of more than 86% (Program Section Usage Summary Report) for C/370 system support routines within the pseudo-module .SYSTEM.
The FUNCTION column shows that the routines are IBM C library subroutines. A high percentage of program activity taking place in the C/370 library routines is common for a program running in z/OS UNIX System Services. Next, refer to the Program Usage by Procedure report for the pseudo-module .SYSTEM to see which C/370 library routines are consuming the most CPU resources.
Program Section Usage Summary Report
The Program Usage by Procedure Report
The Program Usage by Procedure report details the time the CPU spent executing code within each control section of each module of the program or subsystem. In this example, the report shows CPU usage for all system service routines under the pseudo-module .SYSTEM. The header line identifies the pseudo-module (.SYSTEM) and the pseudo-section (.C370LIB). The detail lines show:
- MODULE NAME, the true module name or C library routines
- SECTION NAME, the control section name (if Strobe obtained one during sampling)
- FUNCTION, the function descriptor of the control section (if available) or the function descriptor of the module.
In this Program Usage by Procedure report, you see that more than 35% of the CPU time was associated with the IBM C/370 output routine EDCDFMTO. When system service routines identified in the Program Usage by Procedure report show high CPU time, examine the Attribution of CPU Execution Time report to pinpoint which application code lines invoked the system service routine.
Program Usage by Procedure Report
Identifying Performance Improvement Opportunities in the Source Code
The Attribution of CPU Execution Time report (See the following figure) identifies the lines of code or the offsets within the code that called the system routines and are therefore responsible for high CPU use. In this report, the IBM C/370 library routine CEEEV003 shows that it was invoked by the application module /u/s-ort and section name CUSTOMER.
Attribution of CPU Execution Time Report
In this example, /u/s-ort is a token. Strobe generates a token when a module’s name is longer than eight characters. A token comprises the first four characters of the module or section name followed by a hyphen (-) and then the last three characters of the name. Refer to the Token-Longname Cross Reference report to determine the longname of the token /u/s-ort.
Token - Longname Cross Reference Report
Now that we have determined the actual module name, our next step is to examine the Attribution of CPU Execution Time report to find the location where the application invoked the system service. Note that this Attribution of CPU Execution Time report is indexed; for the modules identified, it displays the source code line number and procedure name in addition to the hexadecimal starting location and interval length. For more information about indexed reports, refer to the Using-the-Strobe-Application-Performance-Measurement-System.
In this example, the area to look for performance improvement opportunities in the source code is in module /u/s-ort at line 70.
Attributing CPU Execution Time Related to Source Code
An examination of the source code for line 70 (Sample Code for Investment Calculation Print Routine) reveals that in control section CUSTOMER, the code is invoking the print routine that produces the investment calculation reports. While the print routine is using a significant amount of CPU, the routine is apparently performing a normal write function and cannot be changed to improve the application’s performance. Instead, our next step is to examine the Program Usage By Procedure report again and locate another system service routine using a high amount of CPU.
Sample Code for Investment Calculation Print Routine
The Program Usage By Procedure report indicates that more than 7% of the CPU usage is associated with the IBM C/370 floating point conversion routine EDCDOFCV. Next, examine The Attribution of CPU Execution Time report to locate the area in the source code that invoked this subroutine.
The Attribution of CPU Execution Time report (See the following figure) for the IBM C/370 floating point conversion routine EDCDOFCV shows that it was also invoked by the application module /u/s-ort and section name CUSTOMER at line 70. This routine is not only performing the normal write function as in our earlier example, but it is also performing floating point conversion. Since the write function cannot be changed, next examine the floating point conversion code for performance improvement opportunities.
Attribution of CPU Execution Time Report
The floating point conversion routine embedded in the printf statement (Sample Code for Investment Calculation Print Routine) is performing a mathematical calculation for the final investment portfolio report. In this example, the variable definition of a, n, r, or i could be causing performance opportunities. Your next step is to see how the variables are defined.
In the following figure, variables r and i, as defined in lines 00025 and 00026 of the source listing, are handling double precision numeric values to complete the calculation task. Upon examination, it was determined that 32-digit arithmetic is not required for these calculations. The calculation was then changed to single precision integers, and a subsequent measurement of the application revealed a significant reduction in the overall CPU use associated with the application.
Variable Definition for IBM C/370 Floating Point Conversion Routine
By comparing the Measurement Session Data reports for both measurements (Measurement Session Data Report and Measurement Session Data Report (After)) you can see a significant reduction in CPU session time, as well as an overall decrease in elapsed time.
By examining and comparing the CPU TIME fields, you see a total decrease in CPU time of almost two minutes.
Measurement Session Data Report (After)
Additional Reports To Examine
To streamline file access activities of z/OS UNIX System Services routines, analyze the Data Set Characteristics and the Data Set Characteristics Supplement reports to identify the data sets exhibiting file access performance opportunities.
Data Set Characteristics Report
The Data Set Characteristics report shows the characteristics of all data sets accessed for the measured job step during the measurement session. The report provides the data set names associated with HFS or z/FS files listed on the Resource Demand Distribution report. For files that are allocated to the measured task, the ddname will be the z/OS ddname. For all other files, the ddname will be .HFSnnnn or .ZFSnnnn, as shown in the following figure.
Data Set Characteristics Report
For HFS data sets, EXCP counts are collected only if Strobe for z/OS UNIX System Services is installed at your site. Verify that the block size is optimal for access to the data stored upon the device.
Data Set Characteristics Supplement Report
The Data Set Characteristics Supplement Report provides additional information on data sets. You can cross-reference this report with the Data Set Characteristics report by data definition name, access method, and data set name. If no information exists, Strobe suppresses the names of the fields. If a data set is accessed from an HFS or z/FS system, the Data Set Characteristics Supplement Report details the following:
- INODE, displays the file identifier
- FILE SYSTEM TYPE, the type of file system defined in the system PARMLIB
- DEVICE NUMBER, an identifier for the mounted file system
- I/O BLOCKS, the number of blocks read from and written to the file during the measurement
- FILE TYPE, the definition of the file. This field can have one of two values: REGFILE for a normal file or FIFO for a named pipe
BYTES, the number of bytes read from and written to the file during Strobe measurement.
Data Set Characteristics Supplement Report
Strobe reports the I/O blocks and bytes statistics only if the System Management Facility (SMF) is configured to produce type 92 records.
To reduce file contention problems, See I/O Facility Utilization Summary Report to see what physical devices are used and where the files reside.