Implementing Candidate Processing and Monitoring

When detection of abnormal behavior occurs, such as when a program’s elapsed time is too long, the program’s CPU consumption is high, or the ratio of I/O to CPU is not within normal limits, monitoring of the candidate begins. After identification, measurement begins of those programs with performance that deviates from normal processing.

Follow these general steps to implement candidate processing and monitoring:

1. Identify mission-critical applications or applications that execute as part of the critical path in the batch processing system. From these, identify candidate programs: for example, those with long runtimes or that use large amounts of system resources. You can select these yourself by either adding a monitoring request, using the SMF Candidate utility, or by waiting for the AutoLoader to determine what are good candidates.
2. Create and submit monitoring requests that target these programs. Then screening will occur for subsequent executions of these programs.
3. Check the status of the monitoring requests. During the first ten executions of the program, collection takes place of basis data — data that defines the program’s normal behavior. This basis data establishes norms for the program’s elapsed time, CPU time, and I/O activity.
4. View the basis data to verify that it is a valid representation of the program’s normal execution. Look at the threshold values that are calculated for elapsed time and CPU time to serve as the boundaries of normal processing. If you think any basis data record are not representative of normal processing, you can delete them. New basis data records will replace the deleted records and the thresholds will be recalculated.
5. Once the basis data exists, subsequent program monitorings are to detect abnormal behavior. When a measurement initiates because of abnormal behavior, it will notify you via a TSO send operation.
   When abnormal behavior is detected, the request associated with the program is put into the “held” state. This suspends monitoring of the program to prevent a continuing detection of the same abnormality on subsequent executions and possibly initiate redundant measurements.
6. Once the initiated measurement has completed, create a Performance Profile and change the program based on your analysis of the Profile. You then may want to measure the program again with Strobe to verify the effect of the changes.
7. Because you have changed the program, the existing basis data is probably not representative of its improved performance. Before monitoring of the program resumes, you should delete the basis data collected before the program was changed.
8. Release the request associated with the program to resume monitoring of the program’s execution and, if necessary, collect new basis data.
9. Periodically review and verify the basis data to ensure that it reflects the program’s current normal behavior.
10. 1 Continue to use monitoring to screen the program’s execution. It will detect any inefficiencies that are introduced and automatically measure the program to capture performance problems when they occur.

Screening can be set up of the execution of batch programs that run on a nightly schedule, weekly schedule or monthly schedule. Candidate monitoring can also detect unacceptable processing in programs that do not run on a normal schedule or follow resource consumption patterns. For these programs you may want to enter thresholds rather than have them calculated for you since the collected basis data may not reflect normal or acceptable processing.

The rest of this section discusses:

• Accessing candidate processing and monitoring
• Configuring and running the SMF Candidate PreLoader utility
• Adding a monitoring request
• Scheduling a monitoring request.

Accessing Candidate Processing and Monitoring

Type A from the Strobe Options menu and press Enter to display the Strobe- AutoStrobe Options panel.

Strobe Options Menu

From the Strobe- AutoStrobe Options panel shown in, you can:

• Add a monitoring request (Option 1)
• Track and manage the request and its associated basis data (Option 2)
• Retrieve and manage archived data, which is data for a request that has been deleted (Option 3)
• Manage monitoring candidate lists (Option 4)
• Run the SMF Candidate PreLoader Utility (Option C)

Strobe- AutoStrobe Options Panel

SMF Candidate PreLoader Utility

This section explains how to use the System Management Facility (SMF) Candidate PreLoader Utility. To identify job steps that have high resource consumption, you can run this utility to scan the SMF data sets. The PreLoader uses a set of thresholds that must be exceeded to qualify a step as a monitoring candidate. After the utility has examined the specified SMF data sets, it builds a list of the job steps that exceed the thresholds and that might require monitoring. Managing-AutoStrobe-Batch-Candidate-Processing-and-Monitoring explains how to manage monitoring candidates.

To run the utility, select the C CANDIDATE CREATION choice from the Strobe- AutoStrobe Options panel. The Strobe - AutoStrobe SMF Candidate Utility panel is displayed. 

SMF Candidate PreLoader Configuration

The SMF Candidate PreLoader Utility adds data for a job step from the SMF data sets to the monitor candidate list only if that step’s usage of the monitored resources exceeded a threshold. You can configure the utility through the Strobe- AutoStrobe SMF Candidate Utility panel shown in the Strobe - AutoStrobe SMF Candidate Utility panel. To configure the utility:

1. We recommend that you exclude system applications such as IEBGENER that should not be considered as candidates. To exclude job steps, type Y in the MODIFY EXCLUSION LIST field and press Enter. The Strobe - AutoStrobe Change Exclusion pop-up window is displayed. Creating a exclusion list is explained in the section Candidate Exclusion List.
2. Change the values in the ELAPSED TIME, TCB TIME, EXCPS and FREQUENCY OF RUN fields to ones you want to use. (Full explanations of the fields and their defaults and minimum values are shown after the following figure.
3. Specify the names of the SMF data sets that should be used as input to the utility.
4. Press Enter and the Strobe- Select Options for Job Stream panel shown in Strobe - Select Options for Job Stream Panel (SMF JCL Popup Panel).

Strobe- AutoStrobe SMF Candidate Utility Panel

SWITCH TO AUTOSTROBE TRANSACTION OPTION

If you type Y, AutoStrobe displays the Strobe- AutoStrobe Create Transaction Candidates panel which enables you to create a list of CICS transactions, called candidates, that exceed CPU and response time thresholds that you specify. For more information, see Using-AutoStrobe-to-Identify-CICS-Transactions-for-Monitoring . The default value is N, which does not switch the panel.

ELAPSED TIME

Minimum number of minutes a step must run to be included in the SMF candidate processing list. (Default is 60 and minimum value allowed is 10)

TCB TIME

Minimum aggregate task time in minutes for all Task Control Blocks (TCBs) in the step for it to be included in the SMF candidate processing list. (Default is 20 and minimum value allowed is 2 )

EXCPS

Minimum number of EXCP requests (direct invocations of execute channel programs, representing I/O) that must occur for a step to be included in the SMF candidate processing list. (Default is 1000000 and lowest value allowed is 20000)

FREQUENCY OF RUN

Average minimum number of times a step must run a week to be included in the SMF candidate processing list. (Default is 30 and lowest value allowed is 7)

SMF DATA SETS

The names of the SMF summary data sets to use as input to the SMF Candidate PreLoader Utility. Read the following note before you specify the SMF data set name.

Running the SMF Candidate PreLoader Utility

The SMF Candidate PreLoader Utility provides JCL that builds the candidate list. You can edit this JCL to run according to the conditions that are required by your system environment.

Strobe- Select Options for Job Stream Panel (SMF JCL Popup Panel)

Select one of the following options:

The SMF Candidate PreLoader utility will then populate the candidate list. You can then select the ones you want to start monitoring. 

Candidate Exclusion List

The SMF Candidate PreLoader utility has to scan large amounts of data in the SMF data sets you want to use. To reduce this data, you should first exclude jobs (such as IEBGENER). If you want to exclude job steps, type Y in the MODIFY EXCLUSION LIST field shown in Strobe - AutoStrobe SMF Candidate Utility Panel and press Enter. The Strobe- AutoStrobe Candidate Exclusions panel is displayed.

Strobe- AutoStrobe Candidate Exclusions Panel

Specify the job step, program and system name of any application that you want to exclude from processing by the SMF Candidate PreLoader utility. You can use a (*) wildcard figure for any of the fields, but not all of them. Pressing Enter adds the job step to the exclusion list. Typing D in the column before the JOBNAME column deletes the entry from the list. Typing C in the same column enables you to change the entry in the exclusion list. If you press Enter after specifying C, the Change Exclusion pop-up window is displayed.

Strobe- AutoStrobe Change Exclusion Pop-Up Window

When the panel first appears, the OLD column and the NEW column show identical values. In the NEW column, you should edit the values for the entry in the exclusion list. In this example, the JOBNAME column value of JOB11 is changed to JOB11* and the SYSTEM column value is changed to * (asterisk) from SYSTEM. Pressing Enter returns you to the Strobe - AutoStrobe Candidate Exclusions panel and now any job step name that begins with the characters JOB11 running on any system is not eligible to be included on the AutoStrobe Candidate list.

Candidate Processing AutoLoader

The contents of the monitoring candidate list built by running the SMF PreLoader utility will only include job steps that exceeded performance thresholds as indicated by SMF data. If a job step does not start performing poorly until after the PreLoader has run, it will not be on the list. To make sure that all job steps that might require monitoring are placed on the candidate list after the SMF PreLoader has run, the AutoLoader continuously checks the resource consumption of batch job steps as they end. If any exceed the threshold values for elapsed time, TCB time or EXCPs, the AutoLoader places the job step on the monitoring candidate list. (The AutoLoader runs only if the AUTOCANDIDATE parameter value in the Strobe installation PARMLIB is YES, the default value.) To change the threshold values used by the AutoLoader, see the Tailoring Generated Candidate Lists.

Adding a Monitoring Request

You add an AutoStrobe monitoring request by selecting Option 1 from the Strobe - AutoStrobe Options menu. (For information about adding AutoStrobe requests from one of the AutoStrobe Candidate Management panels, see Managing-AutoStrobe-Batch-Candidate-Processing-and-Monitoring.)

Select Option 1 (ADD AUTOSTROBE REQUEST) from the Strobe - AutoStrobe Options menu. Press Enter. The Strobe - Add AutoStrobe Request panel is displayed.

Strobe- Add AutoStrobe Request Panel

Designating the Program to be Monitored

This section explains how to designate what program you want to monitor. The specifications you make on the Strobe- Add Monitoring Request panel can include the job name, the job step name, the program name and the system name. Note the following considerations for these fields:

SWITCH TO AUTOSTROBE TRANSACTION OPTION

If you type Y, AutoStrobe displays the Strobe- Add AutoStrobe Transaction Request panel, which enables you to specify CICS transactions for AutoStrobe monitoring. See Using-AutoStrobe-to-Identify-CICS-Transactions-for-Monitoring for more information. The default value is N, which does not change the panel.

Jobname

Designate the job name. This is a required field.

You can specify a wildcard for the job name, but if you do so, you must specify the step name and program name. Monitoring of the program occurs when it runs under any job that corresponds to the wildcard designation. The basis data collected for monitorings of the program under different jobnames will be combined.

Example of a Jobname with a Wildcard

AutoStrobe screens these jobs that were submitted by a job submission package daily with unique jobnames:

Step Name

Designate a step name. The step name is optional except when the request has a wild card job name.

We recommend that you assign step names in the JCL for programs that might be monitored and specify these step names on the monitoring request.

You can complete the step name field in one of three ways:

• Specify a step name. Program monitoring only occurs when it runs in the designated step.
• Leave the STEP NAME field blank. AutoStrobe will monitor the first step in which the target program executes. AutoStrobe will monitor subsequent executions of the identified job step.
• Screening will take place of the first step that executes of the designated job and program whether or not the step that executes is named. Then a request update occurs so that monitoring only occurs for that step in subsequent executions of the program.
• Type two single quotes (‘’) in the step field. Screening will only occur for those instances of the specified program that runs in unnamed steps.

Example of Step Name Designations

The following MVS batch job executes the same job-step program with named and unnamed steps:

Program

Designate a specific program name. This is a required field.

System

Type a system name in the SYSTEM field. This field appears only if Strobe is being run in a multisystem environment. Specify that the program be watched on a specific system or on all MVS systems in a STROBE XCF group. Specify *ALL to monitor on all on MVS images.

Overriding Generated Thresholds

This section explains how to change or suppress monitored condition thresholds (Elapsed time, TCB time and I/O activity). The first ten monitorings of a given job step are used to collect the basis data from which it calculates the thresholds for abnormal behavior for the three monitored conditions.

When you submit a monitoring request, you can specify your own threshold for that job step’s usage of elapsed time or TCB time (or both). These thresholds are then enforced on all subsequent monitorings of the step. Even with a user-specified threshold, the collection of basis data from the first ten monitorings of the step still occurs except for any that exceed the user threshold. After ten monitorings in which no user threshold was reached, thresholds are calculated for all of the conditions.

Even on requests having user-specified thresholds, the calculated thresholds for elapsed and TCB time might be useful for comparison with ones specified by you. Also, if you subsequently changed this monitoring request to delete a user-specified threshold, the corresponding calculated threshold would then be immediately applied to subsequent monitorings.

Suppressing Action for Abnormal Behavior

You can also suppress the use of a threshold value as the basis for starting a measurement request for a job step. You can suppress the action for one or two but not all three of the monitored conditions. For example, if you only want the elapsed time that a job step has to run to cause the initiation of measurement of it, you can type Y in the “Suppress” column for “TCB TIME” and “I/O Activity”. Suppressing all three actions causes the monitoring request to be rejected.

Other Options for Monitoring Requests

After you have specified which programs you want to monitor and determined what threshold values you want to use, you can set three other options for the monitoring request.

Defining a Monitoring Schedule

Defining a schedule for an ADD request causes program screening to occur only on specific dates. If you have a program that runs every day but always has an unusual work load on specific days, a monitoring request to screen that program could specify a schedule that suppresses screening on those days.

To define a schedule for a monitoring request:

• Type the required information on the Strobe - Add AutoStrobe Request panel, specifying Y in the SCHEDULE field. Press Enter.
• Strobe displays the Strobe- AutoStrobe Request Scheduler panel.
• Specify Y below the days of the week on which you want to initiate observation.
• Specify a date on which you want the schedule to begin.
• Specify the number of weeks that you want the monitoring request to run or the date that you want the schedule to end. If you type the number of weeks, the end date is automatically calculated for you.

Strobe- AutoStrobe Request Scheduler Panel

You can customize a schedule by typing a list of specific dates on the bottom half of the Strobe- AutoStrobe Request Scheduler panel. For example, you can create a schedule that would initiate observations on a specific date during each fiscal quarter. To create a list of dates, specify the dates in the lower portion of the panel, and type A in the OPT field to the left of the date.

You can also supplement the schedule for a monitoring request by adding or excluding certain dates on your weekly schedule. For example, you could create a schedule that would automatically initiate monitoring every Monday for the next 22 weeks. However, there are two Monday holidays that you would like to exclude from the schedule, and instead initiate a measurement session on Tuesday. Specifying the Monday dates with an E in the OPT field to the left of the date excludes those dates from the schedule. Specifying the Tuesday dates with an A in the OPT field to the left of the date adds those dates to the schedule.

To customize your schedule, type the dates on the lines at the bottom of the Strobe- AutoStrobe Request Scheduler panel, and specify the add or exclude option in the OPT field. Monitoring will then automatically initiate (or omit, in the case of the exclude option) observations on those dates.

Specifying the ASSOCIATED ACTION

The ASSOCIATED ACTION field value determines what action to take when a monitored condition threshold is exceeded:

• Specifying A for an Add Active request causes a measurement to initiate when the abnormal behavior threshold is reached. Press Enter and the Strobe- AutoStrobe Request - Active Measurement panel is displayed. This measurement, however, could occur too late in the program to obtain significant data. If the duration of the measurement is less than two minutes or the value of the AUTODBMIN parameter, and the threshold exceeded was elapsed time, the data collected is not considered representative, the measurement is discarded, and an Add Queued request is scheduled to measure the next execution of the program. Specifying an active measurement causes measurement of a job step to start immediately if abnormal activity is detected as it is being monitored.
• Specifying Q for an Add Queued request causes initiation of a measurement of the entire step the next time it runs, but it may not be exhibiting abnormal behavior at the time of the measurement. Press Enter and the Strobe- AutoStrobe Request - Queued Measurement panel displays. Specifying a queued measurement causes measurement of a job step showing abnormal behavior during monitoring the next time the job step runs.
• Specifying B to request both an active and a queued measurement causes both measurement of a job step when abnormal activity is reached and the next time the job step runs regardless of whether abnormal behavior is detected. Press Enter and the Strobe- AutoStrobe Request Multiple Measurements panel displays. If you specify the multiple measurement, measurement of a job step begins immediately if abnormal behavior occurs during monitoring and also the next time the job step runs.
• Specifying W causes a warning that abnormal behavior was observed if the threshold values are exceeded for this particular monitoring request only.

Strobe- AutoStrobe Request - Active Measurement Panel

Strobe- AutoStrobe Request - Queued Measurement Panel

Strobe- AutoStrobe Request Multiple Measurements Panel

User ID to Notify

You can use this field to specify a user ID that should be notified when a monitoring request is made.

• The TSO USERID TO NOTIFY field is initialized to the user’s TSO ID; however, you may change it.
• Clear the field to have no notifications sent on behalf of the monitoring request.