HCI diagnosis and debugging

If problems occur with the HCI, we recommend that you provide the following information to BMC Support that aids in diagnosing HCI problems:

• Storage dumps
• HCI journal
• Stub tracing
• Generalized Trace Facility (GTF) trace.

This topic describes the information types.

Storage Dumps

The HCI invokes the z/OS dumping facilities whenever it detects a problem that cannot automatically be corrected. HCI can request two major types of dumps: SNAP dumps and SDUMPs.
SNAP dumps can be directed to a disk data set or to SYSOUT, based on the DMPCLAS, DMPPFX, DMPUNIT, and DMPVOL attributes on the HCICNGCA element. Specifying DMPCLAS makes specifying the other DMPxxx attributes unnecessary. Specifying DMPPFX, DMPUNIT, and DMPVOL makes specifying DMPCLAS unnecessary. SNAP dumps are relatively slow to generate; consequently, HCI execution may be suspended for an unacceptably long length of time.
SDUMPs are always directed to the installation-defined SYS1.DUMPxx (or equivalent) data sets. These dumps can be managed by the installation more easily than SNAP dumps, and they can be copied easily to disk or tape. These dumps are generated in a very short length of time, thus suspending HCI execution for a minimal duration. Request an SDUMP by coding the DMPPFX=SDUMP attribute (and omitting DMPCLAS and DMPVOL attributes) on the HCICNGCA element. We strongly recommend that you code DMPPFX=SDUMP at your site.

Ensure that any dumps sent to BMC contain the following information:

• Abending PSW
• Abending register contents
• Storage: region, private, common, LPA, SQA, and LSQA
• Region system control blocks, such as ASCB, TCBs, IRBs, among others.
• Global resource enqueues
• System trace tables
• Dump summary.

If you are generating an SDUMP, specify the following options:

and

If the HCI is running in a sysplex and one or more of the TP applications are running on a different z/OS image than the HCI, then also specify the following option:

HCI Journal

The HCI journal provides the most comprehensive debugging aid available. Although not required for most HCI execution, when a problem occurs, the journal should be created that contains the execution error. The journal facility is always available in the HCI, but based upon the setting of the journal mask attribute in the HCICNGCA element and the availability of journal VSAM data sets, the journal may not be active at any given time.
If you need to provide BMC with the journal data for analysis, prepare it for SFT transmission.

Preparing Journal Data for SFT Transmission

Reformat the contents of a journal data set into a data set that is appropriate for SFT transmission to BMC's SFT site. The following shows an example of the JCL to perform this reformat.

JCL to Reformat Journal Data

Once the data has been successfully unloaded, the unloaded data set can be transmitted via SFT to BMC's SFT site. See  Using the TCP/IP SFT Program for more information.

Printing the Journal Data

Do not send a print image of the journal data be sent to BMC because doing so will delay problem analysis. However, if you do need to print the journal data, use the JCL shown in the following code block to minimally format the journal data.

JCL to Print Journal Data

GTF Trace

Instances for which a Generalized Trace Facility (GTF) trace may be required include:

• VTAM trace
• TCP/IP trace
• Sysplex trace.

All aspects of preparing, starting and stopping the GTF, as well as sending the GTF data remain the same. The only difference is the GTF parameters.

In an active sysplex environment, you cannot use the HCI journal facility to record the execution of the z/OS systems other than the local z/OS that contains the HCI itself.
Because some sort of journal is required, use the z/OS GTF to record sysplex processing.

Preparing GTF to Gather HCI Data

Do the following to prepare to use the GTF to gather HCI data:

1. Allocate a data set to contain the GTF trace output.
2. Create a GTF procedure to invoke GTF with the data set.
3. Create a SYS1.PARMLIB member to contain the GTF execution time parameters.

The following code block shows an example of a job that allocates a data set to receive the GTF trace data.

JCL to Allocate a GTF Trace data set

Specify a data set name and volume serial number appropriate for your site's installation. Code the DCB parameters exactly as shown.

The following code block shows an example of a procedure to invoke GTF.

Procedure to Invoke GTF

Starting GTF

Start the GTF procedure by entering a command on the system console or from an SDSF panel. The following is an example of this command.

The GTF program displays startup information, including a listing of the parameters that are to be used. If its initialization is successful, GTF requests that the operator confirm the startup parameters by replying to an outstanding WTOR. This message is shown below:

Where nn is the reply number. Using the reply number nn, the operator enters the following on the system console or on an SDSF panel:

Rnn,U

GTF displays more diagnostic information, ending with the following message:

Stopping GTF

Stop GTF by entering the following command on the system console or from an SDSF panel:

GTF responds with the following message:

Sending GTF Data for Analysis

You can reformat the contents of the GTF data set into a data set that is appropriate for Secure File Transfer (SFT) transmission to BMC's Secure File Transfer (SFT) site or placed onto magnetic tape to be mailed. Refer to one of the two applicable sections.

Preparing GTF Data for SFT Transmission

In order to send the GTF data via SFT, reformat the data. The following code block shows an example of the JCL to perform this reformat.

JCL to Reformat GTF Data

Once the GTF data has been successfully unloaded, you can transmit the unloaded data set via SFT to BMC's SFT site.

Using the TCP/IP SFT Program

The programs HCIJUNLD and HCIGUNLD are designed to read the variable length input data and create a fixed length output data set that can be transmitted by File Transfer Protocol (SFT). The format of the records created by HCIJUNLD and HCIGUNLD is known by the corresponding programs that re-create the original data sets.

Invoking SFT

If TCP/IP is available on a z/OS host, SFT can be invoked from a TSO session, with or without ISPF. If TCP/IP is not available, the output of HCIJUNLD or HCIGUNLD must be transmitted to a workstation that does have TCP/IP on it before the data set can be sent to BMC.

The remainder of this section assumes SFT is available on the host.

Assuming a host-based TCP/IP and no fire wall, enter the commands shown in the following example on any ISPF screen. Commands are shown with a right arrow (===>) prefix.

SFT Session with BMC AMI

Storage Estimates

The storage utilized by the HCI is very use-oriented. The type of storage that is being used can be categorized into four types each above and below the line:

• Private
• Common
• Module
• Buffer

Private and common storage can each be more than 1 MB above the line. The storage used below the line for private and common is minimal at around 20 KB. Module storage is under 700 KB, the majority of it above the line (less than 50 KB below the line). Buffer storage can be less than 100 KB above the line. Above the line storage is always used unless unavailable or not permitted to be used by some of the very few z/OS facilities. No storage is allocated above the 64-bit bar.

These amounts are all rough estimates and the actual storage size is directly related to the functions the HCI is performing. The actual storage utilization can be seen using the HCILOOK facility or displayed on the operator console using the DISPLAY STOR HCI command.
To help explain where some of the storage is used, the following table shows those control blocks whose quantity is user-controlled:

User-Controlled Control Blocks Storage

The WREs are usually the largest storage item. Currently the HCI is intolerant of running out of most internal control blocks. Always provide a few extra of each where possible and monitor storage utilization via the TSO/ISPF HCILOOK facility.