IAM's Real Time Tuning

IAM buffering during a file load does not utilize the Real Time Tuning concept, because the file load I/O process is a sequential output process. The buffering technique used by IAM for a file load is described later in this section.

Benefits of Real Time Tuning

The benefit of Real Time Tuning is the ability to easily achieve a level of performance that is difficult or impossible to reach with VSAM, along with a significant reduction in the manual effort and data gathering required for tuning your indexed files to meet the processing requirements. IAM's Real Time Tuning eliminates the need for deciding which buffering technique to use. With VSAM, you have to choose between NSR and LSR buffering or leave it to DFSMS System Managed Buffering or other packages to make the decision between NSR and LSR. IAM has eliminated the need for index buffers, because IAMautomatically keeps the index in virtual storage. For most data sets and applications, the default buffering provides a very high level of performance. The IAMINFO run time report typically provides more than sufficient information to make any manual buffering adjustments that may be desired. This report, which is produced whenever an IAM data set is closed, provided that, there is an IAMINFO DD card allocated to the job step, will even indicate with the IAM368 message if more buffers would have improved performance. The IAMINFO report can also be obtained from the IAM SMF records, if they are being collected, with program IAMSMF. In fact, that program can be used to print just the IAMINFO reports for those data sets for which IAM indicated that more buffers would have improved performance. This can be done by using the IAMINFO command with the ATTRIBUTE=MOREBUFFER keyword.

VSAM System Managed Buffering decides, whether to use NSR or LSR buffering, and will use more buffers than the normal VSAM default. Even with System Managed Buffering, you may still have to indicate through JCL, what buffering technique will work best with the way the particular application uses the files. While jobs using VSAM files with System Managed Buffering may perform better than without, System Managed Buffering does not offer the dynamic buffer management that IAMReal Time Tuning offers. In many circumstances, IAM’s Real Time Tuning still does noticeably perform better than VSAM with System Managed Buffering.

Real Time Tuning for Enhanced Format Files

Real Time Tuning for IAM files includes the following capabilities:

1. As shipped, IAM can have up to 24,576 buffers per file. This capability can be easily increased if necessary.
2. The capability to handle concurrent I/O requests, both at the logical level and at the physical I/O level. This improves IAM responsiveness for online systems, for multi-volume data sets, and for volumes with Parallel Access Volume (PAV) capabilities. IAM’s channel programs are set up to utilize PAV capabilities when they are available for improved I/O responsiveness, which eliminates the need to spread data sets across multiple volumes for performance.
3. Incorporate elapsed time into buffer adjustment algorithms. This will prevent the slow increase in buffers to lightly used files in an online system that is opened for several hours.
4. Provide sequential I/O read ahead, with I/O and processing overlap capabilities for synchronous mode (batch jobs) I/O processing. Up to a full cylinder's worth of data can be read in one physical I/O.
5. For mass sequential updates, IAM will write out multiple updated blocks per physical I/O. This includes the ability to write out multiple blocks per I/O even if they are not contiguous.
6. IAM buffers can reside in 64-bit addressable virtual storage or in 31-bit addressable storage based on Global Options and overrides. Should a buffer end up in 24-bit addressable virtual storage, IAM will release that buffer, to help avoid a potentially critical virtual storage shortage, unless that is the only buffer that IAM has available.
7. For Share Option 1 or 2 files, IAM also has incorporated empty block detection, which will prevent subsequent physical I/O for such empty blocks. This eliminates a problem encountered with rereading empty blocks when there had been large quantities of records deleted from the file.
8. TheIAM I/O driver will also take advantage of the IBM Record Level Caching, for random I/O’s and indicate sequential processing on the devices that provide those capabilities.
9. IAMchannel programs will automatically utilize z/HPF High Performance Ficon channels using the Transfer Command on the volumes that support the architecture. IAM continues to provide optimal use of the latest hardware in the market, by using block sizes that fully utilize the device’s capability, and by using the latest features and capabilities when performing I/O.
10. 1 A TURBO mode, which defaults to being enabled, is provided to allow for faster acquisition of buffers during periods of heavy I/O activity.

Buffering for Random I/O

For random processing, IAM dynamically acquires additional buffers up to the MAXBUFNO value, so that the most frequently referenced blocks can be retained in memory. IAM continually monitors what the added benefit would be for additional buffers at any point in time by tracking contents and frequency of use. By monitoring requests in this way IAM can recognize patterns. After certain predetermined numbers of physical I/O's, IAM will evaluate the buffering. When IAM finds that a sufficient number of requests could have been satisfied without I/O to the disk within the past few minutes if more buffers had been available, an additional buffer is acquired. If the file is in Turbo Acceleration Mode up to four buffers may be acquired at one time. This continues until the maximum number of buffers allowed is reached.

IAM's buffer management techniques also provide the release of buffers, when it determines they are not providing any benefit in terms of reducing I/O within the past several minutes.

Buffering for Sequential I/O

Buffer management for sequential processing differs from random in that IAM attempts to determine how many blocks it should be reading ahead in anticipation, to efficiently service the user. The goal is to be able to read as many useful blocks as possible with each real disk I/O. To accomplish this, IAM monitors the number of blocks that are read consecutively, increasing or decreasing the number of consecutively chained buffers as appropriate. A maximum of up to one cylinder worth of blocks will be read per I/O. More buffers are acquired as necessary, at the rate of one per physical I/O, up to the MAXBUFNO value. Once processing has been completed for each block read sequentially, rather than aging the block out of the buffer pool, it is immediately treated as being the buffer with the least recent reference, because it is not expected that the block will be needed in the near future. For synchronous sequential I/O, which is typically done by batch jobs, when reading multiple blocks per physical I/O, IAM will make the data from each block available to the application program as soon as it is in storage. On mass sequential updates, when multiple consecutive data blocks have been updated, IAM will write out multiple blocks per physical I/O. This process is done asynchronously whenever possible, to allow I/O and processing overlap.

The advantage of reading or writing multiple blocks per I/O is improved performance due to a reduction in resource utilization. The primary savings are at the device level. This type of I/O will eliminate DASD seek and search time, as well as rotational positioning time, because the device is already positioned to process the next block. This device time can frequently be well over 50% of the total device time needed when performing a single block I/O. Also, eliminated is the CPU overhead involved with issuing an EXCP, scheduling the I/O, and the CPU time involved with the I/O completion. Additionally, for batch jobs, IAM will attempt, whenever possible, to allow the sequential I/O to be active while allowing continued processing of the data that is already residing in IAM's data buffers.

Mix of Random and Sequential

Multiple random and / or sequential I/O operations can occur concurrently against any file. The use of the buffers will be balanced between the different requests based on the buffering needs of each request. The request with the highest buffer requirements will tend to utilize the most buffers. IAM does not lockout a request from a data block if another request is also using the block, such as VSAM does with the CI level lockout. A request may be temporarily delayed to access a block of data, only if there is an actual physical I/O occurring, or during the actual processing time to move a record into or out of the buffer. This delay is managed internally by IAM , and does not result in deadly lockouts. The CI lockout problem caused by VSAM resulted in many installations not being able to use VSAM LSR buffering for some files under CICS, or forced files to small CI sizes to minimize the occurrence of the lockout problem. IAM eliminates these VSAM problems. IAM does perform the necessary record level lockout for concurrent updates to the same record to maintain data integrity.

Synchronous vs. Asynchronous I/O

For IAM data sets defined with Share Option 1 or 2 that are opened for update, IAM does distinguish the differences between synchronous and asynchronous processing. Synchronous processing is indicated by the RPL OPTCD=SYN, and is typically used by batch jobs. Asynchronous processing is indicated by the RPL OPTCD=ASY or OPTCD=WAITX, which is typically used by online systems, such as CICS. To improve batch performance IAM defers the rewrite to disk of a randomly updated buffer until the block residing in the buffer is forced out of the buffer pool. This technique reduces I/O whenever multiple rewrites are requested against data blocks in the buffer pool. Under Online systems (example CICS), to ensure integrity after a random update, buffers are always immediately rewritten. For sequential processing, the updates are deferred regardless of processing mode until IAM has several blocks to chain write, until the caller issues an ENDREQ request, or until the buffer is needed for another data block.

With sequential synchronous I/O, IAM is very aggressive with increasing the number of blocks that are being read per I/O, as well as being very quick to increase the number of buffers up to MAXBUNO. This is done to improve elapsed time for batch jobs. When asynchronous type of I/O is being done, IAM is less aggressive with sequential chaining. This is done to prevent tying up too many buffers at one time for online systems. This will prevent issuing an I/O request, which will monopolize a device for a relatively long period by reading many blocks per I/O, which could impact the responsiveness of the online system for other transactions.

Setting MINBUFNO and MAXBUFNO

IAM's Real Time Tuning works within an optionally specified range for determining the number of buffers called MINBUFNO and MAXBUFNO. MINBUFNO defaults to 1, unless explicitly overridden on an IAM Override statement. The value for MAXBUFNO can come from a number of sources, as indicated below in precedence order:

1. From the IAM ACCESS override of BUFSP. If provided, this value always has precedence over any other specification.
2. From the IAM ACCESS override value specified for MAXBUFNO.
3. From the IAM CREATE override value specified for MAXBUFNO when the file was loaded or defined.
4. The buffer space value specified on the IDCAMS DEFINE of the data set, providing that it is greater than the IAMGlobal Options BUFSP value.

5. The higher of BUFND, BUFSP, or STRNO specified in either the JCL AMP parameter or explicitly in the ACB, provided that it is larger than the default MAXBUFNO determined from theIAM Global Options Table.

   Note

   JCL specification of one of these values takes precedence over the value specified in the ACB control block itself.

6. A default value from the IAM Global Options Table, which will be the higher of either the MAXBUFNO Global Option, or the BUFSP Global Option (CICS will use CICSBUFSP) value divided by the file's block size.

For a typical 1/4 track blocked IAM data set on a 3390 device, using the default Global Option values, of 65,536K for batch, and 1,024K for CICS IAM will of set the MAXBUFNO value to 4904 for batch, and 76 for CICS.

When an IAM data set is opened, IAM will determine the values that it will use for MINBUFNO and MAXBUFNO as described above. If MINBUFNO value exceeds MAXBUFNO, it will be reduced to the MAXBUFNO value being used. IAM will then decide on a starting value. For batch jobs, IAM will initially acquire the higher of number of buffers required for 16 tracks or MINBUFNO if it was specified as an override, but will not exceed MAXBUFNO. For CICS IAM will initially acquire the smaller of the number of buffers to read the index or needed for 15 tracks but will not exceed MAXBUFNO. Under CICS, the number of buffers will be reduced at completion of open processing to the number of strings (STRNO) or if overridden the MINBUFNO value.

As a general rule, there should be very little necessity to specify a MINBUFNO value. With the TURBO mode buffering enabled, buffer acquisition is very fast for up to 80% of the MAXBUFNO value. MAXBUFNO can be changed as desired, and generally should be increased when IAMINFO indicates that more buffers will be helpful, and you need to improve the performance for the data set.

Statistics from IAMINFO

The IAMINFO report, which is produced when an IAM file is closed, will provide a variety of statistics relating to IAM Real Time Tuning and its effectiveness. These fields include the following:

DISK BLOCKS READ - Indicates the number of physical I/Os (EXCPs) that were done to read data into the IAM buffer pool.

DISK BLOCKS WRITTEN - Indicates the number of physical I/Os (EXCPs) that were done to write data out to the file.

The TOTAL EXCP's issued by IAM for this data set can be calculated by adding the DISK BLOCKS READ plus the DISK BLOCKS WRITTEN. That number will be equal to the EXCP value reported by SMF for this data set, except for a file load with a temporary work data set. For a file load with a temporary work data set, the IAMINFO values are inflated by the I/O's done to the temporary data set.

DYNAMIC BUFFER RETRIEVALS - Displays the number of I/O requests satisfied from the IAM buffers without performing a physical I/O. This statistic essentially identifies the savings of I/O due to IAM's Real Time Tuning.

MAXIMUM BUFFERS USED - Indicates the maximum number of buffers that were used.

MINIMUM BUFFERS USED - Indicates the smallest number of buffers that were used.

MAXIMUM BUFFERS AVAILABLE - Indicates the MAXBUFNO value in effect for this file.

SEQ CHAINED BLOCKS READ - Indicates the number of data blocks that were read in as part of a sequential multiple blocks per I/O. These blocks were read as part of IAM's Real Time Tuning anticipation that the application program will use the blocks. This value plus the DISK BLOCKS READ value indicates the total number of blocks physically read into storage for this data set.

SEQ CHAINED BLOCKS WRITTEN - Indicates the number of data blocks that were written as part of a sequential multiple block output I/O. This indicates the savings of physical I/O (EXCPs) due to IAM's Real Time Tuning accumulation of updated data blocks. This value plus the DISK BLOCKS WRITTEN value equals the total number of blocks written out to the data set.

z/HPF I/O REQUESTS - Indicates the number of EXCP’s issued using the High Performance Ficon architecture I/O. This value is provided for users to verify IAM’s utilization of the High Performance Ficon.

ECKD I/O REQUESTS - Indicates the number of EXCP’s issued using standard ECKD or CKD I/O architecture. This value may be non-zero even when z/HPF is being utilized for I/O issued for writing the DASD EOF, new extent processing, recovery from an I/O error, or when z/HPF has been temporarily turned off for a volume in use.

Do I Need More Buffers?

While all of the above information is useful in evaluating the benefits of IAM's Real Time Tuning, there may still be a question as to whether or not more buffers would have been useful. The IAMINFO report will come out and tell you via an IAM attention message, message id IAM368 that is displayed in the run time INFO report. A flag is also stored in the IAMSMF record.

The IAM368 message is just an indication that the I/O performance for the specified file could potentially be improved by providing a larger MAXBUFNO value. It may very well be that the performance level of the job processing this data set is quite acceptable, and if so there is no need to increase buffering. However, if you are looking for ways to improve the performance, then this message is an indicator of one of the possibilities for improving performance.

If the IAM368 message does not appear, then providing more buffers is very unlikely to change the performance. However, for some applications that are very randomly processing the file, it may be beneficial to provide larger values for both MINBUFNO and MAXBUFNO, in case the reference pattern encompasses more blocks than what IAM’s Real Time Tuning has been tracking so it hasn’t been increasing the number of buffers.

For more information on tuning and getting the most out of IAM, see to the IAM-Tuning-Guidelines in the IAM User's Guide portion of the space.

File Load Buffering

As indicated at the top of the Real Time Tuning section, a file load does not make use of the Real Time Tuning concept. This is because a file load is exclusively a sequential output process. A fixed number of buffers are acquired at open time, based on the value for the CRBUFOPT Global Option or IAM Override. The required channel programs and I/O control blocks are built and initialized when open, based on the quantity of buffers acquired. By default, for files that are allocated on cylinder boundaries, IAM will use enough buffer space to hold two cylinders, and write out a full cylinder per physical I/O. The default for track allocated files is buffer space for two tracks, and writing out one track per I/O. This buffering concept provides for I/O overlap, even for programs using synchronous processing. The EXCP counts will reflect the actual EXCP (physical I/O) operations done not the block count.

I/O processing during file load is different for DFSMS Extended format data sets. While the number of buffers will be the same as for EXCP processing described above, IAM is using BSAM access method to write out the data to the extended format data sets. IAM will allow BSAM to write out multiple blocks per I/O. However, with BSAM output, it is not possible to control how many blocks will be written out per physical I/O. Also, because BSAM is being used, the EXCP count for file loads will be the actual number of blocks written out to the data set, and not the number of physical I/O requests. So, even though multiple blocks are being written per I/O, saving is not reflected in the EXCP counts.

z/HPF

High Performance Ficon (z/HPF) is an IBM architecture for physical I/O that enables more efficient utilization of I/O channels resulting in improved performance and greater capacity. This function has recently become available for IAM utilization with hardware and software enhancements that enable utilization by software that uses EXCP level I/O. Based on the results at BMC, we see a 20% to 25% reduction in channel connect time when IAM uses z/HPF, and a small improvement in elapsed time.

The z/HPF support requires IAM Version 9.2 along with z/OS 2.1, or z/OS 1.12 and above with enabling PTF. On the hardware side, it requires DASD subsystem support of the functions provided by z/HPF for EXCP. IAM validates eligible devices through the use of the IOSZHPF service. Users should contact their DASD subsystem vendor to determine what is necessary for EXCP I/O support.

64-Bit Virtual Storage Buffers

IAM Version 9.2 can optionally utilize buffers in 64-bit virtual addressable storage. This is primarily of benefit for CICS, IAM/RLS, and IAM/PLEX address spaces where there are 31-bit addressable storage constraints. This may cause an increase in CPU time due to fix processing for 64-bit virtual storage page.