IAM Storage Usage

Storage Overview

IAM does utilize virtual storage to provide the services requested by application programs. IAM uses virtual storage to reduce I/O and the CPU time required to process an indexed data set. One of the features of IAM is that, it keeps the index to the file in virtual storage. There are no index buffers, or index I/O after the file has been opened. Data buffers are the other major component of virtual storage usage. These large storage areas are always requested from the 31-bit addressable area of memory (above the line) or optionally from 64-bit addressable storage. This use of storage, seldom is a problem for batch jobs. However, large online regions that have hundreds of files open at any point in time may run into virtual storage constraints. This section will attempt to explain IAM’s storage usage. Users are encouraged to contact BMC for IAM technical support for assistance with any storage usage issues.

Virtual Storage Management Features

To help minimize the need for virtual and real storage tuning, IAM has several special features to aid in the dynamic management of virtual storage for jobs using Enhanced Format IAM data sets. IAM can put the index for an open IAM data set into either 64-bit virtual storage or a z/OS data space, which is referred to as an Index Space. This feature alleviates virtual storage contention by moving a lot of IAM storage into storage areas without the constraints of normal address space storage. IAM can also dynamically increase the above the line storage region based on values from the IAM Global Options Table. This provides a quick way to dynamically adjust to unexpected storage requirements, without having to change the IEFUSI exit or the job’s REGION parameter. When acquiring a non-critical area of storage, such as acquiring an additional buffer, if the storage was acquired below the 16-megabyte line, IAM will release that storage. This way, once the above the line region is filled, IAM will not unnecessarily use below the line storage, which could quickly disappear. IAM also monitors buffer usage, as a part of the Real Time Tuning, and will release infrequently referenced buffers as long as there is some I/O activity on a data set.

Version 9.2 introduced the ability to use 64-bit addressable virtual storage for the I/O buffers. This can be done either by using the ACCESS override of BUF64=YES and the CREATE override of CRBUFOPT=64BIT. The IAM Global Options could also be changed to make these the default values. Be aware, that using 64-bit virtual storage for buffers can increase the amount of CPU time being used, so if that is a concern, use this option only when necessary. The MELIMIT value may also need to be increased when using 64-bit virtual storage for buffers as well. Version 9.4 makes the use of 64-bit addressable virtual storage the default for CICS. Experience has shown that under CICS the reduced I/O from having a high number of buffers in 64-bit virtual storage reduces CPU usage more than its usage inherently causes extra CPU overhead.

Base Storage Requirements

The actual amount of storage used to process each file, excluding load modules is provided on the IAMINFO report. This includes the amount of the total storage required and the amount of that storage which was above the line. IAM will generally allocate virtual storage in multiples of 4K size areas, and manage the allocation of that storage. This is done to help prevent storage fragmentation and to improve reliability by reducing the chances of storage corruption that can easily occur when multiple programs are sharing the same page of virtual storage.

The minimum storage requirement per open Enhanced format file is 24K, which is divided into five separate areas (not including the index, buffers, and load module storage). The amount of storage above the line, will vary quite a bit, depending on the options being used. The amount is larger than in prior releases due to the support for z/HPF, format 1 CCW’s, the IOBE and IEDB, which were not used in prior versions and 64-bit virtual for buffers.

Below the Line Storage

IAM limits usage of 24-bit addressable memory (below the line) as much as possible. IAM generally requires only 4K of storage below the line to handle the I/O control blocks. In version 9.2 the actual channel programs have been moved above the line due to use of format 1 CCWs. The IOBs and DCB still need to be below the line. This amount may be larger if more IOBs are needed, or if the file has a very large number of extents. Note that the initial number of IOBs obtained is based on the STRNO value provided. For CICS with the BELOWPOOL=YES Global Option the amount per file will be considerably less than the 4K.

IAM requires approximately 4K of virtual storage to hold the simulated VSAM control block structure, which may reside either above or below the line, depending on what was specified in the ACB. The base VSAM control block area is 2352 bytes for a KSDS type of file, or 752 bytes for an ESDS plus the storage required for each string, or place holder. This area can also exceed 4K, if a larger value is specified for STRNO, which indicates the number of place holders. The place holder size is also impacted by the key length, as described above. For CICS, the VSAM control block area is above the line in 31-bit addressable storage.

CICS Below the Line Storage Reduction

IAM provides a capability to reduce IAM’s use of below the line virtual storage under CICS for opened Enhanced Format IAM files. As stated above, IAM typically uses 4K of below the line storage for each opened Enhanced Format file to contain I/O control blocks (DCB, IOB, and JFCB) and channel programs. With the Below the Line Pooling feature enabled, IAM storage usage will be reduced by at least 50% for IAM Enhanced Format files, when several files have been opened. The feature is enabled by setting the IAM Global Option BELOWPOOL to YES, which is the default value as IAM is shipped. The storage is managed similar to a z/OS cell pool type of structure. As a result, the storage areas for the pools are retained when IAM files are closed, but remain available for reuse, when IAM files are reopened.

Above the Line Storage

IAM keeps the rest of the required control information, buffers, and indexes above the line. The base IAM control block area requires 4K. IAM consists of a buffer table that will fit within an additional 8K, as long as the MAXBUFNO value does not exceed 128. This buffer table increases in size automatically, to keep 64-bit addresses. There is a prime block to overflow table, with a minimum size of 4K or larger, depending on the file size. There are some work areas for data compression, index decompression, and the high level extended index, which depending on their size requirements may fit within the other IAM storage areas.

Buffer Storage

The buffer storage is broken down into single block areas, the size of each is rounded up to a 4K value. A buffer for a typical ¼ track blocked IAM data set, on a 3390 type of device requires 16K. The maximum buffer storage used, is easily calculated by taking the buffer size value, and multiplying it by the maximum number of buffers used from an IAMINFO report. Whenever IAM acquires a buffer and the storage assigned is below the 16-megabyte line,IAM will release the storage.

When the buffer storage is in 64-bit addressable virtual storage, the storage is acquired in 1 megabyte increments. Because it is managed in larger increments than 4k pages, the number of buffers contained in a megabyte is somewhat higher. For example, a typical 1/4 track blocked file will have 76 buffers per megabyte in 64-bit virtual storage, whereas due to rounding of each buffer to a 4k page size, 64 buffers are in a megabyte of 31-bit addressable storage. IAM manages storage in page size boundaries, to reduce storage fragmentation issues in 31-bit virtual storage.

Index Storage

There are three different index areas for an IAM data set. The first is the prime index, which is created when the file was loaded. The second is the index for the Extended PE area, and the third is for the Extended Overflow area. Both the second and third index areas are dynamic, and will change as the file is updated. The Extended PE index is saved within the data set, but the Extended Overflow index for the basic file format is always dynamically built during open processing. The Extended Overflow index for PRO format files is saved in the data set and read in when during open processing.

Prime Index

The prime area index is fixed in size at completion of the file load or reorganization, and will never be updated. This index is based on the high key in each prime block. IAM provides the capability to compress the prime index in a proprietary format that can greatly reduce the amount of storage required for the index. Index compression is an automatic feature, that will be used whenever the prime index exceeds 8000 bytes, and the attributes of the key, fit within the compression criteria. The amount of storage required for the prime index is provided in the IAMINFO report and on the LISTCAT IAMPRINT report. Take the indicated value from one of those reports, and round it up to 4K to determine the amount of storage that will be used for the prime index. See to IAM-Tuning-Guidelines for additional information on the prime index, and controlling the prime index size.

Extended PE Index

The next index storage area is for the Extended PE blocks. This index, like the Prime Index, is based on utilizing the high key within each data block. The size of each entry is the key length plus four bytes, with an entry for each Extended PE block. Because of the internal structure of the Extended PE index, which is organized based on an internal grouping of the extended index blocks, the total storage used for the Extended PE is difficult to predict. Files with large quantities of Extended PE blocks, which are clustered together, may not necessarily use any more storage than a file with a few Extended PE blocks that are sporadically spaced throughout the extended area of the file. While this index is not compressed, it is still a relatively efficient format, because very few files actually have need for this index.

Extended Overflow Index

The last segment of the index storage is for the Extended Overflow blocks. This is a record based index structure, consisting of an entry for each record in overflow. This index is a subset of smaller groups, where each grouping consists of the overflow records from a particular prime block. An overflow index search is done, once it is determined that the prime (or Extended PE) block that should contain the record has associated overflow records. This type of structure is expected to reduce the number of overflow index searches, reduce the number of entries any single search has to scan, and reduce the IAM CPU time for many functions related with overflow.

Estimating the actual storage requirements for the overflow index is difficult. The entries within each subset have compressed key format, but each subset also has header information. Each subset may have some empty entries. As a rough estimate, add four to the key length, and multiply that by the number of records in overflow. The result is the size of the overflow index prior to compressing, which may reduce the storage requirement, although the headers for the subsets will increase the storage requirement.

PRO Overflow Index

When using the PRO overflow format, the index is based on one key per overflow block. This structure may significantly reduce the amount of storage required for the overflow index. However, it comes with a cost of using more disk space than the standard Extended Overflow. PRO format is recommended for files with a large number of overflow records, approaching a million or more.

Index Space

The IAM Index Space virtual storage can reside in either 64-bit addressable virtual storage in the user address space, or as a z/OS Data Space. IAM will place all of the index areas for a file in one of those Index Space areas if available. By relocating these potentially large index areas into the IAM Index Space, there is more virtual storage available within the 31-bit and 24-bit addressable areas of the job step region. This is expected to be beneficial for large online regions, which may have several large IAM files open. The IAM Index Space storage is dynamically acquired by IAM when the first file is opened, that will be using the Index Space, and will be retained until the job step terminates. For any job step, there will be only one IAM Index Space, either in 64-bit virtual or a data space, with all open IAM files, using storage from the same Index Space. The advantage of 64-bit virtual storage is the elimination of the maximum 2 gigabyte sized data space, which was becoming a constraint for some users.

As IAM is shipped, it will use 64-bit virtual storage by default, if available. If insufficient 64-bit virtual storage is available, then IAM will try to acquire a data space. If data space storage is unavailable, then IAM will use 31-bit and potentially 24-bit virtual storage for the index. This can be changed by either changing the IAM Global Options, or on a job step and file by file basis with the IAM ACCESS Override INDEXSPACE.

When using a z/OS data space, the size of the data space requested, is taken from the IAM Global Options Table value for DATASPACE. The Index Space is created to be extendable, with the maximum size set to the smaller of four times the DATASPACE value or 2048 megabytes. The default value for the data space size is 2048 megabytes. Specification of sizes less than 512 is not recommended, because it may cause failures that could be avoided.

Storage Monitor

IAMCMON is a program that can be used to monitor storage usage in long running non-swappable address spaces, such as CICS, IAM/RLS, or IAM/PLEX. The report includes information on the amount of storage used by IAM in the 31-bit addressable space, in a data space if it is being used for the index, and in 64-bit virtual storage. The monitor will also report on storage used by data set for each of those areas when the LISTDSN keyword is specified. Alternately, a WTO message is displayed, if the storage exceeds a user specified maximum quantity, for each of the three areas being reported. For more information see IAMCMON-Storage-Monitor. An example of the report output is shown:

Load Module Storage Requirements

Shown below is a list of the modules required for accessing Enhanced Format IAM data sets, with their approximate virtual storage requirements. Only one copy of each module is loaded, as required, regardless of the number of IAM data sets opened by a task.

Table of IAM Load Module Storage Requirements

For processing typical data compressed Enhanced format files, without any IAM overrides, there is a requirement of approximately 5K of below the line storage, and 265K above the line, for the IAM load modules. If there are overrides, then the below the line storage will increase to 21K, and above the line storage to 317K. Additional storage will be necessary during file open, close, and extend processing.