Processing IAM data sets

After the IAM data set has been defined and subsequently loaded with data, the IAM data set is ready to be used for typical online and batch processing. Processing an IAM data set instead of a VSAM cluster will generally significantly reduce CPU times, physical I/O’s, and elapsed times for most programs. The IAM VSAM interface was developed with the intent of eliminating, as much as possible, the need to change any program, JCL, or CICS tables. Rarely, there may be a need for a change, which can frequently be accomplished with an IAM override. IAM supports almost all of the typical VSAM processing options, including specification of LSR pools, user exits specified via the VSAM EXLST macro, asynchronous I/O, and so forth.

IAM does not support RBA or control interval access of KSDS types of files, and does not support control interval updates for ESDS type files.

Open processing

As part of the functioning of IAM’s VSAM Interface, IAM will screen all open requests for VSAM ACBs. If the ACB being opened is for an IAM data set, IAM routes control to the appropriate IAM module to perform the OPEN processing. If the open is not for an IAM data set, the IAM routes control to the IBM VSAM open processing modules, and does not further screen or interfere with VSAM open or I/O processing.

One of IAM’s major features is that the index to the file is kept in virtual storage while the file is open. This feature not only eliminates I/O for the index component, but also provides for savings in CPU time. IAM can always get to any specifically requested record with no more than one I/O. Because the index is read into storage at open time, open processing for an IAM data set may take longer than a VSAM open. An IAM open also involves acquiring virtual storage required for processing, acquiring buffers, and setting up the I/O control blocks, and loading the necessary IAM load modules. IAM’s self-defining file structure eliminates the need for retrieving information about the data set from the system catalog, which eliminates that VSAM overhead.

Reading the index to the prime area and the prime extension is a straightforward process. Building the overflow index involves more processing, because the overflow index itself is not saved within the IAM data set unless PRO is being used. The basic overflow index contains one entry for each record in overflow, unlike the prime and prime extension index that contains the high key in each block. To build the overflow index, IAM must read all of the used overflow blocks, determine the key of each overflow record, and insert the key into the proper position in the overflow index. As more and more records are put into overflow, this process will take longer to complete. Regular file reorganizations will help to keep the open processing time at a minimum.

IAM Open processing also automatically performs an implicit verify operation. On files that have been opened for update, IAM will update the extended index and end of file information if necessary.

Once the file open process has completed, IAM is ready to handle the application I/O requests.

I/O request processing

I/O to an IAM data set is performed by using the standard VSAM application programming interface (API). Standard VSAM protocol is used by IAM, including error reporting. IAM does not abend when an I/O request fails, the appropriate return codes and failure codes are provided, and the appropriate error exit routine is also invoked if so specified by the calling program. As with VSAM, it is the responsibility of the application program to determine the appropriate action for any particular type of error. The various error codes set by IAM are documented in IAM I/O Request Error Codes of the IAM space.

For KSDS types of files, IAM handles all requests except those that specify Relative Byte Addressing (RBA) or Control Interval access. Such requests are failed with an error code of 104 (x’68’) and a return code of 8. For ESDS files, IAM does not support Control Interval updates, which will fail with an error code of 68 (x’44’) and a return code of 8. For fixed length record RRDS files, IAM supports control interval reads and updates when the record size and control interval size are such that only one record will fit within a control interval.

Buffering

IAM’s dynamic buffer management technique, Real Time Tuning, works automatically in response to the application I/O requests. The IAM buffering technique dynamically adjusts buffering and I/O techniques, along with adjusting the number of buffers, based on the types of requests being issued by the application. IAM takes the guesswork out of the tuning process. There is no need to be concerned about whether or not to use LSR type of buffering, or how many buffers are needed for optimum I/O performance. IAM will select the buffering technique and quantity, within specified or defaulted limits, automatically. Buffering and I/O techniques are adjusted dynamically in response to the application requests.

Real Time Tuning works within a range for quantity of buffers, referred to as MINBUFNO and MAXBUFNO. These ranges can be specified through the IAM Override facility, using the MINBUFNO and MAXBUFNO keywords. Default values for MINBUFNO and MAXBUFNO are determined from the IAM Global Options Table. Using the default Global Option values will eliminate the need for increasing the MAXBUFNO value for all but the most heavily accessed data sets. The MAXBUFNO value can also be increased by the BUFND parameter specified in JCL as a sub-parameter of the AMP parameter, or within the program in the ACB.

64-bit virtual storage

The default as shipped is that, IAM will use 31-bit addressable storage for the buffers it acquires for address spaces other than CICS regions. For CICS, the default is to use 64-bit addressable storage for the buffers. The use of 64-bit storage for buffers is specified by either the IAM ACCESS Override of BUF64=YES or by enabling the IAM Global Option BUF64. For users concerned about saving CPU time, please be aware that the use of 64-bit virtual storage for buffers does increase the CPU time for batch jobs due to z/OS page fix processing. For CICS, this extra CPU time is more than compensated by saved I/O processing CPU time, due to using a large number of 64-bit buffers.

For more information on IAMs Real Time Tuning capability, see IAM-s-Real-Time-Tuning of the IAM space.

Turbo mode buffering

Turbo mode is invoked during periods of heavy I/O activity on any particular IAM file, and will increase the frequency of acquiring additional buffers up to the MAXBUFNO limit, so that IAM will be more responsive to periods of heavy I/O requirements. With the IAM Global Options as shipped, Turbo mode will be invoked as necessary. Coupled with the default limits for MAXBUFNO Turbo Mode will help reduce the need for providing MINBUFNO and MAXBUFNO overrides.

z/HPF high performance FICON

IAM will automatically utilize z/HPF I/O architecture channel programs when zHPF processing is enabled in the system, and when the operating system and DASD devices being used support the z/HPF functions required for use by EXCP. This I/O architecture provides for more efficient use of the FICON channels, increases channel capacity to support more I/O activity, decreases channel connect time, and may reduce elapsed times for jobs that can utilize this function.

Caching overflow blocks

For files with large overflow areas that are being sequentially processed, IAM has a feature that uses z/OS 64-bit virtual storage, referred to as above the bar storage. This capability can help improve run time for sequential processing, including backups and reorganizations of Enhanced Format IAM files with a large number of records in overflow. During OPEN processing, IAM reads the all of the overflow area blocks to build the record level index. With this new caching feature enabled, the overflow blocks will be copied to an above-the-bar virtual storage area for later reference during file access. When one of these blocks is needed for a data record, IAM will copy the data block from the above-the-bar storage area into a normal I/O buffer, rather than performing a physical I/O.

The benefits for installations with sufficient real storage are reduction in physical I/Os and elapsed times to read the file sequentially. Installations without sufficient real storage will likely find that the time spent in paging will be detrimental to job performance. To use this new feature, simply specify the IAM ACCESS Override of CACHE64. IAM will calculate the amount of above-the-bar storage required to hold the overflow blocks, and acquire it. Many users may also need to specify the MEMLIMIT= parameter on their execute cards to make above the bar storage available to the job step, for example MEMLIMIT=4G for 4 gigabytes of above the bar storage.

Dynamic tabling

Dynamic Tabling is a special feature of IAM that keeps the most frequently referenced randomly read records in a virtual storage cache. It is a different concept than buffering which handles blocks containing multiple records. Dynamic Tabling is record oriented, and is useful when a subset of the records scattered throughout the file are repeatedly referenced, potentially reducing the virtual storage needs to have a large number of buffers. This facility provides substantial benefit particularly for some online applications, by eliminating physical I/O as records are repeatedly retrieved from the table. Applications that may potentially benefit from this feature are those that have a high volume of GET RANDOM requests, combined with a very low quantity of PUT UPDATE and ERASE requests. The update requests are detrimental because they will force IAM to read the data block into storage, to perform the update. Other types of I/O requests will not be impacted by the use of the dynamic table.

Dynamic data space

The primary choice for placement of the dynamic table is the IAM Dynamic Data Space feature, where the dynamic table is placed in a z/OS Data Space. The use of a Data Space for the dynamic table allows for a larger number of records to be placed in the table. For the Dynamic Data Space, IAM has revised the algorithms used for storage management, record management, and the record search to provide for more effective utilization of storage resources, a faster record search capability, and reduced CPU overhead when inserting and removing records from the table. When records need to be removed from the table to make room for more recently referenced records, IAM now utilizes a recently used algorithm for selecting the records to be removed.

The use of the Dynamic Data Space is on a file by file basis, with each individual data set for which the function is requested using its own data space. The capability is easily implemented by the use of the IAM ACCESS Override of DYNDS. Using the DYNDS= override, users simply specify the amount of storage that IAM is to use for the Dynamic Data Space in megabyte increments, up to a maximum of 2048, or 2 gigabytes. The virtual storage required for management of the Dynamic Data Space will come out of the Data Space itself so that use of the Dynamic Data Space will not impact virtual storage resources within the address space processing the dataset itself.

DYNCORE

DYNCORE is the original IAM dynamic tabling facility, which uses above the line virtual storage within the requesters address space. Therefore the amount of virtual storage is more limited than with the Dynamic Data Space. DYNCORE can be used on all IAM file formats, and is useful when the table size requirements are not too large.

This capability is easily used. Simply specify the table size, in kilobytes, using the IAM ACCESS DYNCORE override. Statistics on the DYNCORE usage are provided in the IAMINFO report.

File expansion

IAM Enhanced Format files are dynamically expandable, through the use of secondary extents. IAM files that are not SMS Extended Format can be extended up to a maximum of 16 extents per volume for a maximum of 944 extents across 59 volumes. IAM files that are SMS Extended Format can be extended up to a maximum of 123 extents per volume, for up to 59 volumes. IAM Enhanced Format data sets do not have an explicit total maximum number of extents. As records are added, or updated records increase in length, additional DASD space is acquired and formatted as needed. This expansion area is called the IAM Extended Area. The Extended area consists of Extended Overflow blocks and Extended PE blocks. The Extended PE blocks are used to handle the addition of records with key values higher than the highest key value in the data set, while the Extended Overflow blocks will handle all of the other file expansion needs.

File expansion processing now updates the first block in the file, which includes updating the file statistics and the file update time. This enables improved recovery should a failure occur that does not properly close the file, or when copying a file while it is opened and has gone through a file expansion process.

For IAM files that are not SMS Extended Format, if they experience larger than anticipated growth, IAM can dynamically increase the size of the secondary space being requested. Once a data set has five extents on any particular volume, IAM will increase the secondary space by the MAXSECONDARY factor, providing that the value does not exceed the original primary space requested, and that there is sufficient space on the volume for the desired quantity.

Index space

The IAM Index Space is an area of virtual storage that IAM uses for to contain the indexes to the open IAM files. The Index Space can reside in 64-bit addressable virtual storage, or within a z/OS Data Space. Using the 64-bit addressable virtual storage eliminates the restriction of a maximum of 2 gigabytes for the index storage that exists for a data space that some users have encountered. IAM by default uses the 64-bit virtual if available, then a z/OS data space, and if neither of those are available then storage within the address space that is 31-bit addressable will be used. Users may need to code the MEMLIMIT JCL parameter on the EXEC to use 64-bit virtual storage.

Close processing

Once processing has been completed for the IAM data set, the application program should explicitly close the data set. If the data set is not explicitly closed, then task termination will close the data set when the task that opened the data set completes. The basic functions of close processing are as mentioned below:

On files opened for update, close processing will write out any updated blocks to the data set that have not yet been written, and update the file statistics and control information.
Close the internal DCB used by IAM to access the data set.
Free main storage used to access the data set.
Delete the processing modules used for accessing the data set.
If an IAMINFO DD is present, format and produce the IAMINFO report.
Produce an SMF record of file activity, if so enabled by the IAM Global Options.
Disconnect the ACB from the control blocks built, and from the system.

Once the data set is closed, no further I/O activity can take place. The application program should check the return code from the Close, to make sure that it was successful. A return code of 4 indicates that the Close failed. It is rare that a Close will fail, but it can happen. The main reason for a Close to fail is that there is insufficient storage for the close to process. This is indicated by an error code of 136 (x’88’), that is returned in the ACB error code field. If this error occurs, the application should free some main storage, or close other files, then re-attempt to close the IAM file.

File full or Sx37 error

For IAM files opened for update, a file full error is not considered by IAM to be a fatal error. The Sx37 abends will be masked out by IAM for Enhanced Format IAMdata sets, and treated as a file full logical error. The updated or inserted record will not be written out to the data set. For updated records, the non-updated version of the record will remain in the data set. The error conditions will be reported back to the calling program, using standard VSAM logical error codes, and exits where applicable. To prevent getting into a reoccurring Sx37 loop, IAM will not try to take any additional extents for a file once such an error occurs, unless the data set is closed and reopened.

File full errors can also occur when IAM is unable to obtain additional virtual storage for the expanding index structure. This type of file full condition is accompanied by an IAMW03 error message.

This section provides more information about the following topics: