Reorganizing IAM Data Sets

When to Reorganize?

Many applications that have been using VSAM or IAM data sets already have regularly scheduled data set reorganizations. Generally, there should be no need to change the schedule. For data sets that are only reorganized when needed, the difficulty is in determining when an IAM data set should be reorganized. With Compatible format IAM files, it was relatively easy because of the fixed size of the overflow area. With Enhanced format files the ability to acquire extents and now with variable overflow, it is not quite as clear cut. Amongst the criteria to consider are:

1. Amount of virtual storage required for the Extended Overflow Index. This can be estimated by multiplying the number of records in Overflow by the key length plus 4.
2. Percentage of records that are in Extended Overflow. For example, if more than 10% of the records in the data set are in Extended Overflow, then the data set should be reorganized.
3. Number of extents being used for single volume data set.
4. The Extended area of the file exceeding a specified quantity of DASD space.
5. Exceeding specified quantity of records in Extended Overflow.

To aid in determining when a data set should be reorganized, IAM will produce warning messages during open or close, if certain conditions are detected, for which reorganization might be warranted. The messages including the numbers have been changed in Version 9.2. The message numbers are IAMW21, IAMW22, IAMW91, and IAMW92. These messages are meant to bring to attention that reorganization should be considered for the indicated data sets. Certainly some data sets may continue processing for several days or weeks without a reorganization after hitting a threshold condition, but only experience will be able to indicate if that is true. Other data sets, such as small data sets with very high activity, may need to be reorganized more frequently. The IAM thresholds include the Extended Overflow index exceeding sixty four megabytes of storage, using thirteen or more extents for a single volume data set, or when the overflow area exceeds one thousand cylinders. The Overflow override can be used to monitor how much overflow space is being used, similar to what was done for Compatible format IAM files, except that the limit is no longer the absolute limit. By providing the override, IAM will monitor and inform you via an IAM message that reorganization is recommended because the Extended Overflow area exceeds the number of records specified.

Reorganization can be done automatically. BMC offers a software package, FDRREORG, which can automate file reorganizations for IAM, VSAM, and PDS types of data sets. Other alternatives used by some customers include, using the IAM SMF reports or the IAM SMF records directly to trigger data set reorganizations, and using the IAMW22 message. Also, the IAMPRINT report produced by a LISTCAT can trigger data set reorganization.

How To Reorganize?

IAM data sets can be easily reorganized with the FDRREORG product from BMC. FDRREORG can search out all or the selected IAM data sets, and if they meet the specified criteria, it will be automatically reorganized. FDRREORG maximizes buffering for fast reorganizations, can reorganize multiple data sets concurrently, and has a feature to DELETE and DEFINE the data sets that are being reorganized. FDRREORG provides the capability to read from each volume of a multi-volume IAM data set concurrently (the MAXPARALLELBACKUPS= keyword), which can result in faster reorganization times for very large data sets.

IAM, starting with IAM Version 9.3/02 Spin 12, in conjunction with the FDRREORG Version 5.4/85 or higher provides enhanced reorganization processing for IAM Enhanced and Encrypted format data sets. See IAM-Enhanced-Data-Set-Reorganization for information on using IAM Dynamic Reorg and/or the IAM IAMSRORG batch utility to perform reorgs. The goal of IAMDynamic Reorg and the IAMSRORG utility is to provide increased data availability for IAM data sets, by reducing the time of data unavailability due to data set reorganizations. IAM Dynamic Reorg can reorganize an IAM data set while it is open for update in a CICS region and starting with IAM 9.4, the IAM data set can be under IAM/RLS control.

The other common way to reorganize an IAM data set is with IDCAMS REPRO. With appropriate buffering, IDCAMS REPRO can perform the reorganization optimally when using this utility. The general technique is to REPRO the data from the IAM data set into a sequential data set, then optionally delete and redefine the IAM data set, followed by an IDCAMS REPRO REUSE from the sequential data set into the IAM data set. An alternative with IDCAMS is to REPRO from one IAM data set into another IAM data set without going to a sequential data set. This could save considerable time, provided, there is the DASD space available for both copies of the data set, and that a sequential back up of the data is not required.

Some installations have written programs to read the IAMSMFVS report for IAM data sets or to read the optional IAMSMF records directly, and then build and submit jobs to reorganize the selected files. The IAMPRINT LISTCAT output has also been used for this purpose.

Some application software packages provide their own data set reorganization utility. The one thing to observe with such utilities is, to determine if they are doing a single record load followed by a mass insert. Such a process will result in the data records all being placed within the extended areas of the file, which may adversely impact the performance of the file in subsequent executions. If such is the case, then an additional reorg by IDCAMS or IAMSRORG should be done after the application reorg to ensure optimum performance.

Alternate Index

IAM base cluster types of data sets with alternate indexes are marked as non-reusable by IAM, same as with VSAM. The base cluster cannot be reorganized without deleting and redefining the base, and all of the related components except by FDRREORG. This is to help confirm that the alternate index files retain their integrity with the base cluster. The IAM Alternate Index data sets can be reorganized as needed, or they can just be rebuilt. FDRREORG V5.3/50 or higher is required to reorganizeIAM Alternate Index data sets, or FDRREORG V5.4/20 if any of the IAM data sets are hardware compressed or in a SMS Extended format.

RRDS

IAMRRDS data sets (both fixed length record and variable length record) may need to be periodically reorganized. The need for RRDS reorganization is based upon a large percentage of the records being in the Extended Overflow area of the data set. IDCAMS can be used to reorganize RRDS data sets, however, there are some special considerations. The unique nature of RRDS files is the location of the “key”, that is, the relative record number is not stored within the data record as seen by application programs. If there are empty slots (that is., unused record numbers) within the data set, IDCAMS does not retain the record number for a typical REPRO when the data is copied to an intermediate sequential data set. Therefore, to use IDCAMS to reorganize RRDS files, one must either perform a REPRO from an RRDS data set directly into an RRDS data set. With IAM, there is another option which is to use the IAM BACKUPCOMPRESSED option.

If you use the method of performing the REPRO from an RRDS directly into another RRDS, you may also want to use the MODEL parameter when defining the target RRDS to make sure it has the proper attributes. The MODEL works fine for fixed length record RRDS files, however, it results in a KSDS for variable length RRDS files, because IDCAMS does not recognize that the data set being modeled is a variable length record RRDS. (This is true for both VSAM and IAM). To use the MODEL parameter successfully with a variable length record RRDS, the RECORDSIZE parameter and the NUMBERED parameter MUST BE SPECIFIED.

When using the IAM BACKUPCOMPRESSED option with IAM RRDS files, the relative record number will be included in the record provided to IDCAMS. When reloading an RRDS with BACKUPCOMPRESSED specified,IAM will check the input record for the relative record number to be used for each record. When performing such a reorganization, the sequential output file must be allocated as a variable length (RECFM=VB) file, with an LRECL of the record length in the IAM file plus 12 bytes.

See Examples F, G, and H for sample JCL to reorganize IAM RRDS files.

Backup of Compressed Data

AnIAM feature that may help speed up the reorganization of software compressed data sets is to not decompress the data while it is being read, and to accept the IAM software compressed data format on the reload. This feature will eliminate the CPU time spent decompressing and compressing the data, and eliminate I/O transfer time to the output device. If the output is a tape device that offers compression, there normally will not be any savings in tape usage because the data is already compressed. FDRREORG will automatically invoke this facility. For IDCAMS REPRO, the IAM ACCESS and CREATE Override of BACKUPCOMPRESSED must be specified on both the backup and the reload portion of the reorganization.

The compressed backup file can be converted to an uncompressed sequential file by the IAMRECVR DECOMPRESS command. Alternately, IAM provides a callable service that will read and decompress a sequential file with compressed IAM data, and can also write  user data to a sequential file in IAM compressed format. Refer the Reference Section of this space for further information on using the callable service. 

Example A: Multiple IAM Data set Reorg with FDRREORG

The first example of reorganizingIAM data sets is with FDRREORG. In this example, all catalogedIAM data sets that have a high level index of MYIAM will be considered for reorganization. The criteria being specified is that if any of the following are true, the data set will be reorganized:

• The Overflow Index exceeds 4 megabytes.
• More than 10% of the records in the file are in the Extended Overflow area.
• There are more than 1,000,000 records in Overflow.

Defaults for the backup data sets and for the maximum tasks will be used.

Reorganization of Multiple IAM Data sets Selected by FDRREORG (EX1081A)

Example B: Parallel Volume I/O with FDRREORG

In this example, a single multi-volume IAM data set is reorganized in parallel mode. This feature allows concurrent reading of the different volumes on which the IAM data set resides, which can reduce reorganization time depending on the I/O configuration for both the source IAM data set, and the backup data sets. Note that as required by FDRREORG, the BACKUPINDEX is specified with a ? embedded so that each volume backup has a unique name.

Reorganization with Multiple Volume Parallel Read (EX1081B)

Example C: IDCAMS Reorg with Delete and Define

In this example, the IAM data set is reorganized with an IDCAMS REPRO. IAM Overrides are used to not decompress the data. The sequential data set is being written to a 3390 type of DASD volume, where ½ track blocking is being specified. A permanent data set name is being given for the sequential file, and that the data set is being kept and cataloged, even if the job abends. It can be deleted once the reorganization has been verified as successful. Extreme caution must be used to prevent the loss of valuable data. The use of the IF MAXCC conditional operation under IDCAMS, will stop the reorganization if any errors occur. This is done to provide maximum protection for the data being reorganized.

The IAM data set is referenced by the IDS and ODS parameters. This will cause IDCAMS to dynamically allocate the IAM data set, which is being done just in case the define results in placing the IAM data set on a different volume.

Example of using IDCAMS to reorganize an IAM data set (EX1081C).

Example D: IDCAMS Reorg with REUSE

In the next IDCAMS example, the IAM data set is reorganized without doing a DELETE and DEFINE. Therefore, the IAM data set can be specified in JCL without any concerns.

Example of Reorganization with IDCAMS, without a Delete and Define (EX1081D)

Example E: IDCAMS REORG into a second IAM data set

The following example demonstrates a different approach. In this example, one IAM data set is copied over another IAM data set with IDCAMS REPRO REUSE, resulting in a reorganized image of the original data set. This will reduce the reorganization time by eliminating the writing to and reading from the sequential backup file. After the reorganization, renames of the data sets are done. Using this type of procedure will enable a fast reorganization to minimize the time that the data is not available. The original copy of the data set can be used for subsequent backup or for read only processing, while normal update processing can then be resumed on the receiving data set.

IDCAMS Reorganization to another IAM Data set (EX1081E)

Example F: IDCAMS REORG of an RRDS

The example below demonstrates how to reorganize an IAM RRDS file, using a sequential file as an intermediate storage place for the data. The IAM BACKUPCOMPRESSED option must be used when performing reorganization, because IDCAMS will not retain the record numbers in the sequential output data set. With BACKUPCOMPRESSED, the record numbers will be included by IAM in the output record for the sequential file, and will likewise be present when reading the sequential file so that IAM can pick up the record number on the reload. This example does include a delete and define of the RRDS, however a REPRO REUSE could be used instead of performing the delete and define.

Example of RRDS Reorganization with IDCAMS (EX1081F)

Example G: IDCAMS Reorg directly into another RRDS

In the example below, a fixed length record RRDS is reorganized directly into another fixed length record RRDS. To make sure that the target data set has identical attributes, it is defined with the MODEL parameter specifying the input data set. If the copy is successful, then the original data set is deleted, and the new data set is renamed to the original data set name. This example is only valid for fixed length record RRDS data sets, if you need to reorganize a variable length RRDS using this method, see Example H below.

Example of RRDS reorganization directly into another RRDS (EX1081G)

Example H: IDCAMS Reorg into Variable RRDS

This example is similar to Example G above, except that this is for a variable length record RRDS. When using the MODEL sub-parameter of the DEFINE request for variable length record RRDS files, one must also specify both the RECORDSIZE parameter and the NUMBERED parameter, otherwise IDCAMS defaults to a KSDS type of data set. For an IAM variable length record RRDS, you can leave off the RECORDSIZE parameter, however, it is needed for a real VSAM RRDS.

Example of Variable RRDS Reorganization (EX1081H)