IAMDRCC — Dynamic Reorganization Overview

IAMDRCC Overview

The IAM Dynamic Reorganization process improves data availability by enabling a major portion of the data set reorganization process to be performed concurrently with online CICS region updating of the data set. The reorganization process will run as a batch job and is designed to be performed automatically without any need for human interaction.

Dynamic Reorganization can optionally be performed with manual control via WTOR prompts if desired. A summary of the Dynamic Reorganization process is as follows:

1. Establishes communication with CICS region and IAM/RLS region if applicable that has the specified IAM data set opened for update, and set up a mechanism to track any changed, deleted, or inserted records. IAM/RLS using batch jobs can not have the data set being reorged open for update processing while the IAM Dynamic Reorg is running and this is verified.
2. Calls the IAMSRORG reorganization copy program to make a copy of the IAM file as it is being used under CICS into a new IAM data set specified by the TODSN parameter.
3. As that copy is being performed, CICS continues with normal update processing of the IAM data set, while IAM tracks any updates that are taking place.
4. When the IAMSRORG copy function has completed, Dynamic Reorganization will do an initial update phase by applying the tracked changes to the new reorganized copy of the file.
5. Optionally activate two CICS GLUE exits that will cause any transaction Reads and Writes to the reorganized files across process #6 (below) to tolerate the file Close, Rename, and Open. This is done by delaying any Read or Write activity that would get caught in the item 6 process, such that any active I/O will still receive a zero CICS response code and all I/O to the reorganized files will be satisfied for the entire duration of the Dynamic Reorg running.
6. Close the file under CICS, then apply any additional updates to the new copy of the data set, do the renames, then signal CICS to re-open the data sets to resume processing.
7. Deactivate the two CICS GLUE Exits described in Step 5 (above) if applicable.
8. The backup of the data set is in the renamed original data set. This can be copied if desired into a flat file with the IAMSRORG program EXPORT function or FDRREORG for example.

IAMSRORG uses various optimization techniques to perform the reorganization copy quickly. While it is important to complete that phase quickly, the critical time period is the brief period the file is closed for Step 6 above to complete the updates, rename the data sets, and re-open the data set to CICS, which is expected to complete in a few seconds.

When to Use ?

The Dynamic Reorganization is designed and intended to be run during periods of low data set activity on the data sets being reorganized. This is recommended to minimize the impact on the CICS region, and to minimize the amount of time that the data set will need to be closed for the synchronization of the updates on the new file with the original.

IAMDRCC can run multiple concurrent data set reorganizations to either the same or different CICS address spaces. This process is facilitated through the use of multiple subtasks and the specification of multiple REORG statements. The maximum number of subtasks IAM Dynamic Reorg is to use is specified by the EXEC card PARM=nn where nn specifies the maximum number of subtasks, up to 16. We recommend that you start with a small maximum value of 2 to 4, and then adjust accordingly based on the throughput at a particular value in your configuration, and determine what is reasonable for use based on your system’s capacity.

As of IAM Version 9.4, data sets that are controlled by IAM/RLS and open to both CICS and IAM/RLS can now also be used with IAM Dynamic Reorg. The only stipulation is that the data set can not be open for update from a non-CICS address space while the IAM Dynamic Reorg utility is processing the data set.

CICS Customization

The Dynamic Reorganization feature requires some customization of the CICS address spaces that contain the data sets that the reorganization process is to be used with. This includes setup for the EXCI interface utilization and related resources, the dynamic reorganization programs and transaction that will be executed. As of IAM Version 9.3/02 Spin 10, IAMDRCC provides the capability to dynamically enable two CICS GLUE exits that will allow CICS transaction Reads and Writes to complete successfully across the last reorg steps, the Close, Rename, and Open process. The CICS definitions needed for the two GLUE exits are provided. The information on doing this is provided in Section 43.05.

Automatic Operation

Automatic operation of the Dynamic Reorganization Process is the default and involves automatically closing the IAM data set under CICS when IAMDRCC is ready to swap in the new data set, automatically perform the renames, and then automatically have CICS open the reorganized data set. If necessary, these actions can be controlled by the AUTO parameter. Any specification of the AUTO= keyword will turn off full automation, and set as automatic only the actions specified, or if AUTO=NO will turn off all three automatic functions. Each of these automatic functions is described.

Close Process

Once Dynamic Reorg has completed the copy of the file and the initial update phase, it is ready to close the file under CICS. This is done automatically unless otherwise specified by the AUTO= keyword. If the automatic close is disabled then Dynamic Reorg will issue a WTO message IAML0318 prompting the operator to close the data set under the CICS address space. Delaying the close will elongate the time needed to activate the reorganized copy because updates are continuing to the original file that will have to be applied to the reorganized copy once the file is closed.

When IAMDRCC confirms that the file is closed under the target CICS address space, it will perform Phase 2 of the resynchronization process of updating the new copy with any records that were updated subsequent to Phase 1 completion. When phase 2 of the update synchronization completes, Dynamic Reorg is ready to rename the data sets. This is done automatically unless the RENAME function is disabled.

Rename Process

Dynamic Reorg will rename the old data set based on the SAVEDSN, SAVESFX or SUFFIX parameter, If automatic rename is disabled then the operator will need to reply to the IAML0319 message to allow the rename to be performed. If the operator replies ‘T’ to that WTOR, then the Dynamic Reorg will terminate, leaving the data set disabled to the CICS address space. The data sets will not be automatically renamed but left as they were. So there will be the new reorganized and synchronized data set and the original data set for the user to decide what actions if any they want to take.

Open Process

Once Dynamic Reorg has completed the rename process, then Dynamic Reorg will re-open the newly reorganized data set under CICS and then terminate. If the automatic open process is disabled then the data set will not be automatically opened and Dynamic Reorg will terminate leaving the file status as close and disabled.

Reorg Completion

Upon completion, you’ll need to have a plan to manage the renamed original data set. It should be retained in some form for a reasonable period of time as a backup. One possible suggestion is to use IAMSRORG to export the data set to a tape or disk. The z/EDC compression can be used for the exported copy on disk to further reduce the amount of space utilized for the backup. The IAMSRORG IMPORT can be used to restore it if necessary. That process will result in a reorganized data set ready for use.

Special Considerations

The IAM data set to be reorganized must be open for update in the CICS address space that IAM Dynamic Reorg is going to communicate with. The data set must not be open for update by any other address space except for IAM/RLS or IAM/PLEX address spaces. It is presumed that the data set will remain open for update until such time as the reorganization process is ready for the data set to be closed.

Unavailable File Status Codes

During the time that the data set is unavailable, file control requests will get a response of either Response Code 19 with Response Code2 60 or Response Code 84 with Response Code2 50. An application could retry any such request in one or two seconds.

Unavailable File Status codes can be avoided by making use of the Dynamic Reorg High Activity Toleration CICS GLUE Exits. The exits can be dynamically enabled and disabled by using the BEGIN CICS=xxxxxxxx and END CICS=xxxxxxxx SYSIN statements. See the EX4304B example on how this is done in IAMDRCC-Examples.

Open to Multiple CICS Regions

If the data set is open to other CICS regions for read-only, then you’ll likely need to run the dynamic reorganization manually to facilitate closing the file in the other CICS regions. Closing the data set prior to that time will result in turning off the change tracking process, and result in potentially having to restart the reorganization process if it is opened for update either under the CICS region, or by some other job or CICS region.

Alternate Index

Dynamic Reorganization can reorganize a base cluster with one or more alternate indexes and can reorganize an alternate index. All of the IAM alternate index information is kept based on the original names. It is expected that the reorganized data set will be renamed back to the original data set name so the IAM recatalog process is not needed. It is not necessary to rebuild the IAM alternate indexes after the base cluster has been reorganized using this process. Please note that all of the catch up update processing for the reorganized data set will be done in phase 2.

The Dynamic Reorg High Activity Toleration CICS GLUE exits, that are enabled with the use of the BEGIN CICS=xxxxxxxx statement and disabled with the use of the END CICS=xxxxxxxx statement, only support base clusters that have no more than one alternate index. If a base cluster has more than one alternate index then the reorg of such a cluster using the IAMDRCC utility needs to be done during times of low to no activity as was the case before the HAT CICS Exits were introduced.

CICS Transaction Programs that issue SYNCPOINTs

When a CICS File Close is issued any transaction that is in flight and using the file that is being closed will be allowed to complete before the file is actually closed. At the time that IAMDRCC needs to perform the file Close, Rename, and Open, if enabled, the Dynamic Reorg High Activity Toleration CICS GLUE Exits are dependent on this behavior to achieve their goal of having all CICS Reads and Writes complete successfully. If a CICS transaction program issues a purposeful EXEC CICS SYNCPOINT call, then CICS will immediately close the involved file if a file Close has been issued. If the transaction program then goes on and does further Reads or Writes the Dynamic Reorg HAT Exits will not be able to prevent File Unavailable CICS response codes for such a transaction during the reorg file Close, Rename, and Open process. However, since such programs would be subject to the same conditions under a non Dynamic Reorg File Close, it is likely that programs that issue CICS Syncpoints include code for dealing with File Unavailable response codes.

Copy Only Function

Starting with IAM Version 9.4, the IAM Dynamic Reorg utility now has a COPY function, and not just a REORG function. Soon after IAM Dynamic Reorg began to be used by customers, the ability to use the utility to get non-fuzzy backups throughout the day was exploited with great success. IAM Dynamic Reorg with the AUTO=NO keyword and a reply of ‘T’ (terminate) to the AUTO=NO associated WTOR, effectively provides the user with just a backup file. With IAM 9.4, users can use the COPY verb to achieve the same effect with no WTOR that requires a console or console automation product reply.

Performance Example

A test was performed in our laboratory on a z/13s processor using a base cluster with an alternate index with the following characteristics:

• Record Length of 325 bytes
• Key Length of 43 bytes
• Alternate Key Length of 52 bytes
• Loaded with 9,000,000 records
• Then inserted 500,000 records prior to CICS processing
• At time of reorganization, it had 9,508,792 records

There were 4 different tests run as follows with time unavailable to CICS:

• IDCAMS with VSAM data set using SMB - 7 Minutes 55 Seconds
• IDCAMS with an IAM data set - 2 Minutes 38 Seconds
• IAM Special Reorganization of IAM data set CICS - 23 Seconds
• Dynamic Reorganization of IAM data set while open to CICS - 2 Seconds

The results below indicate the time that the data set was unavailable to CICS for processing in seconds:

Graph of Data Set Unavailable Time in Seconds

Concurrent Reorg Performance

Additional tests were performed to determine the benefit of the new concurrent reorganization process. Three VSAM KSDS files were reorganized using standard IDCAMS reorganization process with three jobs running concurrently. Then the three files as IAM were reorganized with the Dynamic Reorganization concurrent processing with 3 subtasks. The attributes of each file were:

• File 1: 3,003,677 records RL=200 KL=41
• File 2: 3,003,188 records RL=322 KL=42
• File 3: 9,508,792 records RL=325 KL=43 with an AIX

The VSAM batch jobs, which were all started at the same time, were completed in a best time of 8 minutes, plus 23 seconds, for a total of 503 seconds. To that time, the time to close the files under CICS and the time to re-open the data sets would need to be added.

For the IAM reorganizations, one job ran each reorganization as a subtask. The processing time for this job includes the automatic open and closing of the data sets as part of the reorganization process. The best time for the concurrent reorganizations with IAMDRCC was 53 seconds. The time the data set was unavailable to CICS was 3 seconds.

The reorganization of the three files by IAMDRCC ran 90% faster than the jobs performing the standard IDCAMS VSAM reorganization process. For IAM, this included the time needed to automatically close and reopen the data sets when needed. The files were unavailable to CICS for 3 seconds for a 99.5% reduction in the time that the IAM files were unavailable compared to VSAM using IDCAMS to reorganize the three data sets.