FDRREORG overview

FDRREORG consists of several parts:

• The FDRREORG program, described in this chapter, automates the reorganization of the data set types listed above.
• The FDRCOPY program, although it is a part of the base FDR product (described in Working-with-FDRCOPY), it is enhanced when FDRREORG is licensed to include a REORG statement, used to reorganize PDSs. The REORG statement is described in FDRCOPY REORG Statement. FDRCOPY REORG is invoked internally by FDRREORG to reorganize PDSs, and it can also be executed directly.
• FDRREORG has its own option table, separate from the FDR Global Options. The FDRREOZO program is used to set and display FDRREORG options and defaults. It is described starting in FDRREOZO JCL Requirements.
• FDRREORG can be invoked for individual data sets from the FDR ISPF panels, as described in FDRREORG ISPF Interface.

FDRREORG overview

FDRREORG logically reorganizes IAM and VSAM KSDS data sets or AIXs, and compresses partitioned data sets, based on user specified exclusion and selection criteria. Data sets can be reorganized on an as needed basis using operands that define the characteristics of data sets requiring reorganization. FDRREORG can also be run in simulation mode to obtain a report of those data sets meeting the users’ selection criteria.

FDRREORG attempts to obtain exclusive control of a data set during its reorganization. For IAM data sets, FDRREORG issues the appropriate IAM enqueues to lock out all updates to the data set while it is being reorganized. For VSAM data sets, FDRREORG sets the update inhibit indicator for the data component of the VSAM cluster being reorganized. If FDRREORG is unable to obtain exclusive control, the data set is bypassed if the default DSNRETRY option is being used. FDRREORG can be instructed to retry data sets that were in use and can also issue the appropriate system enqueues to process a data set when it becomes available. If it is critical that a particular data set or set of data sets be reorganized, FDRREORG can be instructed to wait for these data sets to become available.

All IAM or VSAM data sets selected for reorganization are backed up to either TAPE or DASD, and then immediately reloaded from the backup. The backups are logical backups obtained by using the standard access method interfaces. The striped file format can be used for backups to DASD. VSAM data sets with records that are larger than 32,763 are not supported. The FDRREORG backups can be used to reload a data set using IDCAMS REPRO (except for IAM RRDS data sets) or any other standard file copy utility that supports VSAM data sets. If you are creating compressed IAM backups, you must use the BACKUPCOMPRESSED IAM override to use the backup with IDCAMS REPRO. See the IAM manual for additional information. The user has the option of creating the backup data sets as either standard sequential data sets or as GDGs. If GDGs are chosen, FDRREORG dynamically defines the base GDG to an ICF catalog if one does not already exist. GDGs cannot be used with the parallel backup option. The names of the backup data sets are generated dynamically from the target data set names using the BACKUPGROUP= or BACKUPINDEX= operands to provide user control of the backup data set name. All backup data sets are cataloged after they have been created and are uncataloged and deleted after the reload has completed if the backups are not to be kept. FDRREORG provides last tape support on a job name basis for users who wish to have subsequent executions of the same REORG job append new backups to the last tape used in the last execution of FDRREORG.

FDRREORG provides complete dynamic allocation of all target and backup data sets. There are numerous operands provided to allow the user to control the type and number of devices allocated by FDRREORG. Backups on tape are automatically stacked up to the user specified MAXFILE= value.

FDRREORG has a multi-tasking capability that can be activated by providing selection criteria that allows multiple volumes to be processed. Each volume selected for processing is processed by a separate sub-task. Up to 15 concurrent sub-tasks can be active provided there is sufficient virtual storage available within the region. To minimize the below 16M virtual storage requirements, VSAM buffers and control blocks are allocated above the 16M. IAM provides this capability automatically for the backup and provides it for the reload if BSAM mode has not been specified for IAM file load processing via the IAM global option table. QSAM buffers used for the backup are also allocated above the 16M line if possible.

In order to minimize the manual effort required to recover from a failure during FDRREORG processing, a checkpoint and log file are created to record information that simplifies the recovery of a failed reorganization. The checkpoint file records information on the current status of all active file reorganizations and the log file is used to record information about unsuccessful reloads. FDRREORG's RECOVER statement can dynamically allocate the checkpoint and log files and use the information in these files to complete reload processing for any or all of the data sets impacted by a system or FDRREORG failure.

FDRREORG performance

FDRREORG has been designed to provide capability and ease of use in reorganizing VSAM, IAM, and PDS data sets without sacrificing performance. FDRREORG can reorganize simultaneously 1 to 15 volumes (default set at 4). FDRREORG gives users intelligent choices in reorganization. These features plus the special functions detailed in the following paragraphs give FDRREORG unrivaled performance and capability.

VSAM performance

Normally, VSAM data sets must be deleted and defined before they can be reloaded by IDCAMS. FDRREORG does not need to delete and define most VSAM data sets even if they are marked NOREUSE. VSAM defaults to recovery on a load of a KSDS or AIX. This requires VSAM to pre-format each CA prior to loading records into the CA. FDRREORG eliminates the need to reformat the CA’s even if recovery is on.

Most important for performance, FDRREORG calculates the optimal buffers required to sequentially read and reload the file.

FDRREORG reduces the wall clock, CPU and I/O operations required to reorganize your VSAM files by 20 to 60%.

PDS performance

FDR uses a proprietary technique to compress PDS data sets that reduces the wall clock, CPU and DASD EXCPs to compress source libraries by 30 to 60% and load libraries by 80 to 98% as compared to IEBCOPY performing the compression. FDR is capable of compressing PDSs from many volumes without individual JCL and control statements.

“SYS1.” data sets

FDRREORG does not allow you to compress “SYS1.” PDS data sets directly. These data sets can be compressed with the REORG statement of FDRCOPY that does not look at the FDRREORG list of unmovable data sets (see FDRCOPY REORG Examples for FDRCOPY examples).

Parallel option

Many users have IAM and VSAM files that exceed 1 GB in size. Some IAM files exceed 10GB. Reorganizing large files can take a long time reducing their availability to online systems.

FDRREORG Parallel option backs up each volume of a multi-volume file concurrently to separate tape or DASD files (up to a specified maximum). For example, if a file resides on 4 DASD volumes the backup time can be reduced by 75%.

3380 to 3390

While FDRCOPY and FDRDSF can be used to move VSAM KSDS data sets from 3380 to 3390, FDRREORG can also be used to move VSAM KSDS files between device types with the added flexibility of changing the primary and secondary allocation quantities, in addition to changing the CI/CA free space and reorganizing the cluster during the move.

REORG basics

As shipped, FDRREORG has default criteria that selects those data sets that are in the most need of reorganization. Using the data set names or filters that you provide, FDRREORG automatically reorganizes the following data sets:

• VSAM KSDS data sets that have taken control interval or control area splits in 10% or more of the total control intervals or control areas respectively, and have a CA and CI freespace value that is less than 50%.
• IAM data sets that have used at least 90% of the independent overflow area (compatible format only), or that have at least 10% of the records in the file in the independent overflow area, or require more than 1 megabyte for the overflow index.
• Partitioned data sets that are at least 90% full.

These defaults can be permanently changed or disabled by using the FDRREORG option change utility FDRREOZO. You can also override these defaults at run time by providing your own values.

Using the appropriate SELECT operands, you can easily provide your own rules for selecting data sets for reorganization. We recommend that you use the SIMULATE statement before running a REORG for the first time, or after making changes to an existing set of selection criteria. SIMULATE produces a report of the data sets that would be selected for reorganization if the REORG statement been used. This report can be used to fine-tune your selection parameters so only the data sets you want are selected for reorganization. Keep in mind that SIMULATE does not test the availability of the data sets selected. It is possible that any number of the data sets that meet the selection criteria will be in use by other jobs when the REORG statement is run and not reorganized. FDRREORG performs a full file reload and must have exclusive control of a data set in order to reorganize it.

VSAM KSDS data sets and AIXs

Due to the nature of the insert strategy used by VSAM, KSDS data sets need to be reorganized from time to time. When records are inserted into a VSAM KSDS, there may be insufficient free space available in the control interval where the inserted record belongs. When this occurs, VSAM “splits” the control interval to insert the record. If there is insufficient free space in the control area to hold this split control interval, a control area split also occurs. Over time, and without periodic reorganization, these control interval and control area splits eventually cause a degradation in performance, and in the case of control area splits, cause a waste of valuable DASD space.

FDRREORG can be used to logically reorganize these data sets on an as needed basis. By using either the default selection criteria or your own, FDRREORG could be run on a daily basis against these files, and would only reorganize them when needed. This is a much more efficient use of valuable computer resources than simply scheduling file reorganization on a daily or weekly basis. With FDRREORG's approach, you can be certain that files are reorganized when they meet your specified criteria, instead of by the best guess method you are probably using today.

FDRREORG supports any VSAM KSDS that has been defined with a maximum record length of 32,763 or less. For VSAM KSDS data sets that are the base cluster of one or more Alternate Indexes (AIXs), FDRREORG deletes and defines the base cluster and all associated AIXs and Paths. After the base cluster has been reorganized, FDRREORG performs a fast build of all the related Alternate Indexes (AIXs).

Innovation Access Method

Innovation Access Method (IAM), BMC’s proprietary access method, provides a transparent replacement for many VSAM KSDS, RRDS, and ESDS applications. IAM provides improved performance, better space utilization, and built in data compression. FDRREORG completely reorganizes IAM files by placing all records from independent overflow and prime extension into prime data blocks. After reorganization, 100% of independent overflow and prime extension are available for inserts or adds.

By using the appropriate operands, you can re-size independent overflow and prime extension for files in compatible format. This might be desirable if the IAM file was originally created with a single or dummy record load. IAM treats single or dummy record loads as a special case and dynamically increases the size of independent overflow and prime extension depending on the key of the record loaded. If you completely reorganize an IAM file, retaining the generous amounts of independent overflow and prime extension is in all likelihood inappropriate. Sometimes the generous amounts of independent overflow or prime extension may have been provided when the file was defined because it was not clear how much of these areas would be required. In other cases files may have been defined with insufficient independent overflow or prime extension. FDRREORG allows you to dynamically adjust the size of independent overflow and prime extension for these files. By using the PCTOFKEEPSINGLE= operand for files created with a single record load, and the PCTOFKEEPMULTIPLE= operand for files not created with a single record load, you can specify the percentage of independent overflow to retain during reorganization. To dynamically increase the size of independent overflow, specify a value greater than 100. The PCTPEKEEPSINGLE= and PCTPEKEEPMULTIPLE= operands provide the same support for IAM's prime extension. IAM's file load has been enhanced to recognize that FDRREORG is re-loading the IAM file and accepts the temporary overrides to re-size independent overflow and prime extension. If the IAM file is subsequently re-loaded using whatever procedure is normally used to create this file, IAM reverts to the original sizes specified when the IAM file was defined. This ensures that your definition parameters are always preserved.

IAM files with Alternate Indexes (AIXs) can also be reorganized. Unlike native VSAM AIXs, IAM AIXs do not have to be rebuilt when the base file is reorganized by FDRREORG. FDRREORG can also reorganize IAM RRDS files.

IAM Release 09.03.02 SPIN=10, in conjunction with FDRREORG Release 5405.04.89 or higher, provides enhanced reorganization processing for IAM Extended Format Data Sets. The goal of these enhancements is to provide increased data availability for IAM data sets by reducing the time that the data is not available due to data set reorganizations. See Dynamic Reorganization of IAM Data Sets on the FDR website for more details. The new functions and features included with this enhancement are:

• Dynamic Reorganization process to perform the bulk of the reorganization process while the IAM data set remains completely available to CICS online transaction processing.
• Employs various techniques to enhance the performance of the reorganization process with use of the latest IBM Z features including 64-bit storage, z/Architecture instructions, and z/HPF I/O. Unlike typical reorganizations, that use an intermediary flat file, the data is written directly into a new IAM data set. An optional flat file copy of the data set can be concurrently created by this process for backup purposes and optional use of z/EDC to reduce the size.
• Eliminate the need to decompress and compress data for data sets that use any form of IAM data compression. While that has been available for IAM software compressed files, this is a new feature for hardware compressed files. This is a very significant improvement in both CPU time and clock time to reorganize hardware compressed IAM data sets.
• Export and Import functions are available that create a flat file image of the IAM data set that has the compressed data, the file attributes for subsequent import, and the hardware compression dictionary for the data set if applicable. This is an excellent alternative to using IDCAMS REPRO for application data set backups. The exported data set can be compressed with z/EDC to further reduce the amount of space needed for the backup copy.

Partitioned data sets

When existing members of a partitioned data sets are updated, the partitioned data set access method (BPAM) stores the updated member at the end of the data set. The space occupied by previous versions of these updated members become dead space and cannot be re-used unless the data set is compressed with a utility such as IEBCOPY. Over time, the PDS eventually exhausts all the remaining free space and has to be compressed before additional members can be added or updated members saved. FDRREORG dynamically compresses these data sets based on the default PDSFULL percentage of 90%, or using a value that you specify. If the PDS meets the specified PDSFULL percentage and the PDS has just been compressed, there is no difference in the amount of available free space after FDRREORG has compressed the data set.

Data set integrity during reorganization

To ensure that the data sets being reorganized cannot be updated while a reorganization is taking place, FDRREORG uses techniques that are appropriate for the data sets access method to lock out updates by other jobs. FDRREORG allocates all data sets using DISP=OLD.

Although this can be changed by using the IAMDISP=, PDSDISP=, or VSAMDISP= operands, we recommend that you allow FDRREORG to allocate using DISP=OLD.

For VSAM, FDRREORG alters the files share options to (1,3) and set the update inhibit flag before starting the backup. If another job has the data set opened for update, FDRREORG is unable to open the file. If this occurs, FDRREORG restores the file’s share options, turns off the update inhibit flag, and bypasses the data set. For IAM, FDRREORG issues IAM's enqueues before opening the data set. If exclusive control is not obtained, FDRREORG bypasses the data set. If you use GRS or CA MII (Multi-Image Integrity component of CA MIM), we recommend that all IAM and VSAM enqueues be propagated if the data sets selected for reorganization can be in use on other systems in your complex.

Recovering from a failed reorganization

FDRREORG creates a checkpoint and log file when a REORG statement is executed.

The data set names used for the checkpoint and log file contain the name of the job, the date the REORG statement was started, and the time the REORG statement was started. In the event of a sub-task abend, FDRREORG abend, or system failure, the checkpoint is kept and can be used with FDRREORG’s RECOVER statement. If none of these conditions occur, the checkpoint file is deleted by FDRREORG when the job step ends. The log file is kept if any file re-load failures have occurred. The log is also kept if a system failure or FDRREORG abend occurs, although it is possible that the log file is empty in this situation.

To actually recover from a failed REORG, execute FDRREORG with the RECOVER statement and code the JOBNAME= operand specifying the name of the job that failed. FDRREORG’s recovery processor allocates the most recent matching checkpoint and log file for the job name specified, and re-loads any data sets that had started the re-load phase without completing. Data sets that failed during the backup phase are not recovered because they are still loaded and usable.

Data set availability and the DSNRETRY operand

During the course of a reorganization run, it is possible that data sets selected for reorganization are in use by other jobs. In many cases, these data sets become available before FDRREORG completes. By using FDRREORG’s DSNRETRY= options, you can request that FDRREORG make additional attempts to reorganize these data sets. Depending on the importance you associate with reorganizing a particular data set or group of data sets, there are four options that can be specified to control retry processing.

• For those data sets that are only reorganized if they are available at the time FDRREORG first attempts to reorganize them, use DSNRETRY=NO (the default) on the SELECT statement.
• For those data sets that you wish to reorganize if they become available during the course of execution, use DSNRETRY=RETRY.
• To improve FDRREORG's chances of successfully allocating an unavailable data set, use DSNRETRY=ENQ. With this option, FDRREORG issues an enqueue for the data set. This data set is the next data set processed if it becomes available.
• For those data sets where reorganization is critical, use DSNRETRY=WAIT. In addition to issuing the enqueue for these data sets, FDRREORG waits for them to become available before terminating. If you use the WAIT option, we recommend that you specify the RUNTIME= or STOPTIME= operand on the REORG statement, or use an operator STOP (P) command, to prevent FDRREORG from waiting indefinitely.

DSNRETRY= is not supported in a JES3 installation; use the default of DSNRETRY=NO.

Backup data sets

In order for FDRREORG to reorganize VSAM and IAM data sets, it must create a logical backup of these data sets. This backup is used to reload the selected data set. The backup data sets created by FDRREORG are standard sequential data sets and can be used with IDCAMS REPRO (except for IAM RRDS data sets). If you are creating compressed IAM backups, you must use the BACKUPCOMPRESSED IAM override to use the backup with IDCAMS REPRO. All of the backup data sets have a record format of VBS and a logical record length of 32,767. The block size is an optimum block size for the backup device. The name of the backup data set is generated using the BACKUPGROUP= or BACKUPINDEX= specified on either the REORG or SELECT statement, or from the FDRREORG option table. To avoid having the backup data sets cataloged in your master catalog, be sure to define aliases for any high level indexes that you intend to use for FDRREORG backup data sets

By default, FDRREORG deletes the backup data set after the target data set has been successfully reorganized. If you would like to keep FDRREORG's backup data sets, code BACKUP=PERM or BACKUP=GDG on either the REORG or SELECT statement, or use the option change utility FDRREOZO to change the default. GDGs cannot be used with the parallel backup option. When PERM is used, the name generated by applying the BACKUPGROUP= or BACKUPINDEX= is left cataloged. You must provide some method of deleting these data sets if FDRREORG is to be run again against the same target data sets. FDRREORG is not able to reorganize a VSAM or IAM data set if a data set with the same name as the one generated for the backup data set already exists. The GDG option is probably more useful for retaining FDRREORG's backup data sets. With the GDG option, FDRREORG uses the name generated for the backup data set to define a Generation Data Group (GDG) (if one does not already exist), and creates the backup data sets as a +1 generation. When defining these Generation Data Groups (GDGs), FDRREORG uses NOEMPTY, SCRATCH, and LIMIT(5). If you prefer to use different parameters, you can use the GDGEMPTY, GDGNOSCRATCH, or GDGLIMIT= operands on the appropriate SELECT statement.

Last tape support

If you are using BACKUP=PERM or BACKUP=GDG and are creating your backup data sets on tape, you can direct FDRREORG to append new backups in subsequent runs to the same tapes. To do so, specify LASTAPE on the REORG statement. FDRREORG creates a catalog entry on a job name and sub-task basis that is used in next run of the same job to reuse these tapes. The data set name is generated using either the default high level prefix in the FDRREORG option table that defaults to FDRREORG, or the prefix specified with the LASTAPEPREFIX= operand, followed by the job name, TASKnn where “nn” is the taskid, and finally LASTAPE. The catalog entry also contains the tape volume and the next file sequence number that is used for this tape volume. In the event that the file sequence number of the last file created is either 255 or the same as the value specified with the MAXFILE= operand, no catalog entry is created. This feature is not supported with parallel backups.

Error handling

FDRREORG has been designed to continue processing if an error occurs while reorganizing an individual data set. FDRREORG makes a distinction between environmental errors, system errors, and fatal internal errors. For a fatal internal error, FDRREORG allows any in progress reorganizations to complete if possible, and terminates. For environmental and system errors, FDRREORG by default, allows up to 99 of both types to occur before shutting down. Please keep in mind that this discussion applies to errors encountered while backing up and reloading data sets only. Any abends that occur in FDRREORG's main task results in an immediate abnormal termination. Similarly, any abends that occur in FDRREORG's volume processor tasks result in the immediate abnormal termination of that task. The remaining volume processor tasks continue to run, although resources allocated to the failed task may not have been released.

To allow you to control how FDRREORG handles these errors, a number of operands have been provided. For a volume processor sub-task abend, FDRREORG by default attempts to complete the active REORG function without the abended task and terminate. If there are additional REORG statements, they are not processed. In addition, any data sets that would have been selected for reorganization on the volume being processed by the abended task is not selected by the remaining tasks. If you prefer that FDRREORG shutdown when a volume processor sub-task abends, specify SUBTASKABEND=TERM on the REORG statement. FDRREORG allows all active reorganizations to complete and then terminate.

Environmental errors are categorized as errors related to the current availability of resources required to reorganize a single data set. Insufficient space during allocation of backup data sets is an example of an environmental error. FDRREORG allows up to 99 environmental errors. You can use the MAXENVERR= operand on the REORG statement to change this value.

System errors are categorized as any system abend or as any error that prevents FDRREORG from completing a reorganization. I/O errors, GETMAIN failures, and other system service failures are examples of system errors. FDRREORG allow up 99 system errors. You can use the MAXSYSERR= operand on the REORG statement to change this value.

Cancel protection

Never cancel FDRREORG. Cancellation could result in data corruption or loss.

To protect against this, FDRREORG has automatic CANCEL protection. If the system operator or a user tries to cancel an FDRREORG step, FDRREORG intercepts the CANCEL and issues an FDRW99 message to the system operator asking for instructions. The operator can reply:

I

To ignore the CANCEL and continue processing.

S

To stop after completing the data sets currently being reorganized.

C

To accept the CANCEL and terminate immediately. This may result in data corruption or loss.

The operator needs to be instructed to always respond “S” to the FDRW99 message, and FDRREORG stops after processing the data set it is currently working on.

IFANY and IFALL operands

In most cases, you want a data set excluded from or selected for reorganization if any of the appropriate thresholds are met or exceeded. For those situations where a stringent exclusion or selection process is desired, you may want all of the appropriate thresholds to be met or exceeded. The IFANY and IFALL operands can be used to modify FDRREORG's evaluation of the following thresholds in just such a manner:

By default, FDRREORG evaluates these thresholds as if the IFANY operand was specified on the EXCLUDE or SELECT statement. If you want all of these thresholds to be met or exceeded before a data set is excluded or selected, use the IFALL operand on the EXCLUDE or SELECT statement. If you specify any other criteria on your EXCLUDE or SELECT statement, they all have to be satisfied as well.

DSN= and CATDSN= Differences

Although the end result is usually the same, your choice in using the DSN= versus the CATDSN= operand can significantly impact FDRREORG’s processing. Choosing between these two operands is actually simple once you understand the differences in FDRREORG’s behavior when you choose one over the other.

The most significant difference between DSN= and CATDSN= relates to partitioned data sets. If you use CATDSN=, uncataloged PDSs are never selected. This does not apply to IAM or VSAM data sets because they must always be cataloged.

When CATDSN= is used, FDRREORG scans the appropriate catalogs when the control cards are parsed and limits its volume processing to the volumes that match the filter or names you specify. Use of the VOL= operand is optional and if specified is used to restrict the data sets to the volumes or volume groups you provide. The remaining selection criteria is not evaluated until the volume is processed during the actual simulate or reorganization phase. Care needs to be taken when using the data set filtering capabilities because the data set names are kept in storage throughout SIMulate or REORG processing. Avoid coding statements such as CATDSN=** without a restrictive volume list. Doing so would cause a large data set list to be built that would most probably result in an abnormal termination of FDRREORG. CATDSN=** also causes FDRREORG to read every available catalog which could take a considerable amount of time if there are a large number of user catalogs.

When DSN= is used, a catalog scan does not occur. FDRREORG performs most of the exclusion and selection process when reading the volumes VTOC for IAM and PDSs, and the VVDS for VSAM. Catalog locates are issued for IAM and VSAM data sets for additional information after the data set and volume filtering is done. Because the primary source of information is the VTOC or VVDS, the VOL= operand must be provided. Every volume that matches a volume or volume group in the volume list is processed.

As a rule, use the CATDSN= operand when the number of data sets that would be returned from the catalog scan is small, or the volume naming conventions in use at your installation do not translate well into volume groups. Use the DSN= operand when many data sets could be returned from the catalog scan, or you are more interested in processing particular volumes as opposed to specific data sets. If you have uncataloged partitioned data sets that you want to reorganize, you must use the DSN= operand.

Sample FDRREORG output of PO data sets

Sample report compressing PDS data sets using the catalog and selecting all PDS data sets on TSO volumes.

Sample FDRREORG output of VSAM data sets

Sample Report of reorganizing VSAM data sets. The backup is created as a GDG. Only data sets with either CI or CA split ratios greater than 4 are reorganized.

Sample FDRREORG output of IAM data sets

Sample Report of reorganizing IAM data sets. Data sets in this application starting with PROD.BILLING are selected based on defaults of FDRREORG of either having 80% of the overflow area or 100% of prime extension used.