IAM/RLS record lock recovery

Record lock recovery enables IAM/RLS to reestablish the record locks for recoverable files when an IAM/RLS address space is restarted after a shut down or failure. This function enhances the data integrity by providing persistent record locking within the IAM/RLS sphere, which will enable recovery of the work that was in process at the time of the incident before any other attempts to update those records through IAM/RLS. Using a two-phase recovery process, as explained below, IAM will re-establish the record locks as they were at the time of failure based on data from both the record lock journal and the IAM/RLS System Logger journal.

Record lock recovery process

The record lock recovery process retains information about the record locks that are currently held on disk in an IAM data set that is referred to as the Record Lock Journal. The presence of the record kock journal in the IAM/RLS JCL is the indication that the record lock recovery process is to be activated. Each IAM/RLS address space will need to have its own record lock journal data set. To minimize the performance impact, the I/O on the data set utilizes deferred writes which delays the actual physical writing of the record lock status. At periodic intervals, referred to as the record lock checkpoints, all the buffers containing updated record lock data are written out to DASD and the current position in the IAM/RLS journal is recorded.

The checkpoint interval can be set through the IAM/RLS parameter LOCKCHKPT, which will specify the checkpoint interval in minutes. The default if the LOCKCHKPT parameter is not provided is 5 minutes. Using shorter times may increase the overhead during regular processing but decrease the recovery time. Using longer checkpoints will have the opposite effect.

The actual record lock recovery process is done in two phases while the IAM/RLS address space is starting up. The record lock recovery must complete before the IAM/RLS address space will start processing I/O requests. The first phase is to read through the Record Lock Journal data set and establish the record locks based on the information from that data set. If the status indicates that the IAM/RLS address space had been shut down properly, meaning that the record lock journal has all of the lock data, then the recovery is complete.

If the IAM/RLS address space was not normally shut down, then the record lock recovery process will go into the second phase. The second phase will read the IAM/RLS journal - either the sequential journal data sets or the System Logger can be used. Positioning is established at the place in the journal where the last record lock checkpoint had occurred, then proceed forward in the journal from that point. The recovery process will then update the record locks in the record lock journal based on the activity indicated by the journal records.

It is recommended that users backup the record lock journal prior to restarting the IAM/RLS address space from a failure. This can be done with a copy, a backup with FDRDSF, FDRABR or similar product, or by an IDCAMS REPRO. If a failure should occur on the restart of the IAM/RLS address space, one will then be able to restore the record lock journal and retry the restart process again, otherwise a cold start may be necessary.

On completion of the record lock recovery, a summary of the activity performed by the lock recovery process will be printed on the RLSLOGDD data set. Then the IAM/RLS address space will finish initialization and start accepting requests. To find out what record locks have been recovered one can either use the IAM/RLS display retained locks command or use the IAMBMON facility under the IAM ISPF panels. If any of the work is not going to be recovered, the associated locks can be released using the IAM/RLS release locks command.

Allocating the record lock journal

To use the record lock recovery function, a record lock journal data set must be pre-allocated using an IDCAMS DEFINE prior to use for each IAM/RLS address space that record lock recovery is going to be used. The record lock journal data set contains a control record, lock owner records, and record lock records. The data set will be formatted when it is first used based on the IAM/RLS parameters of MAXJOBS, MAXLOCKS, and MAXTRANS. Below is an example of the JCL and control cards that are used to define the record lock journal data set:

Example of JCL to define record lock journal, member LKRCVJRN in IAMSAMP

//DEFRLKJR EXEC PGM=IDCAMS
//IAMPRINT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//IAMOVRID DD *
CREATE DD=&ALLDD,DATACOMP=NO,DSNTYPE=LARGE,RELEASE=NO
//SYSIN     DD *
DELETE (IAMV.RLOKJRNL.rlsid.CLUSTER)
DELETE (IAMV.RLOKJRNL.rlsid.CLUSTER) NOSCRATCH
SET MAXCC=0
/* RECOMMENDED SPACE ALLOCATION VALUES:                */
/* CYL( 60       6) WITH MAXLOCKS UP TO   133,103    */
/* CYL( 120      12) WITH MAXLOCKS UP TO   266,239    */
/* CYL( 230      23) WITH MAXLOCKS UP TO   532,453    */
/* CYL( 460      46) WITH MAXLOCKS UP TO 1,064,957    */
/* CYL( 900      90) WITH MAXLOCKS UP TO 2,129,903    */
/* CYL(1350     135) WITH MAXLOCKS UP TO 3,193,873    */
/* CYL(1800     180) WITH MAXLOCKS UP TO 4,194,301    */
DEFINE CLUSTER -
       (NAME(IAMV.RLOKJRNL.rlsid.CLUSTER) -
       CYL( 900 90)     -
       VOL(* * * )       -
       CISZ(12288) REUSE -
       STORCLAS(TSTDATA) -
       OWNER($IAM)      -
       RECORDSIZE(100 328) KEYS(5,0) -
       FREESPACE(0,0) -
       SHAREOPTIONS(2,3))
LISTC ENTRIES(IAMV.RLOKJRNL.rlsid.CLUSTER) ALL
/*

When using the above JCL, users will need to change the data set name and the STORCLAS. Also the space requested should be updated based on the MAXLOCKS value they are using. Please note that one needs to use the next larger value from the table if the MAXLOCKS value chosen exceeds the prior value. For example if MAXLOCKS=133104 is specified, then the space allocation used must be the one for up to 266,239 lock records. All of the other parameters and overrides shown in the example above must be used to provide proper functioning and performance of the lock recovery function. If the value for MAXLOCKS is ever increased, the record lock journal should be re-allocated with the required space values indicated.

Initializing the record lock journal

After defining the record lock journal, users must then make the following changes to the IAM/RLS JCL:

Add the RLOKJRNL DD statement to the IAM/RLS JCL specifying the newly defined data set.
Add a card in the IAMOVRID member input that specifies the following:
ACCESS DD=RLOKJRNL,NORLS,JRNAD=NONE

When the IAM/RLS address space is started, IAM/RLS will automatically determine that the data set is unloaded and format the data set with the appropriate number of pre-formatted records. This pre-formatting is completed quickly and will provide performance benefits for the operation of the lock recovery function. If any of the key IAM/RLS parameters are changed, that is MAXLOCKS, MAXJOBS, or MAXTRANS, then the data set should be re-initialized by specifying the IAM/RLS parameter of LOCKSTART=COLD which will trigger the reformatting of the data set. If the value for MAXLOCKS has been increased then the data set may need to be deleted and re-defined with the appropriate amount of space.

Normal operation of record lock journal

It is highly recommended that a backup copy of the record lock journal be taken whenever the IAM/RLS has come down, whether it was scheduled or unscheduled. The data set can be backed up with a copy, FDRDSF, FDRABR or similar type of backup, or by an IDCAMS REPRO. Then if there is a problem on the restart, the record lock journal can be restored prior to retrying the restart.

Backups can be taken at other times while the IAM/RLS address space is running, however, the restored copies can not really be used to restart the IAM/RLS for other than a cold start (LOCKSTART=COLD). This is because the restored copy is far from being synchronized with the actual state when the IAM/RLS address space had terminated. A cold start is also likely to be required when starting IAM/RLS address spaces at disaster recovery sites.

There generally should not be any need to reorganize the record lock journal. The define parameters and overrides in the example above were carefully chosen along with the way the data set is processed to minimize the need for use of the overflow area. This is particularly true, when appropriate values were specified on the IAM/RLS parameters such that they were sufficient to meet the processing that was done. One can either look at a LISTCAT output or at the IAMINFO report to determine if there are a lot of records in overflow. If there are, then one can check on the maximum locks used from the RLSINFO report that is produced or from the IAMBMON display to make sure that sufficient lock records had been formatted. One should also check that all of the appropriate parameters in the define example above were used. It may be necessary to delete and redefine the data set such that the overflow area usage will be minimized. BMC technical support can provide assistance if needed.