IAM/PLEX Batch Considerations
Overview
The ability to maintain and achieve data integrity can be a complex matter when dealing with non-transactional processing, as is the case with many batch applications. Such batch applications range from performing a relatively simple and straightforward update process on one or only a few files, to very complex processing involving updates to a large number of related data sets and databases. Additionally, such applications may not have been designed or written with record level data sharing as a consideration. While software products, such as IAM/PLEX, can support and facilitate data sharing, there may be a need to change some batch applications to facilitate good performance and data availability in a data sharing environment.
The discussion of IAM/PLEX and batch processing below is intended to provide assistance in determining what, if any, changes may be necessary to the batch job streams that will be processing files managed by IAM/PLEX. The main considerations that should be kept in mind are what are the recovery requirements for the files being processed by the batch jobs and what other concurrent activity is likely to be occurring on the shared files while the batch jobs are executing. One of the key points is that with concurrent updating going on, you cannot just restore all of the files that were updated by the batch job and rerun it, because updates made by the other concurrent processing will then be lost. Different recovery techniques will have to be used.
IAM/PLEX and Batch Jobs
IAM/PLEX can be used with batch jobs that read and update IAM files. With IAM/PLEX automatic data set selection, there are no changes required to JCL for batch jobs to use IAMfiles under IAM/PLEX. The main area that requires some consideration is for those batch jobs that update recoverable IAM files with IAM/PLEX. For the purpose of this discussion, recoverable IAM files refers to those files that are utilizing the IAM/PLEX before image journaling. Because such files can have updates backed out should the batch job fail, the record locks are retained until the batch job step ends normally, until a batch syncpoint call or automatic syncpoint is performed, or if the job step fails, until a recovery is performed.
A job step failure from the IAM/PLEX viewpoint is an abend. IAM/PLEX does not look at application return codes, or the condition codes set for a particular job step. If the job step abends, it is considered a failure. If a job step terminates without abending, then IAM/PLEX considers it successful. This is significant because if a recovery is needed, a valid recovery would consist of only recovering to either the last syncpoint, if any, or to the beginning of the job step if no syncpoints were taken. Attempting to backout updates performed by prior job steps may cause a loss of data, because other jobs or online systems may have updated those records subsequent to their completion. Those updates would be lost if a recovery is performed for job steps other than the failing job step. Likewise, attempting to backout updates for a job step the application considers failed due to condition codes could also result in data loss. Backing out of all of the updates for such a job could only be safely performed if the only update access was being performed by the failing job. This can only be guaranteed if the failing job allocates the IAM data sets with DISP=OLD. Otherwise, there is nothing to prevent the IAM files from being updated by other jobs or online systems.
Batch jobs that only read shared IAM files can be executed under IAM/PLEX generally without any JCL or programming changes. Batch job steps that run relatively quickly, say in less than 15 to 30 minutes, and that update a relatively small portion of the records in the data set, say less than 10%, and that acquire record locks that do not exceed the MAXLOCKS value should be able to use IAM/PLEX recoverable files without any changes.
Batch jobs that update IAM/PLEX files, which have long run times or update a large portion of the records in recoverable files may need to be changed, particularly if they are running concurrently with online systems or other batch jobs that are also updating the same files.
Batch Syncpoint Processing
IAM/PLEX has a syncpoint capability that can be used by batch applications. When recoverable files are being updated, the record locks are retained for records that are updated, added, or deleted until the job step terminates. If many records are being updated, then the number of locks held could be quite high, resulting in effectively locking out CICS transactions until the batch job completes. Such lock retention could also result in deadlocks between CICS and the batch job, resulting in failed requests. To prevent these types of problems, customers are encouraged to consider the use of the IAM batch syncpoint for such programs. Customers with other VSAM sharing packages may already have such logic built into their batch programs, and will only need to change the syncpoint program they are calling.
The IAM batch syncpoint service will perform a few different functions. First, it will make sure that all updated blocks for IAM files that are being processed by IAM/PLEX for this job have been physically written to DASD. If blocks have not yet been written, such output I/O will be performed. Upon completion of any outstanding buffer writes, the batch syncpoint will then write out a syncpoint record in the journal, that can be used if a back out recovery is necessary. Then, any record locks held by that batch job will be released.
Implementing Batch Syncpoints
The amount of effort to implement a syncpoint within a batch job, along with the associated restart processing may be minimal, or may be rather substantial. This all depends on the nature of the updates being performed, the restart capabilities that the application program may already have, and how other updated files are handled. There are certain types of updates for which an application program can simply be restarted, and it can reapply those updates without the need for any recovery, and retain full data integrity. An example of this could be a job that updates addresses of customers. Such updates may not matter if they were already done before, the program will just end up repeating the update with the same data, or the application may have a way to check to see if the record was previously updated, and if so it would not have to reapply the update. For these types of programs, all that would need to be done is to periodically call the IAM batch syncpoint to release the locks of records updated to that point in time, or use the automatic syncpoint function.
For other applications, such as those that update account balances, a more complex approach may be necessary. These types of programs will need to implement a more complex syncpoint and restart mechanism, such that the program can be restarted from a specified point in time, and be able to resume processing from that syncpoint. When an application is able to do that, it can issue IAM batch syncpoints. If it needs to be restarted, back outs can be performed to for example the most recent syncpoint and the job will not perform any double updates. The batch application program will in effect, have to perform a processing checkpoint, and include in that the execution of the IAM batch syncpoint. The application will have to implement their own type of checkpoint processing, as z/OS Checkpoint cannot be used withIAM files being processed under IAM/PLEX.
Calling theIAM Batch syncpoint is easy. Just issue a call to IAMSYNC. No parameters are necessary. The program IAMSYNC is a small stub, and designed to be link-edited in with the application program. An alternative is for the application to LOAD the IAMSYNC module, saving the address and calling it as necessary.
Automatic Syncpoints
As an alternative to revising the actual program, IAM/PLEX has an automatic syncpoint capability. This ability is enabled by use of the IAM ACCESS override of AUTOSYNCPOINT=nnnnn. This override is specified on the job step that is updating IAM files through IAM/PLEX for a selected key file that is processed using IAM/PLEX. The AUTOSYNCPOINT specifies that after the specified number of I/O requests that result in record lock being acquired are performed, IAM will automatically invoke the batch syncpoint process. This facility is useful for circumstances where it is not possible to change the application programs.