IAM/PLEX Record Level Sharing

Overview

IAM/PLEX Record Level Sharing (RLS) is an optional additional cost feature of the IAM product. With IAM/PLEX, batch jobs, TSO users, and CICS regions concurrently running on multiple LPARs within a SYSPLEX can safely update shared IAM files. An IAM/PLEX is activated by defining an IAM/PLEX address space with a matching RLSGROUP on each LPAR that is participating with the IAMfile sharing. The goal of IAM/PLEX is to provide performance along with ease of use in a multiple LPAR data sharing environment. IAM/PLEX is different in design and implementation than VSAM/RLS. In an IAM/PLEX environment, only one IAM/PLEX instance will perform the actual I/O processing for any specific IAM file. The requests are passed to the target IAM/PLEX instance using SYSPLEX XCF communication. This eliminates the need for locking and caching structures in a Coupling Facility, and the potential for multiple interactions with the CF for one logical I/O request.

Base requirements for IAM/PLEX usage include multiple z/OS systems and a coupling facility. For optimal performance the use of Channel-To-Channel (CTC) type connections are recommended for XCF communication.

The intention of the design of IAM/PLEX is to provide high availability of data access. IAM/PLEX does not guarantee continuous availability or eliminate single points of failure.

IAM/PLEX Capabilities

IAM/PLEX provides the following capabilities:

Ability to share IAM files for update with data integrity between concurrently executing batch jobs, TSO users, and CICS regions across multiple LPARs in a SYSPLEX.
Locking during update processing is performed at the record level, utilizing the IAM/PLEX record locking function. Record locks can be reestablished after a failure or shutdown of an IAM/PLEX instance to facilitate recovery processing.
For recoverable files, record locks are held until the job or transaction terminates, or the job or transaction issues a SYNCPOINT.
Potential deadlocks within the resources managed by each individual IAM/PLEX address space are detected, and the associated request will be failed to prevent a deadly embrace.
Journaling and recovery capabilities are provided for data sets processed under IAM/PLEX and optionally for IAM data sets not processed under IAM/PLEX. Journaling in the IAM/PLEX RLS environment requires using the z/OS System Logger. Ability to dynamically back out updates made to files processed by IAM/PLEX when the batch job step that performed those updates abends.
A callable batch syncpoint process is provided to prevent batch programs from locking out access to large portions of recoverable files when mass updates are being performed.
An AUTOSYNCPOINT feature that will automatically issue syncpoints without changes to the application can be used to avoid changing batch application programs when syncpoints are required.
An Automatic Disconnect and Reconnect capability within CICS is provided to dynamically disconnect and reconnect to an IAM/PLEX instance that has terminated and subsequently has been restarted. Any pending in flight updates by CICS transactions to IAM files owned by that IAM/PLEX instance at the time of its termination will be scheduled for backout. However, since the IAM files are not available, any backout to an unavailable IAM file will fail. CICS then marks such failed backouts as pending Shunted Unit-of-Work (UOWs). Subsequently, when the terminated IAM/PLEX instance becomes available again, IAM will cause the shunted UOWs to be retried. Then IAM sets the files within CICS as being open and available for normal processing as IAM will have then fully reconnected to the restarted IAM/PLEX instance. Additionally, the CICS region will not have to be restarted or recycled as in prior IAM releases.
An ISPF interface to monitor activity within the IAM/PLEX address space, and issue some of the available operator commands.
Depending on the RLS selection criteria implemented at an installation, there generally will be little or no JCL changes required for IAM/PLEX. For CICS, the installation and activation of certain exits is required, as documented in the IAM/PLEX CICS Considerations in IAM-PLEX-CICS-Considerations. For batch jobs that perform a large volume of updates to recoverable files being processed by IAM/PLEX, the implementation of the IAM Batch Syncpoint either with the AUTOSYNCPOINT feature or through program changes and appropriate restart capabilities is necessary. Recoverable files are those for which image journaling is being performed.

IAM/PLEX Definitions

To help explain how IAM/PLEX works and how it is used we will start with some definitions of the terminology that is used when discussing IAM/PLEX. The terms are defined in more of a logical sequence than an alphabetic sequence

RLSID: This is a unique 4-character identifier used to name each IAM/PLEX and IAM/RLS address space that is active within a SYSPLEX. It is specified in the initialization parameters for each IAM/PLEX and IAM/RLS address space via the RLSID= parameter, example RLSID=RLS1. Each data set that will be processed by an IAM/PLEX or IAM/RLS address space is assigned to an RLSID through the IAM/PLEX data set name table, IAM overrides or defaulted to from the IAM Global Options Table to indicate which IAM/PLEX or IAM/RLS address space that will perform all of the I/O for that data set.

IAM/PLEX Instance: Refers to an IAM/PLEX address space. Each IAM/PLEX instance is assigned a 4-character identifier called the RLSID as described above. Any particular application address space (that is, batch job, CICS region) can directly connect to only one IAM/PLEX instance, through which it can access IAM files managed by other IAM/PLEX instances that are members of the same RLSGROUP. Each IAM/PLEX instance by default can perform both ROUTER and TARGET functions.

RLSGROUP: This refers to a group of related IAM/PLEX instances that form an XCF group, and therefore are able to communicate with each other. Any particular application address space (CICS region, batch job) can be connected to only one RLSGROUP, and can only access those IAM data sets that are being handled by an IAM/PLEX instance within that RLSGROUP. If no RLSGROUP is specified, the IAM/PLEX instance effectively runs as an IAM/RLS instance.

ROUTER: Refers to the IAM/PLEX instance that an application (that is, batch job or CICS address space) is directly connected to that being used to send (route) I/O requests for a particular file to another IAM/PLEX instance that typically would reside on a different LPAR within the SYSPLEX. For any specific RLSGROUP there will be only one ROUTER IAM/PLEX instance per LPAR.

TARGET: Refers to the IAM/PLEX instance that is processing the I/O requests and handling record locking for a particular set of IAM data sets. The TARGET receives the requests from an application address space through a ROUTER IAM/PLEX instance. The TARGET IAM/PLEX instance is typically on a different LPAR than the ROUTER IAM/PLEX instance.

REMOTERLS: This attribute refers to an IAM/PLEX instance that performs file I/O only, it does not have any direct connections with application address spaces. It is an IAM/PLEX instance that can operate only as a TARGET. This attribute is given to IAM/PLEX instances that are running on the same LPAR as another IAM/PLEX address space that is a member of the same RLSGROUP.

IAM/PLEX Processing

IAM/PLEX instances will connect to other IAM/PLEX instances that are members of the same RLSGROUP using SYSPLEX XCF services. Each IAM/PLEX instance within a SYSPLEX is uniquely identified by a 4-character identifier called the RLSID. On any particular LPAR, there can be only one IAM/PLEX instance that can have ROUTER capabilities per RLSGROUP You can start more IAM/PLEX instances for the same RLSGROUP on the same LPAR using the REMOTERLS parameter during startup. If there is already a ROUTER IAM/PLEX region for the RLSGROUP on the LPAR, then the new one being started will default to REMOTERLS. The REMOTERLS address spaces will serve as TARGET-only IAM/PLEX address spaces in the RLSGROUP. Each IAM/PLEX instance continually monitors through the z/OS XCF facility the status of the other IAM/PLEX instances, that are members of the same RLSGROUP. Each of the IAM/PLEX instances establishes a direct logical XCF connection with all of the other IAM/PLEX address spaces that are members of the same RLSGROUP.

An application address space (that is, batch job step or CICS region) will create a connection with an IAM/PLEX instance when the first IAM data set that requires IAM/PLEX processing is opened. The connection is made to the IAM/PLEX instance with routing capabilities that is a member of the RLSGROUP containing the IAM/PLEX instance that is required to process the data set being opened, which is the one with the matching RLSID. All IAM files that are subsequently opened by that application must belong to an RLSID that is being handled by this RLSGROUP. IAM files will be opened only in the IAM/PLEX instance that has the RLSID that was specified when the file was defined, loaded, via override, or entered in the DATASET NAME TABLE. The open request and any subsequent I/O requests will be sent to the TARGET IAM/PLEX instance of the LPAR it resides in the SYSPLEX. On the next page is an example of an IAM/PLEX configuration.

IAM/PLEX Diagram

Below is a diagram of a basic 2-LPAR PLEX running one IAM/PLEX instance on each LPAR.

In the above diagram there is one RLSGROUP named RLSPROD, which has two IAM/PLEX instances named RLS1 and RLS2. Each instance is running on a separate LPAR. For example. let us say that CICS-1 wants to read a record for update in IAM DSN=PROD.B, which is owned by IAM/PLEX_RLS2. CICS-1 issues a standard read for update, which IAM processes and sends the request from IAM/PLEX_RLS1 to IAM/PLEX_RLS2 using XCF services. IAM/PLEX_RLS2 will obtain the record lock, retrieve the record, then send the record through XCF back to the CICS-1 address space through IAM/PLEX_RLS1. In this scenario, IAM/PLEX_RLS1 is known as the router, and IAM/PLEX_RLS2 is the target.

Data Set Name Table

The Data set Name Table provides the capability to group data sets by RLSID. A single Data set Name Table must be used by all IAM/PLEX instances running in the SYSPLEX. The first IAM/PLEX instance that is started on an LPAR will read the data set name table data set and build the table in storage. All subsequent IAM/PLEX instances on the same LPAR will use the table already in storage, even the IAM/RLS address spaces not running in IAM/PLEX mode. If a change is required to the table, the CHANGEDSNT command can be used which will signal all the IAM/PLEX instances in the group to refresh their copy of the table. If an IAM/RLS address space running in single system or non IAM/PLEX mode requires the new changes to the table, then the CHANGEDSNT command will have to be entered specifically for that IAM/RLS.

Record Locking

IAM utilizes its own record level lock manager. Records are locked by their key within each IAM data set being processed by IAM/PLEX. The record locks are maintained in 64-bit virtual storage in the IAM/PLEX address space that is responsible for the data set containing the record. The IAM lock manager does check for potential deadlocks within the scope of the IAM data sets that it is managing. If waiting on the record lock for the current request will result in a deadlock, the request will be failed with a logical error. The trigger for IAM to release a record lock will depend on the environment in which the lock was requested.

For CICS transactions, records are locked by transaction and unit of work (UOW) identification. IAM determines from CICS which IAM files are recoverable, and will hold the record locks until either a SYNCPOINT is executed, or the transaction ends. If the IAM file is defined with the explicit option of JRNAD=NONE or the IDCAMS parameter LOG(NONE), then IAM will assume that the file is not recoverable under CICS. When an IAM file is detected as being not recoverable under CICS, then the record lock is only held from the time of the GET for UPDATE until the record is actually updated or erased, or for records being added, only for the duration of the actual add processing.

For other than CICS processing, that is batch jobs or TSO users, IAM will generally only hold the record locks from the time of the GET for UPDATE until the record is actually updated or erased, or for records being added, only for the duration of the actual add processing. The exception to this is if IAM/PLEX is journaling before images of records, implying that a back out could be performed if there is a failure, then the record locks will be held until the program either calls the IAM batch syncpoint, an AUTOSYNCPOINT is taken, or the job step terminates. If a job step abends while processing a recoverable file, then any record locks obtained for that job step will be retained until a recovery takes place. If there were any other jobs or CICS transactions waiting for the record locks that are being retained until recovery, those requests will be failed when the abend has been detected. The recovery can be performed by IAM’s Dynamic Job Back Out function, by IAMJREST, or by whatever other procedures or recovery software that may be available. Information about any retained locks can be found by using the IAM/PLEX operator command DISPLAY,RETAINEDLOCKS, described in IAM-PLEX-Operator-Commands. Such retained locks will be released upon successful recovery by IAMJREST or Dynamic Job Back Out. The retained locks can also be released by the RELEASELOCKS command.

Record Lock Recovery

IAM/PLEX has an optional capability to re-establish record locks on restart of an IAM/PLEX address space after either an unplanned or a planned outage. Only the record locks that are for recoverable IAM data sets will be re-established. The record lock status for locks eligible for recovery, are kept in a separate lock recovery data set. To minimize the performance impact, the records will only be periodically written to disk after a specified lock check point time interval.

When the IAM/PLEX address space is started, the record lock recovery process will read the lock recovery data set to re-establish record locks. If the last status was not from a proper shut down then the IAM/PLEX journals will be read from the time of the last lock checkpoint to update the record locks from the recorded activity. Once the record locks are re-established, then the IAM/PLEX address space is ready to resume processing.

Journaling

In addition to handling the I/O and locking services, the IAM/PLEX address space can also be used for journaling before and / or after images of updated records. When using journaling with IAM/PLEX the System Logger must be used and the LOGSTREAM must be defined to a structure in the coupling facility. The IAM journaling services are primarily provided to allow for the back out of updates performed by failing batch jobs, for record lock recovery, or to perform a forward recovery of updates if a file has encountered media damage. CICS will handle its own transaction back out and other recovery as it does today, using its own logging mechanisms that is independent of the journaling provided by IAM. Therefore, unless either record lock recovery is active or journaling of CICS transactions is explicitly requested, the before images for files processed under CICS will not be written to the IAM/PLEX journal. To request IAM/PLEX to journal the before images from CICS transactions users can specify the IAM/PLEX startup parameter CICSJOURNAL=YES.

Information and instructions on setting up IAM/PLEX journaling is presented in IAM-PLEX-Journaling.

Replication Logging

If IAM/RLS is setup to journal to the z/OS System Logger, then the journal records produced will be in the same format written by CICS in the CICS journals to support the use of IBM replication software such as Infosphere CDC.

Dynamic Job Backout

A Dynamic Job Back Out function for IAM files opened through IAM/PLEX is available. Whenever a job step abends, all updates done by that job step can be automatically removed by Dynamic Job Backout. Control of Dynamic Job Back Out is provided through IAM/PLEX DJB startup parameter. If permitted by the startup parameter, DJB can also be controlled via an IAM ACCESS Override. If the batch job step or TSO user has taken IAM syncpoints, then the backout will be performed to the most recent syncpoint taken by the job step prior to abending. If no IAM syncpoints have been taken, then all of the updates performed by the failing job step will be removed on files accessed through IAM/PLEX. Upon successful completion of the Dynamic Job Backout, all of the retained locks for the failing job will be released.

Security

The IAM/PLEX address space will have to be given the appropriate security authority for the IAM files that are going to be processed by IAM/PLEX Record Level Sharing. This will include CREATE authority for files that can be extended to additional volumes. IAM does issue the RACROUTE macro within the individual job’s address space to validate that the requesting user does have authority to read or update as appropriate while the IAM data set is being opened prior to requesting that IAM/PLEX open the data set. If the RACROUTE indicates that the user or job does not have authority to access the data set, then the OPEN request is failed. The failing job will receive an IAMW18 error message. Additional security will be performed by z/OS services when IAM attempts to take an extent

Reliability

IAM/PLEX utilizes the various z/OS error handling and recovery facilities to recover from errors and abend conditions that may occur. The two goals in providing error recovery routines are to provide continuous availability of the services being provided by the IAM/PLEX address space, and secondly to automatically collect enough information about any failures such that problem determination and correction can be performed from the single failure. The job that had submitted an I/O request to IAM/PLEX that has experienced a failure will generally be failed with a VSAM logical error code. It will be up to the program to decide whether it can continue processing.

The error data collection will include various messages to the job log, system log, and the RLS log indicating the abend code, general registers, access registers if applicable, and the failing module. The error routines do attempt to not duplicate diagnostic information that is produced by z/OS, but rather provide additional diagnostic information to be combined with the information provided by z/OS. The error information contained in these messages may be sufficient for problem determination and resolution, particularly if a problem had been previously reported to BMC. If the error occurred within the IAM/PLEX address space, an error trace table will be kept in storage for reference, particularly in situations where multiple errors have occurred. Most error situations will also result in a request for a dump to be taken to a system dump data set, which may include both the IAM/PLEX address space and possibly the address space of the job that submitted the failing request. The IAM index space associated with the IAM/PLEX address space may also be dumped.

Serviceability

To aid in serviceability a mechanism is provided for system support personnel to apply critical fixes to the IAM modules within the IAM/PLEX address space, without the need to shut down the address space. This facility can also be used to backout fixes, should they cause other problems. See to the IAM/PLEX operator commands APPLY and RESTORE in IAM-PLEX-Operator-Commands.

HiperDispatch Enhancements

Some performance issues were discovered when running with HiperDispatch mode enabled on LPARs that had more than 4 engines assigned when doing heavy batch processing. The key problem areas were determined to be local lock contention, and dispatching overhead that was impacted by the lock contention. For most I/O requests, IAM used fewer cycles than it was taking to dispatch the work. Based on that information a number of enhancements have been made to the IAM/PLEX processing to alleviate those issues, which include the following:

Implemented use of the Perform Locked Operation (PLO) instruction when updating various queues that had been serialized with the local lock.
Revised key paths to eliminate use of various SVC services that could serialize on various z/OS locks including GETMAIN, FREEMAIN, and MODESET. To improve short term storage acquisition and release was revised to utilize internal cell pool structures.
Implemented use of z/OS high performance Pause and Release services, that replaced WAIT and POST for scheduling the I/O tasks, and replaced SUSPEND and RESUME used to ready completed I/O requests for batch jobs.
Revised I/O task dispatching and management, as explained below.

The IAM/PLEX I/O task management has been revised to provide improved throughput by using fewer I/O tasks. Rather than attaching a new task when an I/O request is received and there are no available I/O tasks, the I/O request is put on a single common work queue. Then the currently active I/O tasks will pull work from the work queue whenever they are done with the I/O request in progress, eliminating any z/OS services from being involved. I/O requests are added to and removed from the work queue with the PLO instruction, avoiding use of z/OS locks. When an active I/O task finds the work queue empty, it uses the Pause service to cause it to wait.

The IAM/PLEX workload manager will manage the number of active I/O tasks between the MINIOTASK and MAXIOTASK values to provide good throughput with the least number of I/O tasks. When an I/O task is no longer necessary to handle the workload, it is placed on an idle queue, where it will remain ready to be reactivated when necessary. When the number of I/O requests on the work queue exceeds internal thresholds, an I/O task will be reactivated if available. If none are available, then an additional I/O task will be attached as long as the total number of I/O tasks is less than the MAXIOTASK value. The values are reported on in the periodical RLSINFO reports if you have them enabled.

It is expected that these changes will reduce the number of I/O tasks being utilized. To help facilitate that, we encourage users to reduce the MINIOTASK value to 4. You can adjust MINIOTASK to determine what value provides the best throughput for your workload, however, making it too high can be counter productive with HiperDispatch enabled.

Record lock contention can result in an increase in active I/O tasks. From the IAM/PLEX perspective, a wait on a record lock can be a long wait. When an I/O request has to wait for a record lock, IAM checks to see how many other tasks are waiting for locks. If this new request would result in all active tasks waiting for record locks, it will invoke the workload manager to add another task. If the IAM/PLEX region is already at MAXTASK, IAM/PLEX will prevent a complete locked situation by failing I/O requests at the time that need to wait on a lock with a logical error code of x’18’. While record lock contention is generally low, this could be a factor in how high a value to use for MAXIOTASK.