Completing Deferred Merge Backups

USTMERGE Overview

The various modes of backup available under the UPSTREAM system are described in System Overview. That chapter also provides numerous backup scenarios, which illustrate the utilization of disk and/or tape media to hold those backups.

A type of backup described in System Overview is the synthetic “full merge” backup. There are three types of “full merge backups, the Full Merge, Deferred Full Merge and the Synthetic Differential Full Merge Backup. They are similar, either copying unchanged files from previous backups (already stored at the z/OS Storage Server) or recording reference pointers to those files (Synthetic Differential Full Merge Backup) and “Merging” with updated files transmitted from the Client to create a new

full backup. These methods have the advantage of creating the new full backup with only a fraction of the overhead - i.e., only the files updated since the most recent (incremental) backup are actually transmitted from the Client to create the full.

To carry out this merging process, a full merge backup has to gain access to one or more previous backups; either the last merged full backup or the Master full backup, plus also any incremental backups taken after it. If either the past full and/or the incremental backups are located on tape, the full merge process will require the use of several tape drives to complete the process. In addition to needing an “output” tape drive to hold the new merged backup, it may also require one or two “input” drives to process the previous backups. Depending on the type of tape drives (e.g., robot or stand-alone), the merging process may also need the intervention of an z/OS operator to mount the required tapes.

In some installations, sufficient tape drives and/or z/OS operations personnel may not be available at the exact time that the full merge backups are being run. For example, it may be most convenient from the Client perspective to run the backups at 22:00, while the most convenient time for doing the merge from a tape-usage perspective may be at 07:00 the following morning.

The USTMERGE utility allows you to take your deferred full merge backups at a time that is most convenient from the Client perspective, but to “defer” the final processing of the merge backup until such time as sufficient tapes drives are available, or z/OS personnel are more readily on hand. Completing a Deferred Full Merge with USTMERGE copies the unchanged files from previous backups to tape. Completing a Synthetic Differential Full Merge Backup with USTMERGE, using the MASTER parameter, copies reference pointers to the unchanged client files that were backed up in the Master full backup and the most recent version of ALL of the CHANGED files from the incremental backups since the prior MASTER Full backup.

While a full merge backup is in “deferred merge” status, the UPSTREAM restore programs recognize this and will mount the previous full and/or incremental backups in order to restore the selected file(s). Once the pending merge process has been completed, all future restores will then come from the newly created merged full backup.

USTMERGE Configuration

Several key steps are required before you can use USTMERGE.

The Backup Profiles

For each backup profile that will employ deferred merge processing, you must specify the MERGE=DEFER option (see MERGE=)

The USTMERxx Profiles

The deferred merge process is controlled by one or more special USTMERxx profiles. A default USTMERxx is created for you when you first install UPSTREAM. You can tailor this default and/or create one or more of your own. See UPSTREAM Profiles for full details.

In summary, the sequential tape backup parameters in the USTMERxx profile are used to allocate a “tape retention” data set on the output tape where the new merged full backups will be written.

• The tape retention data set is created as the first file on the output tape holding the merged backups.
  
• The TAPEPREF parameter provides the name of the tape retention data set.
  
• The RETPD/EXPDT parameters are used to set the expiration of the tape retention data set, and of all the other full merged backup files that are placed onto the output tape.
  

USTMERGE Segregation

Since you may want to run a different USTMERGE process for different backup profiles, you can have more than one USTMERxx profile. This allows you to segregate USTMERGE processing across more than one set of controlling parameters (e.g., to apply differing retention periods).

This segregation is controlled through the GROUPID parameter in the backup profiles, which specifies that the backup profile can only be processed by USTMERGE under the control of a matching USTMERxx profile. When USTMERGE is executed, the “xx” suffix of the MERGExx operand on the USTMERGE command determines which USTMERxx profile will be used, and which backup profiles are eligible for processing.

For example, a USTMERGE executed with the MERGE01 operand initiates deferred merge processing controlled by the USTMER01 vaulting profile, which in turn searches for and processes only the backup profiles with a GROUPID=01 coded. Parameters in the USTMER01 profile control the deferred merge processing for that execution of USTMERGE.

Backup profiles that are enabled for deferred merge processing, but which do not specify a GROUPID value can be processed under any USTMERxx profile. However, it is recommended that if one or more or your backup profiles include GROUPID, then all profiles should utilize the GROUPID parameter to avoid confusion on USTMERGE assignments.

If you wish, you can start multiple concurrent USTMERGE operations, using different USTMERxx profiles. However, an attempt to start a second USTMERGE with the same USTMERxx profile waits until the active one has completed.

USTMERGE Workflow

When USTMERGE runs:

It first scans the UPSTREAM repository for backups that were taken with DEFER=MERGE, but for which the final merge processing has not yet been completed - i.e., the final merge processing is still pending. It then matches the GROUPID in the profile with the value specified on the MERGExx command, as previously described.

• For each backup profile that passes the GROUPID validation, USTMERGE then locates the “deferred” full merge backup, which is at this point really just an incremental backup containing only the files updated since the last backup.
  
• Assuming the “deferred” full merge backup was written to disk, this file is copied across to a new output tape. The profile of the backup being processed must be enabled for sequential tape (TAPE=YES) for this to take place.
  
• The tape data set name in the backup profile (TAPEPREF) will be used for the output data set name, unless it is identical to the disk data set name (DASDPREF), in which case the backup data set will not be renamed when copied across to tape.
  
• Processing a Deferred Full Merge backup, USTMERGE locates the previously merged full backup for this profile together will all the subsequent incremental backups that have been taken since that point. Assuming these previous backups are all on tape, USTMERGE mounts the tapes and reads all the required (i.e., unchanged) files, adding them to the end of the output tape to complete the USTMERGE process.
  
• Processing a Synthetic Differential Full Merge backup, USTMERGE locates the prior completed synthetic differential full merge backup for this profile and all of the subsequent incremental backups. It then copies ALL of the CHANGED files since the prior MASTER Full backup. Assuming these previous backups are all on tape, USTMERGE mounts the tapes and copies the changed files to complete the USTMERGE process.
  
• As the backup of each Client file is moved across to the output tape, the appropriate records in the UPSTREAM repository are updated to reflect their new location.
  

Using the NEWTAPE Operand

The NEWTAPE operand of USTMERGE can be used to control the allocation and handling of the output tape(s), as follows:

• If NEWTAPE is not specified, each execution of USTMERGE will call for a new scratch output tape, using just one drive. Even if the original deferred backup is already on tape, it will still be moved to the new output tape.
  
• All deferred merge backups processed by that one execution of USTMERGE will be written to the same output tape. Unless only one backup profile is processed, the tape will thus become a multi-file tape volume. If the amount of data exceeds the capacity of a single tape, more scratch tapes will be requested, making this a multi-file, multi-volume tape aggregate.
  
• As each outstanding deferred merge backup is selected for processing, USTMERGE will add a new file to the output tape.
  
•  If NEWTAPE is specified, USTMERGE will call for a new output scratch tape for each merge backup being processed in this execution, using the sequential tape parameters in the associated backup profile to allocate the tape. Only one merge backup will be written to each output tape.
  

Coordination with USTMIGRT#

As previously described, USTMERGE needs to access previous backups associated with the backup profile being processed in order to copy those files to the new output tape.

If the original backups are on tape and spread across more than one tape volume, USTMERGE may need to de-allocate and reallocate numerous input tapes to complete the process. However, if USTMERGE detects that the next required backup is in another tape file on the same tape volume, it will simply reposition the tape and read the file. This intelligent utilization of the tape drive can be exploited to further improve the efficiency of the overall USTMERGE process.

USTMERGE always processes the specified backup profiles in reverse alphabetical order. When run without the NEWTAPE operand, described previously, it creates a multi-file tape set containing all of the completed full merge backups, in reverse alphabetical order. On subsequent re-runs, it will then process the previously created multi-file tape set in the same order, thus reducing tape mounting to a minimum.

However, if USTMERGE has to read Client files from several sources in order to copy all of the deferred files, it may still be necessary for it to mount several different tapes during the processing of each backup profile. If that is the case, it will be forced to dismount the currently mounted tape and mount another, thus losing the positioning on that tape. If that tape is required for processing of the next backup, it will have to be remounted and repositioned.

This situation is most likely to occur if daily incremental backups were taken to sequential disk, and then USTMIGRT (see Migrating Backups from Disk to Tape.) was used to move them off to tape without using the FORWARD option to “forward merge” them onto one consolidated tape. One solution to this might be:

• Use the FORWARD option of USTMIGRT to create one consolidated tape set containing all of the recent incremental backups for a selected group of backup profiles, identified by GROUPID.
  
• Then run USTMERGE against the same set of backup profiles (again, identified by GROUPID), allowing it to read those backups, in order, without having to dismount the tape.
  

USTVAULT and USTMIGRT Considerations#

USTMERGE cannot operate concurrently with a USTVAULT (Copying Backups with USTVAULT) or USTMIGRT (Migrating Backups from Disk to Tape) using the same GROUPID. In other words, if you start a USTMERGE process with MERGE01 you cannot start a USTVAULT/USTMIGRT process that also targets profiles with GROUPID=01 specified. Any attempt to start one utility operation when the other is operating causes the second operation to wait until the contending task completes.

Any attempt to run USTVAULT or USTMIGRT for any backups that have pending deferred merges are rejected. You may not run those utilities against the backups until the deferred merges have been completed by USTMERGE.

Delaying USTMERGE Processing

USTMERGE processing can be delayed for some time after the initial deferred merge backup. If the delay extends beyond the next full merge, you will have two (or more) deferred merges pending for the same backup profile. By default, USTMERGE only attempts to complete the most recent pending merge; any others remain in deferred status. However, if you specify the optional FORCE operand, it processes the oldest one (instead of the most recent one). If you have multiple deferred merges pending, you must execute USTMERGE with the FORCE parameter repeatedly until they are all processed.

Initiating USTMERGE

USTMERGE can only be executed as a sub-task of the UPSTREAM started task. This sub-task can be initiated in several ways:

• Using the USTBATCH utility
  
• Via the UPSTREAM TSO/ISPF Interface
  
• Through a z/OS Operator command
  
• Through the UPSTREAM scheduler USTSCHED (UPSTREAM Scheduler), or your own scheduler
  

USTMERGE Initiation via USTBATCH

USTMERGE can be initiated via the USTBATCH utility.

The sample USTBATCH JCL below shows the initiation of the deferred merge facility for all backup profiles beginning with PROD and with the GROUPID=05 specified.

This JCL requires some customization for your own site's requirements, such as the JOB card and STEPLIB specifications. Please also review the parameter descriptions that follow.

• UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters:
  

  //jobname  JOB (accounting,information),'job id data',
  //         NOTIFY=userid,MSGLEVEL=(1,1),CLASS=A,MSGCLASS=X
  //*
  //*
  //MAINT     EXEC PGM=USTBATCH
  //STEPLIB   DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP  DD SYSOUT=*
  //USTLOG    DD SYSOUT=*
  //USTPARM   DD *
  
  HOSTDNS=YOUR.LPAR.DNS.NAME
  HOSTPORT=1972 *default
  *
  CONV=WAIT * WAIT for Backup to complete before ending
  COMMAND=MERGE05 PROFILE=PROD* * MERGE Request
  *
  ENDPARM
  /*

• UPSTREAM 3.9.0 and prior using USTBATCH to Started Task APPC Parameters:

  //jobname JOB (accounting,information),'UPSTREAM MERGE',
  //       MSGLEVEL=(1,1),CLASS=A,MSGCLASS=X
  //*
  //* **********************************************************
  //* ***       COMPLETE THE DEFERRED MERGE PROCESS          ***
  //* **********************************************************
  //*
  //MERGE    EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  APPLPREF=UPSTR       * VTAM APPL Prefix
  USAPPL=UPSTREAM      * Name of UPSTREAM Started TASK VTAM APPL
  LOGMODE=#INTER       * VTAM LOGMODE to use from USTBATCH to STC
  CONV=WAIT            * WAIT for Backup to complete before ending
  COMMAND=MERGE05 PROFILE=PROD*    * MERGE Request
  ENDPARM * End of UPSTREAM USTBATCH Parameters
  /*

  When you have completed reviewing the JCL and parameters, you can submit the JCL and the USTMERGE sub-task operation will begin execution immediately.

USTBATCH / USTMERGE Parameters

The key parameters used in the preceding example are explained here. See z/OS Initiation with USTBATCH for a full description of all USTBATCH parameters.

Beginning with UPSTREAM z/OS 3.9.1, the UPSTREAM started task and USTBATCH may now use TCP/IP to initiate requests. The SNA parameters described further below are no longer required but may still be used. They are required for users of UPSTREAM z/OS 3.9.0 and earlier and are listed here for their reference.

USTBATCH to Started Task TCP/IP Parameters (Preferred):

HOSTDNS

The DNS name of the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTIP parameter.

HOSTIP

The IP address of the TCP/IP stack on the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTDNS parameter.

HOSTPORT

The IP port number of the UPSTREAM started task. If omitted, the default is 1972. This parameter may be used with either the HOSTDNS or HOSTIP parameters.

USTBATCH to Started Task VTAM SNA Parameters (UPSTREAM version 3.9.0 and prior):

APPLPREF

The 5 character prefix of the VTAM APPLID to be used by USTBATCH for its communications with the UPSTREAM started task.

USAPPL

The VTAM APPLID of the UPSTREAM started task.

LOGMODE

The VTAM LOGMODE to be used for communicating with the UPSTREAM started task.

Additional Parameters. (These parameters are used with any version of UPSTREAM and may be used with either TCP/IP or SNA USTBATCH to Started Task communication.):

CONV

This controls whether or not the USTBATCH job should wait for the completion of the request submitted to the started task or end immediately after the request is accepted.

COMMAND

This controls the UPSTREAM started task command that is to be issued, in this case a MERGE process.

The final two characters of the specified command (05 in our example) refer to the GROUPID parameter in the backup profiles that will be targeted by this deferred merge process. We have limited the selection of backup profiles by adding a PROFILE= parameter, to select only the backup profiles beginning with PROD. Without this coded, the deferred merge process would select all outstanding deferred backups (i.e., with MERGE=DEFER in the profile) that have GROUPID=05 specified. Addition of the MASTER parameter (,MASTER) will cause USTMERGE to process as a Synthetic Deferred Full Merge backup.

The actual deferred merge process itself will be controlled by the corresponding special merge profile, as described in UPSTREAM Profiles. In our example, that special profile would be USTMER05.

ENDPARM

Indicates the end of the USTBATCH input parameters.

USTMERGE Initiation via the TSO/ISPF Dialog

You can initiate USTMERGE using the UPSTREAM TSO/ISPF dialog. Select option 6 (“Operator Commands”) and press ENTER.

USTMERGE Initiation via a z/OS Operator Command

You can initiate USTMERGE using a z/OS Operator command. This is often useful when initiating via an automated schedule or an automated operations tool. However, unlike with USTBATCH initiation, running the command “manually” in this way does allow you to perform any conditional checking of the results.

• First, enter the z/OS console interface that you would normally utilize to access the z/OS system console. In the example below we are using the IOF product.
  
• Then, on the command line, enter your UPSTREAM MERGE command in the format shown below and press ENTER to execute the command:
  F [started task name],MERGExx [PROFILE=xxxxxxxx][,NEWTAPE][,FORCE][,MASTER]
• In our example below, we will run the process against all outstanding deferred merge backups for profiles beginning with SRV and which have a GROUPID of 05.

You may terminate an active USTMERGE sub-task if, for any reason, you cannot wait for it to complete. This is not recommended. Whenever possible you should always allow the process to complete. However, if you need to prematurely terminate a USTMERGE sub-task, you can do so by:

• Issuing the console command: F UPSTREAM,TERM LU=USTMERxx
• Using the TERM line command on the UPSTREAM ISPF status display (see Controlling UPSTREAM Sub-Tasks).
  
• Using a console STOP (P) command to shutdown UPSTREAM itself. This has the same effect as TERM on a USTMERGE sub-task.
  

When terminated by any of the preceding methods, USTMERGE completes the processing on the current backup. This orderly termination may take a few minutes to complete. If you restart the merge operation at a later time, USTMERGE processes only those backups that were not completed on the earlier run.

USTMERGE should not be terminated while it has an outstanding tape mount, as this may cause premature termination of the utility. If you cannot wait for the orderly termination, as previously described, then you can issue the TERM a second time to force the termination. This should only be done in extreme circumstances as the forced termination of an incomplete sub-task can occasionally cause problems within the main UPSTREAM started task.

Recovering from a Failed USTMERGE

If a USTMERGE run does not complete for some reason (perhaps an abend, or a forced termination) all backups that were successfully merged have been flagged as such, so a later execution of USTMERGE correctly bypasses them.

USTMERGE Reporting

You can use USTRPORT (Reporting with USTRPORT) to produce a report showing all UPSTREAM backups that are in deferred merge status (i.e., the full merge backup has been run but the USTMERGE process, which completes the backup, has not yet been executed).