Migrating Backups from Disk to Tape

USTMIGRT Overview

USTMIGRT is a utility program that will “migrate” recently created sequential disk backups to tape.

USTMIGRT allows you to initially write your backups to disk instead of tape. This improves efficiency, since many backups can operate simultaneously, without being limited by tape mount requirements or available tape drives.

Periodically, you can then collect those disk backups with USTMIGRT and move them across to tape, freeing up the disk space. This can be especially useful if you want to do daily incremental backups to disk, but you do not have enough available disk space to hold the entire set of incremental backups for the week; i.e., between your regular full merge backups. When processing multiple backup profiles, all backups will be placed on a single/consolidated output tape set (unless the NEWTAPE option is specified) reducing the number of tape volumes when compared to writing multiple backups to separate tapes.

Once a backup data set has been migrated across to tape, a restore from that backup will operate exactly as if the original backup had been done directly to tape - it does not have to be copied to disk first. The UPSTREAM repository information is updated to point directly to the new location of the backups on tape.

USTMIGRT vs. z/OS DASD Management Migration

UPSTREAM disk backups can also be migrated to tape by your z/OS DASD Management System, as long as the appropriate auto-recall services are in place to allow UPSTREAM to recall the backup to disk prior to doing the restore. This may, however, be very time-consuming and may require multiple tape mounts when several backups must be read to complete a restore. For this reason, this method is not recommended.

As previously described, with USTMIGRT, the restore can take place directly from the tape, without requiring the backup file to first be recovered to disk before the restore can take place. For this reason, USTMIGRT is the recommended method for moving disk-based backups to tape.

USTMIGRT vs. COPYINCR

The COPYINCR option of full merge backups (Completing Deferred Merge Backups) is another way of moving incremental backups from disk to tape, but this is not done until the full merge backup is run. USTMIGRT, on the other hand, can move backups at any time, and for more than one backup profile.

USTMIGRT Configuration

Several steps are required before you can use USTMIGRT.

The Backup Profiles

Each backup profile for which you intend to utilize USTMIGRT must be enabled for the migration process. This is achieved via the MIGTHRESH= parameter, as follows.

• The default value is 0, which disables USTMIGRT for the profile.
  
• Any value greater that 0 enables USTMIGRT for the profile.
  
• See “USTMIGRT Workflow” for a full description of the MIGTHRESH parameter.
  

Any backup profile that will utilize USTMIGRT must also be enabled for tape backups, even if it will not at any time directly create tape-based backups. This enabling of tape backups is achieved with the TAPE=YES parameter.

See UPSTREAM Profiles for full details on how to create backup profiles.

The USTMIGxx Profiles

The USTMIGRT process is controlled by one or more USTMIGxx migration profiles. See UPSTREAM Profiles for full details on how to create USTMIGxx profiles and also for full descriptions of the parameters mentioned below.

The sequential tape backup parameters in the USTMIGxx migration profile are used to allocate a tape retention data set on the output tape.

• The tape retention data set is created as the first file on the output tape holding the migrated backups.
  
• The TAPEPREF parameter provides the name of the tape retention data set, which is recognized by your tape management system for off-site vaulting.
  
• The RETPD/EXPDT parameters are used to set the expiration of the tape retention data set, and of all the other migrated backups that are placed on the tape.
  
• If the USTMIGxx profile contains the TAPEGDG option, the tape retention data set is created as a GDG; this is recommended so that normal GDG processing uncatalogs the tape retention data set as new generations are created. If you use tape management catalog control (EXPDT=99000 for some tape management systems), the migration tapes are retained for the number of generations specified by the GDG base created for this GDG
  

USTMIGRT Segregation

Since you may want to run a different migration process for different backup profiles, you can create more than one USTMIGxx profile. This allows you to segregate your migrations across more than one set of controlling parameters. As an example, each USTMIGxx profile may provide different retention periods for the migrated backups.

This segregation is controlled through the GROUPID parameter in the backup profile, which specifies that the backups belonging to that profile can only be migrated under the control of a matching USTMIGxx migration profile. When USTMIGRT is executed, the “xx” suffix of the MIGRTxx operand on the USTMIGRT command determines which USTMIGxx profile will be used, and which backup profiles will be processed.

For example, a USTMIGRT executed with the MIGRT01 operand initiates migrate processing controlled by the USTMIG01 migration profile, which in turn searches for and processes only the backup profiles with a GROUPID=01 coded.

Backup profiles that are enabled for USTMIGRT migration, but which do not specify a GROUPID value can be processed under any USTMIGxx profile. However, it is recommended that if one or more or your backup profiles include GROUPID, then all profiles should utilize the GROUPID parameter to avoid confusion on migration assignments.

If you wish, you can start multiple concurrent USTMIGRT operations, using different USTMIGxx profiles. However, an attempt to start a second migration with the same USTMIGxx profile will wait until the active one has completed.

USTMIGRT Workflow

When USTMIGRT is run:

• It first scans the UPSTREAM repository for sequential disk backups recorded under a backup profile whose MIGTHRESH value is greater than 0.
  
• If the number of such backups is equal to or higher than the threshold, the least recently created disk backups will be migrated to tape until the number of disk backups remaining is nn-1 (one less than the threshold).
  
• For example, MIGTHRESH=1 will cause all disk backups to migrate, but MIGTHRESH=3 leaves the 2 most recently created backups on disk.
  
• If the latest backup is a restartable/interrupted backup, it will not migrate until the backup has been restarted and successfully completed.
  

The new tape-based backups have the same data set name as their original disk counterparts. However, if multiple backups are migrated under the same backup profile name (e.g., USTMIGRT is being run infrequently and you are migrating several days worth of incremental backups in one pass), all of these backups will be combined into a single output tape file. The name of that file is the same as the most recently created disk backup that has been migrated.

Assuming multiple backup profiles are being processed in a single pass, the output tape is a multi-file tape volume. If the amount of data being migrated exceeds the capacity of the first tape volume, additional scratch tapes are requested, making this a multi-file, multi-volume tape aggregate. As each disk backup is migrated, the appropriate UPSTREAM repository records are updated to point to the new location of the backups.

Several options are provided to add additional control to the USTMIGRT process.

NEWTAPE

If the NEWTAPE operand is specified, USTMIGRT calls for a new scratch tape for each backup migrated to tape. It uses the sequential tape parameters in the associated backup profile to allocate the tape. The TAPEPREF value is used for the output data set name unless the TAPEPREF and DASDPREF values are identical; in which case, the original backup data set name is used. When NEWTAPE is specified, no dummy file is created at the beginning of each tape.

FORWARD

If USTMIGRT migrates several backups for the same backup profile in one execution, it combines them into one file on the output tape. The FORWARD option enhances this process to also add previous backups that have already been migrated.

With the FORWARD option, USTMIGRT will do a “forward merge” of any incremental backups previously moved to tape by USTMIGRT, back to the last full backup. It reads those backups from past tapes and merges them with the new data being written to the new tape. This effectively creates one file on the output tape for each backup profile being processed, and this file contains data from all of the preceding incremental backups, back to the last full backup.

This consolidation of all backups onto a single tape makes restores very efficient, since a minimum number of tapes must be mounted to complete the process.

The FORWARD migration process requires two tape drives; one for the new output tape, and one to hold the input tape(s) containing the previously migrated backups.

USTMIGRT is very efficient on tape handling during the FORWARD processing. For example, if the next “previously migrated” backup is on an input tape already mounted, even if it is for a different backup profile, USTMIGRT simply re-positions to that file and reads it without first dismounting the tape.

If you do FORWARD migration on the same set of backup profiles every day, all of the data required for each day's migration is on the previous day's migration tape. Since the backup profiles are processed in the same order each time, USTMIGRT is usually able to process that input tape efficiently, without dismounting it.

Once the previous tape backups have been moved over to the new output tape, the input tape data sets are uncataloged. If you have them under tape management “catalog control” they are automatically eligible for scratch.

When FORWARD is specified, no dummy file is created at the beginning of each tape.

MAXV=

The MAXV value (1-99) limits the number of backup versions for a profile that are migrated FORWARD from the previous migration tape data set. It is only valid if FORWARD is also specified.

ALL

Specify ALL to migrate all DASD-resident backup data sets to tape. This keyword is mutually exclusive with the FORWARD option. This causes no backup data sets to be left on DASD.

10.5 USTMIGRT with USTVAULT and USTMERGE

USTMIGRT cannot operate concurrently with a USTVAULT (Copying Backups with USTVAULT) or USTMERGE (Completing Deferred Merge Backups) using the same GROUPID. In other words, if you start a USTMIGRT process with MIGRT01 you cannot start a USTVAULT/USTMERGE process which also targets profiles with GROUPID=01 specified. Any attempt to start one utility operation when the other is operating will cause the second operation to wait until the contending task completes.

For deferred merge backups in “pending” status, the USTMERGE process must first complete the merge process before USTMIGRT can be used to migrate the backups. Refer also to the USTMERGE section for details on using USTMIGRT in conjunction with USTMERGE.

Notes on Utility Execution

• For USTVAULT only - The optional parameter NOINCR causing vaulting to occur only for full backups, bypassing any incremental backups. The optional parameter NOVCHK will cause backups to be vaulted even if they are flagged as already having been vaulting, allowing you to recreate vault tapes or, in conjunction with COPY=, create additional vault copies. The optional parameter COPY=n controls the copy number that will replace the “?” in the backup file names created on the vault (2 through 9, 2 is the default).
  
• For USTMERGE only - If you do multiple Deferred Full MERGE backups without running USTMERGE, resulting in multiple uncompleted full MERGE backups, USTMERGE will normally only process the most recently created Deferred MERGE backup. If you need to retain the other backups, specify the optional FORCE parameter; this will cause USTMERGE to select the oldest Deferred MERGE backup instead. If there are multiple such uncompleted backups, you will have to execute with FORCE multiple times until they are all processed.
  

• For USTMIGRT only - The MAXV keyword specifies the maximum number of backups for a profile that will be processed during a MIGRATE FORWARD operation.

  This feature exists to limit the overhead associated with continual copying and DISP=MOD processing associated with MIGRATE FORWARD execution. For example, if MAXV=3 is specified and one backup is created each day, then on Monday the first backup is copied from disk to tape.

  On Tuesday, the backup on disk is copied plus the Monday backup that is on tape. On Wednesday, the backup on disk is copied plus the two previous backups on tape. On Thursday only the disk backup is copied to tape because there are now 3 backups on the previous migration tape. If the previous backups on tape plus the disk backups exceed the MAXV value, the previous backups will not be moved forward.

  The MAXV value does not apply to disk. If 2 backups for the same profile exist on disk they will all be copied to tape and counted towards the MAXV value. If the previous tape contains two backups then neither backup will be copied forward since it makes no sense to copy some of the backups from the previous tape.

10.6 Initiating USTMIGRT

USTMIGRT can only be executed as a sub-task of the UPSTREAM started task. This sub-task can be initiated in several ways:

• Using the USTBATCH utility
  
• Via the UPSTREAM TSO/ISPF Interface
  
• Through a z/OS Operator command
  
• Through the UPSTREAM scheduler USTSCHED (see UPSTREAM Scheduler), or your own scheduler.
  

USTMIGRT Initiation via USTBATCH

USTMIGRT can be initiated via the USTBATCH utility.

The sample USTBATCH JCL below shows the initiation of the migration facility for the backup profile called TEST.

This JCL requires some customization for your own site's requirements, such as the JOB card and STEPLIB specifications. Please also review the parameter descriptions that follow.

• UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters

  //jobname  JOB (accounting,information),'job id data',
  //         NOTIFY=userid,MSGLEVEL=(1,1),CLASS=A,MSGCLASS=X
  //*
  //*
  //MAINT    EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  HOSTDNS=YOUR.LPAR.DNS.NAME
  HOSTPORT=1972 *default
  *
  CONV=WAIT * WAIT for Backup to complete before ending
  COMMAND=MIGRT01 PROFILE=TEST * MIGRATE Request
  *
  ENDPARM
  /*

• UPSTREAM 3.9.0 and prior using USTBATCH to Started Task APPC Parameters
  

  //jobname  JOB  (accounting,information),'UPSTREAM MIGRATE',
  //        MSGLEVEL=(1,1),CLASS=A,MSGCLASS=X
  //*
  //*  **********************************************************
  //*  ***         MIGRATE PREVIOUSLY TAKEN BACKUPS           ***
  //*  **********************************************************
  //*
  //MIGRATE  EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  APPLPREF=UPSTR        * VTAM APPL Prefix
  USAPPL=UPSTREAM       * Name of UPSTREAM Started TASK VTAM APPL
  LOGMODE=#INTER        * VTAM LOGMODE to use from USTBATCH to STC
  CONV=WAIT              * WAIT for Backup to complete before ending
  COMMAND=MIGRT01 PROFILE=TEST     * MIGRATE Request
  ENDPARM               * End of UPSTREAM USTBATCH Parameters

When you have completed reviewing the JCL and parameters, you can submit the JCL and the USTMIGRT sub-task operation begins execution immediately.

USTBATCH / USTMIGRT Parameters

The key parameters used in the preceding example are explained here. For a full description of all USTBATCH parameters, see z/OS Initiation with USTBATCH.

Beginning with UPSTREAM z/OS 3.9.1, the UPSTREAM started task and USTBATCH may now use TCP/IP to initiate requests. The SNA parameters described further below are no longer required but may still be used. They are required for users of UPSTREAM z/OS 3.9.0 and earlier and are listed here for their reference.

USTBATCH to Started Task TCP/IP Parameters (Preferred):

HOSTDNS

The DNS name of the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTIP parameter.

HOSTIP

The IP address of the TCP/IP stack on the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTDNS parameter.

HOSTPORT

The IP port number of the UPSTREAM started task. If omitted, the default is 1972. This parameter may be used with either the HOSTDNS or HOSTIP parameters.

USTBATCH to Started Task VTAM SNA Parameters (UPSTREAM version 3.9.0 and prior):

APPLPREF

The 5-character prefix of the VTAM APPLID to be used by USTBATCH for its communications with the UPSTREAM started task.

USAPPL

The VTAM APPLID of the UPSTREAM started task.

LOGMODE

The VTAM LOGMODE to be used for communicating with the UPSTREAM started task.

Additional Parameters. (These parameters are used with any version of UPSTREAM and may be used with either TCP/IP or SNA USTBATCH to Started Task communication.):

CONV

This controls whether or not the USTBATCH job should wait for the completion of the request submitted to the started task, or end immediately after the request is accepted.

COMMAND

This controls the UPSTREAM started task command that is to be issued, in this case a MIGRATE process.

The final two characters of the specified command (01 in our example) refer to the GROUPID parameter in the backup profiles that will be targeted by this migration process. We have limited the selection of backup profiles by adding a PROFILE= parameter, to select only the backups belonging to the TEST backup profile. Without this coded, the migration process would select all migrate enabled backup profiles (i.e., with a MIGTHRESH greater than 0 specified in the profile) that have GROUPID=01 specified.

The actual migration process will be controlled by the corresponding special migration profile, as described in UPSTREAM Profiles. In our example, that special profile would be USTMIG01.

ENDPARM

Indicates the end of the USTBATCH input parameters.

USTMIGRT Initiation via the TSO/ISPF Interface

You can initiate USTMIGRT using the UPSTREAM TSO/ISPF dialog. First, select Option # 6 “Operator Commands”.

USTMIGRT Initiation via a z/OS Operator Command

You can initiate USTMIGRT using a z/OS Operator command. This is often useful when initiating via an automated schedule or an automated operations tool. However, unlike with USTBATCH initiation, running the command “manually” in this way will not allow you to perform any conditional checking of the results.

• First, enter the z/OS console interface that you would normally utilize to access the z/OS system console. In the example below we are using the IOF product.
  

• Then, on the command line, enter your UPSTREAM MIGRT command in the format shown below and press ENTER to execute the command.

     F [started task name],MIGRTxx [PROFILE=xxxxxxxx][,NEWTAPE][,FORWARD]
• In our example below, we are running a USTMIGRT against all migration enabled backup profiles (MIGTHRESH greater than 0 specified in the profile) which have a GROUPID=01.

Terminating USTMIGRT

You may terminate an active USTMIGRT sub-task if, for any reason, you cannot wait for it to complete. This is not recommended. Whenever possible you should always allow the process to complete. However, if you need to prematurely terminate a USTMIGRT sub-task, you can do so by:

• Issuing the console command:
   F UPSTREAM,TERM LU=USTMIGxx
• Or, using the TERM line command on the UPSTREAM ISPF status display (see Controlling UPSTREAM Sub-Tasks).
  
• Or, using a console STOP (P) command to shutdown UPSTREAM. This will have the same effect as TERM on a USTMIGRT sub-task.
  

When terminated by any of the preceding methods, USTMIGRT will complete the processing on the current backup. This orderly termination may take a few minutes to complete. If you restart the migration operation at a later time, USTMIGRT will process only those backups that were not completed on the earlier run.

STMIGRT should not be terminated while it has an outstanding tape mount, as this may cause premature termination of the utility. If you cannot wait for the orderly termination, as previously described, then you can issue the TERM a second time to force the termination. This should only be done in extreme circumstances as the forced termination of an incomplete sub-task can occasionally cause problems within the main UPSTREAM started task.

Recovering from a Failed USTMIGRT

If a USTMIGRT run does not complete for some reason (perhaps an abend, or a forced termination) all backups that were successfully copied to tape will have been flagged as being successfully migrated, so a later execution of USTMIGRT will correctly bypass them.