Copying Backups with USTVAULT

USTVAULT Overview

The UPSTREAM vaulting facility (USTVAULT) automates the creation of secondary (“vault”) copies of your sequential disk and tape backup files. The vaulted copies will be placed on tape. Your tape management system can then optionally send these vaulted copies off-site, if required. If you have the UPSTREAM data encryption feature licensed and enabled (UPSTREAM Data Encryption), you can use USTVAULT to create encrypted copies of some/all your “copy 1" backups.

There are several reasons why you may want to use USTVAULT to create secondary copies of your backups:

• For on-site recovery: in case the primary backups are damaged or become unusable.
  
• For disaster recovery: the secondary copies can be stored in an off-site vault for use in the event of a disaster.
  
• For long-term retention: as described in Creating Long-Term Retention Backups, the secondary copies are to be retained for much longer than the primary backups, without requiring additional space in the UPSTREAM repository.
  

A single execution of USTVAULT can create vault copies for specific backup profiles, or for a group of backup profiles, if you choose. USTVAULT will only create vaulted copies going back to the last full backup.

If you have any merge backups in deferred status (see Completing Deferred Merge Backups) these will be automatically bypassed since they are not yet complete. Once they have been completed by USTMERGE, they will be eligible for vaulting.

If a backup was flagged as “interrupted”, it will only be selected for vaulting if it is also flagged as “restartable”. If USTVAULT copies an interrupted backup, and that backup is later restarted, the next execution of USTVAULT which processes that backup profile will correctly copy the backup again, uncataloging the earlier copy.

USTVAULT keeps track of the backups that already have vaulted copies, and will, by default, only create secondary copies for those backups that have not yet been processed. However, the NOVCHK option allows USTVAULT to create vaulted copies for backups that do already have vaulted copies. This allows you to create extra copies of your backups for additional safety, or to create copies to be kept for a longer period. A “copy number” from 2 - 9 can be specified on the VAULT command to identify these additional vaulted copies.

USTVAULT Configuration

Several steps are required before you can use USTVAULT.

The Backup Profiles

Each backup profile for which you intend to create copies of your backups must be enabled for vaulting. There are several steps to this process. See UPSTREAM Profiles for full details on how to create backup profiles and also for full descriptions of the parameters mentioned below.

• The VAULT parameter in the backup profile must be set to “YES”.
  
• The values specified for DASDPREF and/or TAPEPREF must contain a question mark “?”. The question mark will be replaced by a “1” (i.e., COPY=1) during the creation of the primary backups, and it will be replaced by a number from 2 - 9 (default is 2) during the USTVAULT process, when the secondary and subsequent copies are created. This allows the primary, secondary and any subsequent copies of the backups to be properly cataloged under your z/OS system.
  
• If the backup profile specifies DASDGDG and/or TAPEGDG, you must also create the required GDG bases for both the primary copy and also for any secondary vaulted copies of the backups that you may wish to create. See UPSTREAM Profiles for example JCL to create these GDG bases.
• If the backup profile specifies ENCRYPTV, and if you have the UPSTREAM data encryption feature licensed and enabled, the copies of backups created under this profile can be encrypted by USTVAULT.

The USTVLTxx Profiles

The USTVAULT process is controlled by one or more USTVLTxx profiles. See UPSTREAM Profiles for full details on how to create USTVLTxx profiles and also for full descriptions of the parameters mentioned below.

The sequential tape backup parameters in the USTVLTxx profile are used to allocate a “tape retention” data set on the output tape.

• The tape retention data set is created as the first file on the output tape holding the vaulted backups.
  
• The TAPEPREF parameter provides the name of the tape retention data set, which will be recognized by your tape management system for off-site vaulting.
  
• The RETPD/EXPDT parameters are used to set the expiration of the tape retention data set, and of all the other vaulted files that will be placed on the tape.
  

The sequential disk backup parameters in the USTVLTxx profile are used to allocate a vault control data set on disk.

• The vault control data set contains the control records that describe the new vault copies. It is created initially on disk, and then copied across as the last file on the output tape when USTVAULT processing completes. The DASDPREF parameter provides the name of the vault control data set.
  
• If you have the UPSTREAM data encryption feature licensed and enabled, and if USTVAULT has created one or more encrypted copies of backups on this run, the ENCRYPTV parameter causes the vault control file data set to also be encrypted, just prior to the point that it is moved to the output tape.
  

USTVAULT Segregation

Since you may want to run a different vaulting process for different backup profiles, you can create more than one USTVLTxx profile. This allows you to segregate your vaulting across more than one set of controlling parameters. As an example, each USTVLTxx profile may provide different data set names and/or retention for the vaulted copies of your backups.

This segregation is controlled through the GROUPID parameter in the backup profile, which specifies that the backup profile can only be vaulted under the control of a matching USTVLTxx vault profile. When USTVAULT is executed, the “xx” suffix of the VAULTxx operand on the USTVAULT command determines which USTVLTxx profile will be used, and which backup profiles will be processed.

For example, a USTVAULT operation executed with the VAULT01 operand will initiate vault processing controlled by the USTVLT01 vaulting profile, which will in turn search for and process only the backup profiles with a GROUPID=01 coded.

Backup profiles that are enabled for vaulting, but which do not specify a GROUPID value can be processed under any USTVLTxx profile. However, it is recommended that if one or more of your backup profiles includes GROUPID, then all profiles should utilize the GROUPID parameter to avoid confusion on vaulting assignments.

If you wish, you can start multiple concurrent vaulting operations, using different USTVLTxx profiles. However, an attempt to start a second vault with the same USTVLTxx profile will wait until the active one has completed.

USTVAULT Workflow

For each selected sequential backup, USTVAULT will locate the original COPY=1 backup on disk (for sequential disk backups), or it will mount the required tape (for sequential tape backups), and then create a copy of it as the next sequential file on the output vault tape.

The vaulted data set name is the same as the original backup, except that the copy number in the filename (as specified by the “?” in TAPEPREF= or DASDPREF= in the backup profile) is changed from “1” to the copy number of the vault copy. This is controlled by the COPY=n operand on the USTVAULT command.

If you are using the UPSTREAM data encryption feature, and if the profile describing the backup being copied by USTVAULT has the ENCRYPTV option set, USTVAULT encrypts the data during the copy process. As described in UPSTREAM Data Encryption, we recommend that you use high compression when creating the original backups that you later wish to be encrypted by USTVAULT, as this reduces the volume of data requiring encryption and the amount of tape media required to contain the USTVAULT copy.

UPSTREAM repository records describing the vaulted copy are then written to a temporary vault control data set on disk. When the vaulting process has completed for all selected profiles, USTVAULT then copies the vault control data set as the last file on the vault tape and catalogs it. The temporary disk file is deleted, unless the NOSCR operand is specified, in which case it is retained on DASD and is cataloged in place of the tape copy.

The original repository records are not updated; they continue to point to the COPY=1 backup, which is used for any restore requests. If you need to restore files from a vaulted backup of a backup, you must first use the USTREGEN utility to update the repository with information about the vaulted backups, as described in Restoring from Vaulted Backups. Each time USTVAULT is run, two new records are however added to the UPSTREAM repository:

• A record is added under the USTVLTxx profile name, showing a sequential tape backup with the name of the tape retention data set, which was created as the first file on the vault tape. This record documents when the vaulting operation began.
  
• A record is also added under the profile name USTVLCxx, showing a sequential tape backup with the name of the vault control data set created as the last file on the vault tape. This USTVLCxx profile does not exist in the configuration and is used for reporting purposes only.
  

The USTVLTxx and USTVLCxx records remain in the repository for as long as the data sets they point to are still cataloged. The USTMAINT utility (see UPSTREAM Operation) deletes them when they are no longer cataloged. See USTVAULT Reportingfor details on how to report on the USTVLTxx and USTVLCxx records in the UPSTREAM repository.

USTMIGRT and USTMERGE Considerations

USTVAULT cannot operate concurrently with a USTMIGRT (see Migrating Backups from Disk to Tape) or USTMERGE (see Completing Deferred Merge Backups) using the same GROUPID. In other words, if you start a USTVAULT process with VAULT01 you cannot start a USTMIGRT or USTMERGE process that also targets profiles with GROUPID=01 specified. Any attempt to start one utility operation when the other is operating will cause the second operation to wait until the contending task completes.

Any attempt to run USTVAULT for a backup profile that has a pending deferred merge (see Completing Deferred Merge Backups) will be rejected. You may not run USTVAULT against the backups until the deferred merge has been completed by USTMERGE.

Creating Long-Term Retention Backups

The primary copy of each UPSTREAM backup requires space in the “USTCATLG” and “USTFILEI” repository data sets. In particular, the “USTFILEI” cluster contains a record for every file that has been backed up.

Multiple records exist for files that have been backed up multiple times. If you are backing up large servers, or a great number of servers and workstations, this repository storage can be quite considerable (see space calculations in The UPSTREAM Repository).

For this reason, most UPSTREAM backups would be kept only for a limited period of time, perhaps for 1 month, or for a specific number of “versions”, before they are finally uncataloged and expired. This not only frees up the disk/tape media on which the backups reside, but it also allows the USTMAINT utility (see UPSTREAM Operation) to release the space taken up by the related control records in the UPSTREAM repository data sets.

However, if you need to keep certain backups for an extended period of time, for example to allow for long-term recovery or for legal reasons, you can do this with USTVAULT without requiring the additional space in the UPSTREAM repository.

As previously described, USTVAULT can create vault copies of your backups with differing names (i.e., different copy numbers). Thus, you can run a special USTVAULT execution whenever required, which utilizes a different USTVLTxx profile from the norm. This special USTVLTxx profile provides different (i.e., longer) retention periods for those long-term backup copies.

A console command to initiate a special “long-term” USTVAULT might look like this:

The COPY=4 causes USTVAULT to replace the “?” with a “4” in the vaulted filenames. The NOVCHK option allows USTVAULT to create the long-term vaulted copies for backups that already have other vaulted copies.

This process works best if the TAPEPREF prefix names in the USTVLTxx profile are GDGs, so that you can build the GDG bases with the proper number of generations to keep the number of versions you need. For example, 12 generations of a quarterly backup will be kept for 3 years. Tape management catalog control (EXPDT=99000) could then be used to retain and eventually expire the backups.

Initiating USTVAULT

USTVAULT can only be executed as a sub-task of the UPSTREAM z/OS Storage Server started task. This sub-task can be initiated in several ways:

• Using the USTBATCH utility
  
• Via the UPSTREAM TSO/ISPF dialog
  
• Through a z/OS operator command
  
• Through the UPSTREAM scheduler USTSCHED (see UPSTREAM Scheduler), or your own scheduler.
  

VAULT Initiation via USTBATCH

USTVAULT can be initiated via the USTBATCH utility.

The sample USTBATCH JCL below shows the initiation of the vaulting facility for the backup profile called TEST.

This JCL would require some customization for your own site's requirements, such as the JOB card and STEPLIB specifications. Please also review the parameter descriptions that follow.

• UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters:

  //jobname JOB (accounting,information),'job id data',
  //        NOTIFY=userid
  //*
  //* **********************************************************
  //* *** VAULT PREVIOUSLY TAKEN BACKUPS ***
  //* **********************************************************
  //*
  //MAINT    EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  HOSTDNS=YOUR.LPAR.DNS.NAME
  HOSTPORT=1972              *default
  *
  COMMAND=VAULT01,PROFILE=TEST * VAULT Request
  *
  ENDPARM                 * End of UPSTREAM USTBATCH Parameters
  /*

• UPSTREAM 3.9.0 and prior using USTBATCH to Started Task VTAM SNA Parameters:
  

  //jobname JOB (accounting,information),'UPSTREAM VAULT',
  //       MSGLEVEL=(1,1),CLASS=A,MSGCLASS=X
  //*
  //* **********************************************************
  //* *** VAULT PREVIOUSLY TAKEN BACKUPS ***
  //* **********************************************************
  //*
  //VAULT     EXEC PGM=USTBATCH
  //STEPLIB   DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP  DD SYSOUT=*
  //USTLOG    DD SYSOUT=*
  //USTPARM   DD *
  APPLPREF=UPSTR    * VTAM APPL Prefix
  USAPPL=UPSTREAM   * Name of UPSTREAM Started TASK VTAM APPL
  LOGMODE=#INTER    * VTAM LOGMODE to use from USTBATCH to STC
  CONV=WAIT         * WAIT for Backup to complete before ending
  COMMAND=VAULT01,PROFILE=TEST   * VAULT Request
  ENDPARM           * End of UPSTREAM USTBATCH Parameters
  /*

When you have completed reviewing the JCL and parameters, you can submit the JCL and the USTVAULT sub-task operation will begin execution immediately.

USTBATCH/USTVAULT Parameters

The key parameters used in the preceding example are explained here. For a full description of all USTBATCH parameters, see z/OS Initiation with USTBATCH.

Beginning with UPSTREAM z/OS 3.9.1, the UPSTREAM started task and USTBATCH may now use TCP/IP to initiate requests. The SNA parameters described further below are no longer required but may still be used. They are required for users of UPSTREAM z/OS 3.9.0 and earlier and are listed here for their reference.

USTBATCH to Started Task TCP/IP Parameters (Preferred):

HOSTDNS

The DNS name of the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTIP parameter.

HOSTIP

The IP address of the TCP/IP stack on the LPAR hosting your UPSTREAM started task. Mutually exclusive with the HOSTDNS parameter.

HOSTPORT

The IP port number of the UPSTREAM started task. If omitted, the default is 1972. This parameter may be used with either the HOSTDNS or HOSTIP parameters.

USTBATCH to Started Task VTAM SNA Parameters (UPSTREAM version 3.9.0 and prior):

APPLPREF

The 5-character prefix of the VTAM APPLID to be used by USTBATCH for its communications with the UPSTREAM started task.

USAPPL

The VTAM APPLID of the UPSTREAM started task.

LOGMODE

The VTAM LOGMODE to be used for communicating with the UPSTREAM started task.

Additional Parameters. (These parameters are used with any version of UPSTREAM and may be used with either TCP/IP or SNA USTBATCH to Started Task communication.):

CONV

This controls whether or not the USTBATCH job should wait for the completion of the request submitted to the started task, or end immediately after the request is accepted.

COMMAND

This specifies the UPSTREAM started task command that is to be issued, in this case a VAULT process.

The final two characters of the specified command (01 in our example) refer to the GROUPID parameter in the backup profiles that will be targeted by this vaulting process. We have limited the selection of backup profiles by adding a PROFILE= parameter, to select only the backups belonging to the TEST backup profile. Without this coded, the vault process would select all vault enabled backup profiles (VAULT=YES specified in the profile) that have GROUPID=01 specified.

The actual vault process itself will be controlled by the corresponding special vault profile, as described in UPSTREAM Profiles. In our example, that special profile would be USTVLT01.

ENDPARM

Indicates the end of the USTBATCH input parameters.

VAULT initiation via the TSO/ISPF Interface

You can initiate USTVAULT using the UPSTREAM TSO/ISPF dialog. First, select Option # 6 “Operator Commands”.

The COPY field specifies the numeric substitution character that will replace the “?” character specified in the backup profile TAPEPREF/DASDPREF fields when the VAULT copy of the backup data set is created (see USTVAULT Configuration for more details).

VAULT Initiation via a z/OS Operator Command

You can initiate USTVAULT using a z/OS Operator command. This is often useful when initiating via an automated schedule or an automated operations tool. However, unlike with USTBATCH initiation, running the command “manually” in this way will not allow you to perform any conditional checking of the results.

• First, enter the z/OS console interface that you would normally utilize to access the z/OS system console. In the example below we are using the IOF product.
  

• Then, on the command line, enter your UPSTREAM VAULT command in the format shown below and press ENTER to execute the command:

  F [started task name],VAULTxx [PROFILE=xxxxxxxx][,NOVCHK][,NOINCR][NOFULL]
  [,NOSCR][C=n][,NOCRYPT][BLK64K]
• In our example below, we are running a USTVAULT against all vault enabled backup profiles (VAULT=YES specified in the profile) which have a GROUPID=01. The NOINCR parameter allows USTVAULT to vault only the most recent full backup, effectively bypassing any recent incremental backups. The NOFULL Vault parameter instructs USTVAULT to vault only incremental backups, which have not been previously vaulted, back to the last full backup.

• Specify C=n to set the “copy number” of this vault run. The default is C=2.
  
• By default the VAULT utility does not vault a backup that has already been vaulted. Specify NOVCHK to bypass that test. This may be abbreviated as “NOV”.
  
• During the VAULT process, the utility creates a Vault Control File data set on DASD containing the control records related to the backup that was vaulted. At the end of the process, the Vault Control File is moved to the end of output tape as a separate data set, recataloged and scratched from DASD. Specify the NOSCR keyword on the VAULT command to leave the cataloged copy of the Vault Control File data set on DASD.
  
• Specify NOCRYPT to cause execution of the VAULT utility to bypass the specified encryption. This keyword may be abbreviated as “NOC”.
  
• Specifying BLK64K limits the maximum block size to 64KB on the vault output tape data set.
  

Terminating USTVAULT

You may terminate an active USTVAULT sub-task if, for any reason, you cannot wait for it to complete. This is not recommended. Whenever possible you should always allow the process to complete. However, if you need to prematurely terminate a USTVAULT sub-task, you can do so by:

• Issuing the console command:

  F UPSTREAM,TERM LU=USTVLTxx
• Using the TERM line command on the UPSTREAM ISPF status display (see Controlling UPSTREAM Sub-Tasks).
• Using a console STOP (P) command to shutdown UPSTREAM itself. This will have the same effect as TERM on a USTVAULT sub-task.

When terminated by any of the preceding methods, USTVAULT completes the processing on the current backup, and then copies the temporary vault control data set to the tape before finally terminating. This orderly termination may take a few minutes to complete. If you restart the vaulting operation at a later time, USTVAULT processes only those backups that were not completed on the earlier run.

USTVAULT should not be terminated while it has an outstanding tape mount, as this may cause premature termination of the utility. If you cannot wait for the orderly termination, as previously described, then you can issue the TERM a second time to force the termination. This should only be done in extreme circumstances as the forced termination of an incomplete sub-task can occasionally cause problems within the main UPSTREAM started task.

Recovering from a Failed USTVAULT

If a USTVAULT run does not complete for some reason (perhaps an abend, or a forced termination) all backups that were successfully copied to tape will have been flagged as being successfully vaulted, so a later execution of USTVAULT will correctly bypass them.

However, the temporary vault control data set will remain on disk (and cataloged), but it will not have been copied across to the vault tape. This is not generally a problem because the USTREGEN process (see Section “Restoring from Vaulted Backups”) can access the vault control data set directly from disk, rather than off the tape.

In a full-blown disaster, however, where the disk has been destroyed and the vault control data set has been lost, you would need to run the USTREGEN utility against each vaulted backup (i.e., instead of the “lost” vault control data set) before being able to initiate the restores.

USTVAULT Reporting

You can report on vaulting activity with the USTRPORT utility (see Reporting with USTRPORT), allowing you to display information on when a vault process has been run for each USTVLTxx profile. You can also get this information via the Client panels - see the UPSTREAM Client Guide for details.

As an example, a report TYPE=BACKUP that includes the USTVLTxx and/or USTVLCxx profiles will display the data set names and tape volume serials of the tape retention data set and vault control data set created during a vault process. If the vaulting tape is multi-volume, the USTVLTxx profile will show the first tape volser in the set, and the USTVLCxx will show the last tape.

A report TYPE=VAULT, shown below, displays only details of backups that have vault copies. The report shows the data set name and tape volume serials of the vaulted copy (copies 2 through 9).

In this extract from a TYPE=VAULT report you can see the vaulted “COPY2” backups for Clients PROD01 and TESTXP1:

If you are using the UPSTREAM data encryption feature, and if you have used USTVAULT to create encrypted copies of some of your backups, you can report on those encrypted backups using the REPORT statement of USTCRYFM (Chapter 23 “UPSTREAM Data Encryption”).

Restoring from Vaulted Backups

Having created one or more copies of a primary UPSTREAM backup with USTVAULT, there are several considerations for using that vaulted backup during a restore, either at your home site or for disaster recovery. This final section discusses the issues involved in using USTVAULT-copied backups for restore purposes.

Updating the Repository (USTREGEN)

If you need to restore from a vaulted copy of a backup, the control records in the UPSTREAM repository must first be updated to point to that vaulted copy. If you have created multiple copies of your backups, you must choose which copy will be used for the restore.

Having selected the vaulted backup that will be used for the restore, the USTREGEN utility (see Updating the Repository) is then used to read the appropriate vault control data set, which was written as the last file on the vault tape. This data set contains UPSTREAM repository records, which have already been updated to point to the vaulted copy of the backup. USTREGEN reads these records from the vaulted tape and updates the UPSTREAM repository with this information. This is much quicker than having to read through the entire vaulted tape to create the new repository records from scratch.

Once USTREGEN has been run to update the repository, any subsequent restore requests will then call for the appropriate vaulted tape.

USTREGEN must be executed once for each vault tape whose records are to be updated - i.e., each vaulted backup from which you wish to do a restore. The names of the control files on the vault tapes are recorded under the USTVLCxx profile names and can be obtained with USTRPORT (as described in USTVAULT Reporting).

It is possible that your vaulted backups may contain multiple copies of the same Client file. For example, incremental merge backups will normally append each day's incremental to the back of the first incremental - i.e., the incremental backup file is cumulative. However, if you vault that backup every day, USTVAULT will copy the entire backup file, so each copy of that backup is also cumulative, with each one containing one more day's worth of incremental data. When you need to recover from those vault copies you need only to run USTREGEN against the latest USTVAULT copy of the backup.

USTREGEN has several safety catches to prevent an accidental processing of a back-level copy of a backup. First, it compares the tape volume serial in its input records with the system catalog entry for the backup; if the volser in the catalog does not match, it will reject those records with a warning message reminding you that you should do the USTREGEN only against the latest version. Also, if the backup data set in the control records is not cataloged at all, USTREGEN will not update those records in the repository. For UPSTREAM to successfully restore from the vault copies, the vault data sets must be cataloged in the system catalog.

Disaster Recovery

If you are recovering from a full-blown disaster, and you are recovering your entire UPSTREAM system, you must first complete the following steps prior to running USTREGEN to point to your off-site copies of your UPSTREAM backups:

• Recover the UPSTREAM system (e.g., the load library, configuration, logs).
  
• Restore the UPSTREAM repository data sets (see The UPSTREAM Repository) from their most recent backup.
  
• Recover copies of the z/OS catalogs in which the vault control data sets were cataloged.
  

USTREGEN Example

As described in Updating the Repository, USTREGEN can be executed in various ways including a batch job (if the UPSTREAM started task is not active) or as a console command, through the ISPF interface, or via USTBATCH (all of which require the UPSTREAM started task to be active).

Here is an example of running USTREGEN as a batch job. In this instance the USTARCH DD statement points directly to the backup itself, rather than to the vault control data set (but USTREGEN can read the required information from either source).

See Updating the Repository for full details on USTREGEN.