System Overview

An Introduction to UPSTREAM

UPSTREAM is a powerful, reliable and high-performance backup/recovery tool for your open systems data It provides a centralized, automated, and unattended backup and restore of open systems data to your z/OS mainframe.

The open systems backups can be placed on either z/OS DASD, virtual tape, or physical tape, and the backup processes can interact with your existing z/OS infrastructure, including your tape management system, your scheduling system, and your security system. UPSTREAM offers a reliable, flexible, and scalable solution for your open systems backups.

If you are licensed for the data encryption feature, your off-site or VAULT copies of your original UPSTREAM backups can be encrypted. Encryption options include AES-128, AES-192, or AES-256. Please see UPSTREAM Data Encryption for further details.

UPSTREAM Family

UPSTREAM is a member of the “UPSTREAM Family” of open systems Storage Management products. The family includes the following products, which are licensed separately:

• UPSTREAM - Enterprise Storage Management, providing Volume and File-Level backups for data stored on open systems servers and databases.
• UPSTREAM z/OS UNIX - File-Level backups for z/OS UNIX Systems Services (USS) data.
• FDRSOS for EMC and FDRSOS for IBM - Volume-Level backups for open systems data stored on either the EMC Symmetrix VMAX with the z/SOS feature, Symmetrix DMX with the ESP feature, or the IBM DS8700/DS8800 with the zDDB feature.
  
• UPSTREAM/SOS - Volume and File-Level backups for open systems data stored on either the EMC Symmetrix VMAX with the z/SOS feature, Symmetrix DMX with the ESP feature, or the IBM DS8700/DS8800 with the zDDB feature.
• UPSTREAM Reservoir - Enterprise Storage Management, providing Volume and File-Level backups for data stored on open systems servers and databases. Unlike the other members of the family, where the storage server is a z/OS mainframe, the Reservoir storage server can reside on Windows, AIX, x86 Linux, or Linux OS on IBM Z platforms. Backups created by UPSTREAM can be logically interchangeable with Reservoir and vice versa, assuming that the IBM Z media is compatible.

Client Platform Support

The data to be backed up by UPSTREAM can reside on any of the following workstation/server platforms, referred to in this manual as the “Client”:

• Microsoft Windows
  
• X 86 Linux
  
• Linux OS on IBM Z
  
• AIX on IBM Power Systems
  
• Sun Solaris
  
• Sun X86
  
• VMware
  
• z/OS UNIX Systems Services (USS) data
  

Database Support

As well as backing up standard files and documents, the UPSTREAM Client software takes advantage of database API’s and includes specially developed agents to provide online hot backup facilities for the following popular database, messaging, and groupware systems:

• Oracle
  
• IBM DB2 UDB (Universal Database Server)
  
• IBM Notes and Domino
  
• Microsoft Exchange
  
• Microsoft SQL Server
  
• SAP
  

Implementation Overview

The following diagram summarizes a typical implementation of UPSTREAM for Linux OS on IBM Z clients.

Typical Implementation of UPSTREAM

As you can see, UPSTREAM is a dual component system.

• UPSTREAM (UPSTREAM z/OS Storage Server) - This component resides on your z/OS system. It runs as a standard z/OS started task with multiple sub-tasks and it is responsible for the storage and retention of UPSTREAM backups, which can be placed on either z/OS DASD or TAPE. This is also where the main configuration control files reside, together with the repository and history files relating to your UPSTREAM backups.

  This manual describes the UPSTREAM z/OS Storage Server.

• UPSTREAM Client (Client Software) - This component resides on the client to be backed up. It is responsible for accessing open system files and databases during the backup and restore process.

  The UPSTREAM Client Guide describes the UPSTREAM Client software.

UPSTREAM Client Communications

During a backup or restore operation, data is transferred between the two components across one or more connections. UPSTREAM provides multiple choices for the type of connection that can be used, allowing optimum data transfer rates, based on the capabilities of the hardware available.

UPSTREAM z/OS uses Transmission Control Protocol/Internet Protocol (TCP/IP) to communicate with the UPSTREAM client. For users looking to backup Linux OS on IBM Z data, UPSTREAM utilizes Hipersocket connections, if available and configured on your CPU.

Users that have either an EMC Symmetrix VMAX, Symmetrix DMX, or the IBM DS8700/DS8800 systems with the appropriate microcode features (please see The UPSTREAM Family), can take advantage of the enhanced performance UPSTREAM/SOS offers using fiber networks.

Additional Performance Features

Aside from using fastest possible links, UPSTREAM also includes other performance-enhancing features:

• Data Compression - Prior to transmitting data from the UPSTREAM Client to the mainframe, UPSTREAM can optionally compress that data, which can potentially improve backup elapsed times by reducing the amount of data sent across the network. Due to their highly compressible nature, this may be more appropriate for data base files. Multiple levels of compression are available.
  
• Merge backups - Using a sophisticated technique, UPSTREAM can construct a complete Synthetic full backup of a Client without the need to transmit all the files across to the z/OS Storage Server. (See Deferred Merge Backupsfor more details).
  

UPSTREAM Interfaces

Various interfaces are provided to the UPSTREAM system, both from the UPSTREAM z/OS Storage Server, and also from the Client software. UPSTREAM operations can be controlled through:

• ISPF Panels - A full set of ISPF panels is provided with the UPSTREAM z/OS Storage Server component.
  
• z/OS Batch Job initiation - An z/OS batch process can be initiated via the USTBATCH program. (See z/OS Initiation with USTBATCH).
  
• UPSTREAM Web Portal - The Web Portal is a hosted interface that can be installed on any Java Based Server such as USS, Windows, or Linux. It provides a new alternate approach to backup management, client maintenance, and end user restore interfaces (see the UPSTREAM Client Guide)
  
• UPSTREAM Director Interface - The Director is a Java-based interface that can be installed and accessed from any Client in the network. The Director is used by administrators as a “dash board” to monitor and control the overall system.
  

Unattended Secure Operations

Despite the various interfaces that are available to initiate UPSTREAM operations, the system is primarily designed to run in an unattended mode. On a day-to-day basis, UPSTREAM can be transparent in its normal operations. Should a backup fail to complete (e.g., because of a communications failure) UPSTREAM can restart the incomplete backup from the point of failure.

Managing Your Backup Media

UPSTREAM includes several utilities (e.g., USTVAULT and USTMIGRT) for copying backups and for migrating them from disk to tape. With these utilities, you can:

• Create additional or “vault” copies of your backups for long term retention, or off-site storage.
  
• Migrate/merge multiple disk/tape backups onto a single tape or tape set.
  

See Copying Backups with USTVAULT and Migrating Backups from Disk to Tape for more details.

Data Encryption

UPSTREAM includes a data encryption option as an additional cost add-on to your existing UPSTREAM license. With the encryption option licensed and enabled, you can use USTVAULT to create additional encrypted copies of some/all of your primary “copy 1” backups, which may be intended for off-site transportation and storage. The “copy 1” backups themselves cannot be encrypted.

You can chose from among three encryption algorithms, each driven by an encryption key that can either be user-specified, or (as recommended by Innovation) automatically generated by UPSTREAM. The encryption key used for each USTVAULT copy is stored in a special “key file” on disk, which is itself protected against unauthorized access by your z/OS security system. The key file is automatically read by UPSTREAM in order to restore from an encrypted copy of a backup.

In a disaster recovery scenario, the key file must first be securely transported to the DR site and restored to disk before UPSTREAM can restore encrypted copies of any backups. As an alternative, if the key file cannot be made available, the individual encryption keys (if known) can be manually provided to the restore process.

An optional master key is also supported, which is used to create an encrypted copy of the actual key used to encrypt the data. This encrypted key is then saved on the backup data set. The master key can then be used to decrypt any copy of a backup that has been encrypted by USTVAULT, in the event that the key file is either not available, or the actual keys cannot be provided manually.

See UPSTREAM Data Encryption for full details on the UPSTREAM data encryption option.

Ancillary Capabilities

In addition to the primary functions of backup and restore, UPSTREAM includes several other ancillary capabilities:

• File Transfer - UPSTREAM can be used to transfer files from the Client to the z/OS Storage Server, or vice versa. No additional software or communication connection is required. Text files may be translated between ASCII on the Client and EBCDIC on the z/OS Storage Server. The transfers can be requested by either the z/OS Storage Server or by the Client. See File Transfer.
  
• Running an UPSTREAM Client User Process - UPSTREAM also provides a facility for a “user process” to be initiated on a Client machine. This can be any program, batch file, or script that can execute on the Client. Generally, although not always, this process is used to control some aspect of an UPSTREAM operation, such as a backup or a restore. An example may be a process to quiesce update activity to a database prior to backing it up. See Running an UPSTREAM Client User Process.
  

UPSTREAM Backup Fundamentals

The following sub-sections describe the fundamentals of the UPSTREAM backup process:

• Backup Architecture
  
• UPSTREAM Backups to z/OS Tape and DASD
  
• Full and Incremental Backups
  
• Merge Backups
  
• Backup Scenarios - Part1
  
• Deferred Merge Backups
  
• Backup Scenarios - Part2
  

Backup Architecture

UPSTREAM maintains a repository at the z/OS Storage Server containing details of all current backups. This information includes file names, dates/times, locations, and other attributes specific to each backup. (See The UPSTREAM Repository).

All UPSTREAM backups are identified through a hierarchical structure:

• Backup Profiles - These are unique, user-specified definitions (see UPSTREAM Profiles) that identify a backup and control the options associated with that backup. A backup profile has a 1- to 8-character name. Every Client (i.e., workstation/server running the Client software) typically has one backup profile associated with that workstation/server only. If there is a very large amount of data to be protected, you may want to use more than one profile. Backup profiles are specific to a given server and are not used on more than one server.
  
• Version Dates - Each instance of a backup for a given backup profile is identified by a “Version Date”. This is the date and time (on z/OS) that the backup was created. Combined with the backup profile name, the “Version Date” allows the unique identification of a particular backup instance.
  
• File Specifications (“file specs”) - The actual files, directories, or whole drives that are to be backed up (or excluded) within a backup profile are identified in one or more include/exclude file specifications, often referred to as “file specs”.
  

Examples of “include” file specs (to include files in a backup) might be:

An “exclude” file spec (to exclude files from a backup) might look like the one shown below, where all “.exe” files in the previously selected “F:\agenda” folder are being excluded:

F:\agenda\*.exe

UPSTREAM Backups to z/OS Tape and DASD

UPSTREAM supports backups to z/OS DASD, physical tape, and virtual tape.

• Sequential Tape Backup - A sequential tape backup is written as a sequential data set directly to z/OS physical or virtual tape. The data set is dynamically allocated at the beginning of the backup process according to parameters specified in the associated backup profile. Either a data set prefix or a z/OS GDG base name can be specified.
  

• Sequential Disk Backup - A sequential disk backup is written as a sequential data set directly to z/OS DASD. The data set is dynamically allocated at the time of the backup according to parameters specified in the backup profile, which may also include details of DASD unit name or volume serial to be used. UPSTREAM automatically performs the space allocation for the disk data set according to an estimate provided from the Client software at the beginning of the backup process.
  For SMS installations, a default Storage Class, Management Class, and Data Class can be passed to the ACS routines, though these may be later overridden by SMS.

  Sequential disk backups may be managed by a DASD management system, such as BMC’s ABR, CA Disk™ Backup and Restore, or IBM DFSMShsm. This includes both backup and migration. In the case of migration, if a UPSTREAM sequential disk backup is migrated by a data management system, an auto-recall is required to recover the data set to primary disk before UPSTREAM can once again access it. For this reason, we recommend that you use the disk-to-tape migration facilities of USTMIGRT (see Migrating Backups from Disk to Tape) to move your UPSTREAM backups from disk to tape, instead of your z/OS DASD management software.

Full and Incremental Backups

UPSTREAM's flexible backup schemes can be used to protect an entire server, a database, or simply individual folders or files.

These backups are then scheduled to run at either a “full” level, where all files are backed up, or at an “incremental” level, where only changed files are backed up.

A first time full backup copies all of the files from the client server to z/OS DASD or tape.

An Incremental backup copies only the changed files from the client server. This reduces the time required to take daily backups. During the incremental process, our client for Windows uses the Volume Shadow Copy Service (VSS) to “snapshot” the volume(s), detect the changed files, and eliminate file contention. In UNIX and Linux systems, the UPSTREAM client can use its own “local incremental database” (INCRDB) to record the files it has backed up, or can compare the file modification date to the last backup, to determine what files are to be included in the next incremental backup.

Below, are two screens from the Director interface (see the UPSTREAM Client Guide) that is being used to define an UPSTREAM backup task.

On the left-hand screen we have an “include” file spec to perform a full backup “\\.” Note also that we have specified an “exclude” file spec for the directory named ‘1’.

On the right-hand screen is the Target File Explorer used to include or exclude specific files or directories.

Director Interface

Merge Backups

A typical backup cycle may be a weekly first time full backup followed by daily incremental backups. However, because of the potentially large volume of data, first time full backups can consume valuable client-side resources and impose larger than desired loads on the network. UPSTREAM's efficient “Synthetic Merge Backup” mechanism allows administrators to obtain the benefits of a full backup, but without the aforementioned overheads.

To create the full backup, UPSTREAM backs up only the changed files from the server. What makes this backup different from an incremental backup is that it also sends a copy of the directory entries of the unchanged files to the z/OS Storage Server. Then, instead of using network and client-side resources to transmit these unchanged files to the z/OS Storage Server, UPSTREAM simply retrieves the most recent copy of each file from its prior backup.

So, using CPU resources only at the z/OS Storage Server end of the operation, UPSTREAM then combines all of the changed files, and, optionally, retrieves copies of the unchanged files, to create a new “merged”, or synthetic, full backup, which contains an up-to-date copy of every file from the Client.

The Merge backup advantages are:

• The synthetic full backup is created using client-side resources that are only marginally greater than on a standard (i.e., non-merge) incremental backup.
  
• It is easy to use, understand and manage. The benefits of a merge backup is that all the complexity is behind the scenes; it is actually as easy or easier to use than non-merge backups.
  

The Merge Process

A single backup profile name is used for both the full and incremental merge backups. It is recommended that this backup profile represent an unchanging group of file specs (e.g., it identifies a whole server, a range of drives or file systems, or an application). The facility is flexible enough for you to be able to add or remove folders or drives, however it is not recommended that you use a backup profile for more than one entity or server.

The Merge process requires that you perform a first-time full backup of the file specifications that you wish to maintain. In this backup you copy all the files. However, once you've created this full backup, it does not need to be repeated. Incremental merge and full merge backups are performed from this point onward.

The first backup following a full backup always begins a new tape or disk file. This is typically an incremental backup and copies only the changed files to the UPSTREAM z/OS storage server. By default, incremental merge backups to tape are appended to the previous incremental tape backup data set, while incremental backups to disk create new disk files every time.

Another full backup is performed, but instead of a new First Time Full backup, a synthetic Full Merge backup, Deferred Full Merge backup, or Synthetic Differential Full Merge backup is performed. By default, full merge backups are appended to the end of the incremental backup file (if the backup is on tape), or a new file is created (if the incremental backups are on disk).

The following section describes the different types of Full Merge backups in greater detail.

Full Merge Backup Work-Flow

There are three different options using the Merge synthetic full backup process. The “output” of any of these options can be used as “input” for a full system restore. Please refer to Building the USTBATCH Job, and z/OS Initiation with USTBATCH, for further details on creating and running a USTBATCH initiated backup.

Full Merge Backup

This synthetic full backup is a complete backup of all files and meta data on the server at the time the backup is run. The backup of the client server begins by identifying the changed files since the last backup. The changed files and a directory listing from the client server are copied to the UPSTREAM z/OS Storage Server. The remainder of the backup data is not copied from the client server, but from the prior backups on tape or DASD. This results in a co-location of all changed and unchanged files from the client server. The backup uses client resources that are only marginally greater than of an incremental backup. This process significantly reduces the resources used on the client at the time of the backup and is particularly suited for backups that have a high rate of file change.

Full Merge Backup

1. A first time full backup is run producing a backup data set containing every file that existed on the server at the time of the backup. This backup is required to begin the Full Merge process.
2. A series of incremental backups are run, copying only the changed files and placing them in the incremental backup data set. In the example below, files “A, F, K, P, and R” have been changed in the incremental backups taken during the week.
   
3. A Full Merge backup is run specifying MERGE=1 in the backup JCL control cards and the profile MERGE option set to “YES”. The backup copies the files that have changed since the last backup and transfers a directory listing of all of the files on the client file system(s) to the UPSTREAM, z/OS storage server.
   
4. The backup then copies, from prior backups, not from the client server, the files that have changed since the last full backup (“A, F, K, P, and R”) and copies the unchanged files from the last full backup (B, C, D, E, etc), “merging” or co-locating all of the files into the backup data set. The backup is now complete and all of the client files have been copied to the backup data set on tape.
   
5. The process repeats the following week with files “B, G, L, N, and S” changing in the incremental backups taken during the week.
   
6. As with the prior week, a new Full Merge backup is run, copying any files that have changed since the last backup and transferring a directory listing of all of the files on the client file system(s) to the UPSTREAM, z/OS storage server.
   
7. The backup then copies, from prior backups, the files that have changed since the last full backup “B, G, L, N, and S” and copies the unchanged files from the last full backup (A, C, D, E, F, etc), “merging” all of the files into the backup data set. Lastly, the comparison of the client directory listing is done and the backup is now complete. All of the client files have been copied to a new backup data set on tape.
   

Deferred Full Merge Backup

Deferring the co-location of the client files, The Deferred Full Merge backup is a Full Merge backup with an additional step. This new step co-locates the changed and unchanged files similar to that in the Full Merge. However, instead, this step is “deferred” until a later point in time when tape and other mainframe resources are more available.

Deferred Full Merge Backup

1. A first time full or full merge backup has previously been run producing a backup data set containing every file that existed on the server at the time of the backup. One of these backups is required to begin the Deferred Full Merge process.
   
2. A series of incremental backups are run, copying only the changed files and placing them in the incremental backup data set. In the example below, files “A, F, K, P, and R” have been changed in the incremental backups taken during the week.
   
3. Deferred Full Merge backup is created by running a regular Full Merge Backup but with the profile MERGE option set to “DEFER”. The process is as follows:
   
   • MERGE=1 is specified in the backup JCL control cards.
   • DEFER is specified for the MERGE option in the backup profile.
   • Server Files changed since the last incremental backup are copied into the Deferred Full backup data set.
   • Unchanged server files have pointers created to prior backups, they are reflected in the diagram with grayed out file names.
4. The Deferred Full Merge backup runs just slightly longer than a normal incremental backup, using significantly less resources than a traditional first time full backup. Once this backup completes, the backup is viable for a complete system restore, even though the client files are not yet co-located. Any files from prior backups that are needed for a restore are taken from those backups.
   
5. At some later point in time when tape and other mainframe resources are available, the USTMERGE utility function is executed which “merges” and co-locates all of the incremental data (2) and the unchanged files from the prior first time full or full merge data (1). The final result is that the completed Deferred Full Merge has a copy of EVERY file on the server at the time the deferred full backup was run. USTMERGE is a batch operation and can process more than one profile per run. This feature consolidates deferred full merge backups for multiple profiles onto a single tape set saving tape over that needed for a regular full merge or first time full backup.
   
6. The process repeats the following week with files “B, G, L, N, and S” changing in the incremental backups taken during the week.
   
7. As with the prior week, a new Deferred Full Merge backup is run, copying any files that have changed since the last backup and a transferring directory listing of the files on the client file system(s) to the UPSTREAM, z/OS storage server.
   
8. A USTMERGE utility function is executed which again “merges” and co-locates all of the incremental data (5) and the prior completed deferred full merge data (4). The final result is that the completed deferred full merge has a copy of EVERY file on the server at the time the deferred full backup was run.
   

Synthetic Differential Full Merge Backup

The Synthetic Differential Full Merge backup copies only the most recent version of ALL of the changed files since a reference full backup, the “MASTER” full backup was taken. This is identical to the Deferred Full Merge backup with the exception that the USTMERGE process does not co-locate all of the changed and unchanged files but instead copies only the files that have changed since the MASTER full backup. Using this option, the USTMERGE process completes in far less time than with the Deferred Full Merge and uses the least amount of z/OS CPU cycles and tape resources of any of the Merge backup types. Since the unchanged files are kept on the “Master” full backup and not copied forward to a new Full Merge backup, thus eliminating the tape to tape copy of the unchanged files, this process is particularly suited for backups that have a low rate of file change.

Synthetic Differential Full Merge Backup

1. Either a completed Deferred Full, Full Merge or First Time Full backup is run producing a backup data set containing every file that existed on the server at the time of the backup. This is known as your “MASTER” backup and starts the Synthetic Differential Full Merge process. Its retention MUST be long enough to keep it from expiring while the later backups that are dependent on it, remain cataloged to the UPSTREAM z/OS storage server.

1. A series of incremental backups are run, transferring only the changed files over the network and placing them in the incremental backup data set. In the example below, files “A, F, and K” have been changed in the incremental backups taken during the week.
   
2. A Deferred Full Merge backup is created by running a regular Full Merge Backup but with the profile MERGE option set to “DEFER”. The process is as follows:
   
   • MERGE=1 is specified in the backup JCL control cards.
   • DEFER is specified for the MERGE option in the backup profile.
   • Server Files changed since the last incremental backup are copied into the Deferred Full backup data set, as example “J and W” illustrate.
   • Unchanged server files have pointers created to prior backups, these are reflected in the diagram with grayed out file names.
   • As explained in the Deferred Full Merge backup in Figure 2.4:, the backup runs just slightly longer than a normal incremental backup, using significantly less resources than a traditional first time full backup. Once this backup completes, this backup can now be used to restore any and all files, even though the client files are not yet co-located. Any files from prior backups that are needed for a restore are taken from those backups.

3. At some later point in time when tape and other mainframe resources are more available, the USTMERGE utility function is executed with the MASTER option. Unlike regular Full Merge or completed Deferred Full Merge backups that contain co-located copies of all changed and unchanged files on the server, the unchanged files from the prior full backup are not co-located on this full backup. Instead, the backup contains:

   • Reference pointers to the unchanged client files that were backed up in the Master full backup.
     
   • The most recent version of ALL of the CHANGED files from the incremental backups since the prior MASTER Full backup.
     

   In this example, only “A, F, J, K, and W” are on the backup data set. This is where the “differential” name comes from. A full system restore at this point would access the Deferred Full Merge AND the MASTER Full backups.

   WEEK TWO

4. The following weeks incremental cycle continues with files “B, G, and L” changing in the incremental backups taken during the week. File “K” is deleted from the server and is noted in the UPSTREAM repository.

5. As with step 3 above, a new Deferred Full Merge backup is run copying any files that have changed since the last backup as example “R” illustrates above. Then pointers are created to all the other UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.
   

6. As with step 4 above, the USTMERGE utility function is executed with the MASTER option. Reference pointers to the unchanged client files that were backed up in the Master full backup and ALL of the CHANGED files from the incremental backups since the prior MASTER Full backup are stored on this backup data set. In this example, only “A, F, J, W, B, G, L, and R” are on the backup data set. Another differential backup has been created.

   
   

   WEEK THREE

7. Week 3 begins and the Incremental cycle continues with files “N and S” being changed and subsequently backup and “Y” is deleted.

8. Another Deferred Full Merge is run and any files that have been changed since the last backup are copied, as example “N and S” illustrate. Then pointers are created to all the UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.
   
9. The USTMERGE command is executed with the MASTER option. ONLY the most recently changed version of individual changed Incremental files is co-located on to the full. Files from the prior MASTER FULL are NOT co-located and the pointers created in the Deferred Full Merge are copied. The USTMERGE process ends quickly. The final result is the completed Full Merge now contains the most recent copy of every file on the server that has CHANGED since the time the MASTER full was run. In this example, only “A, F, J, W, B, G, L, R, N, S, and X” are on the backup data set. A 2nd differential that now contains the 3rd weeks changes has been created!
   
10. Week 4 begins and the incremental cycle continues with files “C, D” being changed and subsequently backed up. File “T” has been deleted, so this is reflected.
    
11. Another Deferred Full Merge is run and any files that have changed since the last backup are copied, as example “Q” illustrates above. Then pointers are created to all the other UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.
    
12. Since a sufficient amount of incremental changes have occurred to warrant creating another MASTER:
    
    • The USTMERGE command is run WITHOUT the MASTER option.
      
    • The most recently changed Incremental files are co-located on to the full.
      
    • All unchanged files from week 1, the original MASTER full, are co-located to create this new MASTER FULL.
      

The final result is that the completed Full Merge now has the most recent copy of every file on the server. In this example, the most recent version of all changed files from this weeks incremental backups and prior deferred to master backups, are co-located onto the new MASTER Full. Deleted files are reflected.

The following scenarios describe the merge backup process in more detail.

Backup Scenarios - Part 1#

Scenario #1: Full on Tape and Incremental Backups on Disk or Tape

The diagram below shows how a “tape only” deferred to master (Synthetic Differential) backup system works. The advantages of this scenario are:

• Long running “Full” backup run once and the tape is retained as a processing Master Copy with a very long retention period such as 365 days or a GDC with 2 or more generations for a very long period of time.
  
• Daily Incremental backups run backing up file changes since the prior backup.
  
• Once per week, a Deferred to Master Merge runs that consolidates these daily incremental backups onto a new Synthetic Differential Tape/Disk.
  
• This repeats over time with the Deferred to Master Merge process constantly consolidating Incremental and Synthetic Differential Tapes/Disks to a new Synthetic Differential Tape/Disk.
  
• The original Full Backup remains in the system, so the daily and weekly processing only process files that changed since the Full Backup was created. This greatly reduces what is typically a long running tape to tape merge process.
  
• At some intervals – based on how large the Synthetic Differential Tape / Disk is, or how long the weekly merge process runs – a new Full Master might be required to “reset” the weekly tape. This can either be done as a First Time Full Backup or as a regular Full Merge Backup job.

When you run your first-time full backup, a new tape is created that holds all the data selected from the Client (Tape 1). The first incremental backup then creates a new tape (on Tape 2). Subsequent incremental backups are appended to the end of that tape file on Tape 2.

After your first-time full backup, all subsequent backups are then Incremental backups.

During Synthetic Differential (Merge to Deferred Master) processing, the tape/disk holding the previous Synthetic Differential backup (Tape 1) is mounted as well as the tape holding the incremental backups (Tape 2) and a new Tape 3 is mounted to become the new Synthetic Differential Backup tape.

Any files that have not been changed since the last Deferred to Master job are copied from Tape 1 to Tape 3, unless a more recent file are in one of the incremental backups already on Tape 2, in which case the most recent file gets copied from Tape 2 over to Tape.

Scenario #2: Full Backups on Tape, Incremental Backups on Disk

The diagram below shows how a “tape only” merge backup system would work. The advantages of this scenario are:

• No intermediate disk requirements. Data goes directly to tape without having to be staged through disk. This saves on z/OS disk space.
  
• Good for large volumes of data.
  
• Only one tape is created per backup cycle (usually weekly). This saves on tape management.

When you run your first-time full backup, a new tape is created that holds all the data selected from the Client (Tape 1). The first incremental backup then creates a new tape (on Tape 2). Subsequent incremental backups are appended to the end of that tape file on Tape 2.

After your first-time full backup, all subsequent full backups are then “full merge” backups.

In a full merge backup, in the preceding example, the tape holding the previous full backup (Tape 1) is mounted, as well as the tape holding the incremental backups (Tape 2).

Any files that have not been changed since the last backup are copied from Tape 1 to Tape 2 unless they are in one of the incremental backups already on Tape 2, in which case they are simply recorded as being part of the new full merge backup and the incremental backup. They are not copied into the new full. UPSTREAM then requests that the Client software send up any files from the directory listing that could not be matched against previous backups.

Scenario #3: Full Backups on tape Incremental Backups on Disk

The diagram below illustrates a scenario similar to Scenario #1: Full on Tape and Incremental Backups on Disk or Tape, but the incremental backups are now stored on disk. You may want to choose this option if you have sufficient z/OS disk space and do not wish to mount the backup tapes each day. In addition, file restores from the incremental backups can be quicker because they do not involve a tape mount.

The initial process is similar to Scenario #1: Full on Tape and Incremental Backups on Disk or Tape. When you run your first-time full backup, a new tape data set is created that holds all the data selected from the Client (Tape 1).

The first incremental backup then creates a data set on disk (File 1), and subsequent incremental backups create additional new data sets on disk.

When you run a full merge backup, the tape holding the prior full backup (Tape 1) is mounted, as well as the output tape for the new full merge backup (Tape 2).

As in Scenario #1: Full on Tape and Incremental Backups on Disk or Tape, the Client software sends up all the files that have changed since the last incremental, as well as a directory listing. UPSTREAM then copies to the new full backup (on Tape 2) all the files that were included in the directory listing, obtaining their most current backup from the incremental disk files (File 1 and File 2) or from the last full merge backup (Tape 1). It then requests that the Client software send up any files from the directory listing that could not be located on those previous backups.

Using USTMIGRT

If you do not have enough disk space available to hold a week's worth of incremental backups for all UPSTREAM Clients, the USTMIGRT migration utility can be used move accumulated disk backups to tape. (See Migrating Backups from Disk to Tape).

Scenario #4: Full and Incremental Backups on Disk

The diagram below illustrates a scenario similar to Scenario #2: Full Backups on Tape, Incremental Backups on Disk, but where all backups are now stored on disk. You may want to use this scenario for small backups or where restore speed is important.

Figure 2.10:

When you run your first-time full backup, a new z/OS disk file is created that holds all of the data from the Client (File 1 in the figure). Each subsequent incremental backup creates a new z/OS disk file (File 2 and File 3).

When you run a full merge backup, the Client software transmits the changed files, that are written to the new full merge backup file (File 4).

UPSTREAM then takes the directory listing transmitted from the UPSTREAM Client and copies the most recent backups of all other files listed in the directory, from the incremental backups (File 2 and File 3) and the last full merge backup (File 1).

It then requests that the Client software send up any files from the directory listing that could not be located on those previous backups.

Deferred Merge Backups

Although the UPSTREAM merge backup process described above is the most efficient type of full backup that you can create, there might be limitations on the use of this process in some installations, especially when sequential tape is being used:

• The full merge backup process can require two tape drives: one for the output and one to read the previous backups. There may not always be sufficient tape drives to satisfy this need, especially if multiple backups are run at once.
  
• If automated tape libraries are not used or are unavailable, there might not always be an operator around to satisfy requests for specific tape mounts.
  

The UPSTREAM “Deferred Merge” process is designed to circumvent these limitations.

If the MERGE=DEFER option is set in the backup profile, the deferred merge backup does all of the steps described in the previous scenarios. Normal processing also occurs if the required previous backups of files are on disk. However, if any previous backups are on tape, the deferred merge backup does not copy those backups to the new full merge backup. Instead, it simply records in the UPSTREAM repository that the copy process for that backup has been “deferred”.

This means that during a deferred full merge backup:

• It is not necessary to mount any previous input tapes
  
• It still needs to mount an output scratch tape (unless the backup profile is setup to write the full backup to sequential disk).
  

Running USTMERGE

When the deferred merge process is used, the backup is not truly complete until the deferred files have been properly copied to the new full merge backup. The UPSTREAM utility program USTMERGE is used to complete this processing. (See Completing Deferred Merge Backups.)

Normally, you want to run USTMERGE as soon as possible after the deferred merge backup – as soon as tape limitations or operator availability permits. However, while an execution of USTMERGE is pending, UPSTREAM can still restore data from the backup in deferred status.

When run, USTMERGE identifies backup profiles whose most recent full merge backup was done with MERGE=DEFER in effect, and for which the copy process is still pending. It then completes those backups by:

• Copying the full backup data set from disk to tape (if the initial full merge backup was taken to sequential disk). The backup profile must be enabled for both sequential disk and sequential tape backups.
  
• Mounting the required previous full and incremental backup tapes and copying to the new full backup any Client files that were deferred during the full backup.
  

Utilization of the deferred merge process leads to several more backup scenarios.

Backup Scenarios - Part 2#

Scenario #5: Deferred Merge Backups on Disk

The following diagrams show a deferred merge process where the backups are directed to disk. If you have sufficient disk space to keep a week's worth of incremental backups on disk, this is the most efficient way to use deferred merge.

Deferred Merge Backups to Disk

In the preceding scenario:

• Each daily incremental backup is taken to disk (File 1 and File 2).
  
• The weekly full backup, with MERGE=DEFER set in the backup profile, is also taken to disk (File 3).
  

As in previous scenarios, the deferred merge process backs up all the updated files from the Client and then copies the other files that were updated during the week from the appropriate disk-based incremental backups. However, any files required from the previous full backup are “deferred”, since that full backup resides on tape. The interim full backup on disk is not much larger than the daily incremental data sets, since it only contains the accumulated Client files that were updated during the week.

When USTMERGE is executed (below), it copies the full backup from disk (File 1) to tape (Tape 2). It also mounts the previous full backup (Tape 1) and copies any Client files that were “deferred” during the actual backup. The new full backup on tape is now complete and contains a copy of every file that was on the Client at the time of the backup.

USTMERGE Completes Full Backup On Tape

Scenario #6: Deferred Merge Backups on Disk (In Conjunction with USTMIGRT)#

This final scenario shows a deferred merge process where the backups are directed to disk, but the incremental backups are migrated to tape on a daily basis using the USTMIGRT utility. (See Migrating Backups from Disk to Tape) This scenario might be used if you do not have sufficient disk space to accumulate a week's worth of backups before USTMERGE is run.

This scenario also illustrates the more realistic processing of USTMERGE, in that a number of Client backup profiles (for Client #1, #2, and #3) are being processed in one execution, creating multiple data sets on the output tapes, each containing data belonging to one backup profile.

Incremental Backups Migrated to Tape (USTMIGRT)

Each daily incremental backup is written to disk, shown as “Today's Incrementals”. The USTMIGRT utility is then run daily to move to tape the incremental backups for a group of backup profiles (via GROUPID – see Profile Grouping), freeing up the disk space in the process. The FORWARD option of USTMIGRT has been specified, causing it to read the previous incremental backups on tape (Tape 1) and to forward merge them onto the new output tape (Tape 2).

The result is that the tape produced daily by USTMIGRT contains all of the incremental backups taken for these backup profiles since the last full merge backup. Note that you could get a similar result by taking the incremental backups directly to tape (see Scenario #1: Full on Tape and Incremental Backups on Disk or Tape), but this would put each backup on a separate tape, which would require more tape handling during USTMERGE execution.

The weekly full backup is also taken to disk with the MERGE=DEFER option set in the profile. The merge backup process backs up all updated files from the Client, and it requests any files for which it does not have a current backup. All other files are marked “deferred”, since the backups reside on tape.

USTMERGE Completes Full Backup on Tape

When USTMERGE is executed for the same set of backup profiles, it copies the full backups (shown as “Deferred Full Merge”) from disk to tape. It also mounts the previous full backup on tape (Tape 1), plus the most recent accumulated incremental backup tape (Tape 2), and copies any Client files that were deferred during the actual backup.

The new full backup on tape (Tape 3) is now complete, containing a copy of every file that was on the Client at the time of the backup.

You could optionally write the deferred full merge directly to tape. The processing would be similar to preceding, except that USTMERGE (executed with its NEWTAPE option) would simply add the required deferred files from the previous full and incremental backup tapes to the end of the new full backup tape. This also creates a separate tape for each profile that requires more output scratch tape mounting during USTMERGE execution.

UPSTREAM System Components

This chapter provides an overview of the main components of the UPSTREAM system, and also describes some of the system maintenance utilities.

The z/OS Started Task (USTMAIN)

The z/OS started task is the major functional component of UPSTREAM. It contains the processing and communications interface routines to perform the following functions:

• Backup and restore operations
  
• File transfer operations
  
• Communications handling with the Client software
  
• System console communications
  
• Event and status logging
  
• Security authorization
  
• Registered name processing and resolution
  
• Error handling and restart functions
  
• Backup retention and expiration cleanup
  
• Inquiry processing
  

As described in UPSTREAM Operation, the UPSTREAM started task accepts standard commands from the system console (e.g., the STOP (P) command and the MODIFY (F) command) to control its operation. This includes:

• Displaying the current activity
  
• Turning the diagnostic trace facilities on or off
  
• Running utility functions
  
• Starting/stopping UPSTREAM
  

The Configurator (USTCONFG)

The UPSTREAM Configurator program USTCONFG (see UPSTREAM Configurator) maintains a configuration file that is loaded by the z/OS started task (USTMAIN) at start-up time. The configuration file contains:

• General Options, controlling the overall functionality of UPSTREAM (e.g., which security level is to be used).
  
• Profile Options, controlling the behavior of most UPSTREAM functions, including backups, restores and utility operations.
  

The configuration file can be maintained and updated through:

• The USTCONFG TSO/ISPF dialog
  
• The USTCONFG batch program (see UPSTREAM Configurator)
  
• The Client interface (see UPSTREAM Client guide).
  

Repository Maintenance #1 (USTMAINT)

Details of all backups are maintained in the UPSTREAM z/OS repository (see The UPSTREAM Repository).

USTMAINT is a utility program for removing the records of old/expired backups from that repository. When your DASD and/or Tape Management Systems expire and delete your UPSTREAM backup files, USTMAINT detects that they are no longer cataloged and removes their related information from the repository. USTMAINT can also delete records from the UPSTREAM history log (MAXHIST) and remove unused registered name records.

USTMAINT is automatically executed during the initialization of UPSTREAM (USTMAIN), but it can also be initiated via an z/OS console command (see Start the UPSTREAM z/OS Storage Server) or through the USTBATCH interface (see z/OS Initiation with USTBATCH).

Repository Maintenance #2 (USTREORG)

As new backup records are added and older ones subsequently deleted from the UPSTREAM repository, a periodic reorganization is needed to maintain efficiency. USTREORG is a utility to dynamically reorganize the repository files without having to shutdown UPSTREAM. See Repository Maintenance for documentation on USTREORG.

z/OS Scheduler (USTSCHED)

The UPSTREAM Scheduler, USTSCHED, (see UPSTREAM Scheduler) is executed as a sub-task of the main started task (USTMAIN). It can be automatically invoked when the main task is initialized, or it can be manually stopped/started at any time via a console command.

USTSCHED issues z/OS console commands at the defined times and dates to control UPSTREAM operations (e.g., backups and restores). It can also be used to control other events on either the z/OS Storage Server, or an Client (e.g., quiesce a database prior to backing it up).

The schedule definitions, which are maintained though the UPSTREAM TSO/ISPF dialogs, may be updated and refreshed at any time. Multiple copies of USTSCHED can run concurrently, each with a unique schedule definition.

UPSTREAM operations can also be controlled by your existing z/OS scheduler (see Batch Initiation Utility (USTBATCH), or, if preferred, through the scheduler panels included with the Client software (see the UPSTREAM Client Guide).

Batch Initiation Utility (USTBATCH)#

USTBATCH (see z/OS Initiation with USTBATCH) is a utility that runs as a submitted batch job on the z/OS mainframe to initiate backup or restore operations for one or more Clients. It can also be submitted through the UPSTREAM scheduler (USTSCHED), or through your existing z/OS job scheduling system.

USTBATCH parameters specify the Client(s) for which the operation is to be initiated, identified by the Client's TCP/IP network address, or though the UPSTREAM registered name service (see System Overview).

Parameters controlling the actual UPSTREAM operation (e.g., backup, restore) can either be supplied directly by USTBATCH, or they can be picked up from a “parameter file” defined on the Client. The UPSTREAM TSO/ISPF dialog includes facilities for generating, saving, and executing USTBATCH jobstreams.

Backup Utilities and Maintenance

Several utilities are provided within UPSTREAM to assist with the control and maintenance of your backups and the media on which they reside.

USTVAULT

USTVAULT is a utility program to create secondary copies (on tape) of sequential disk or sequential tape backups for selected backup profiles. These secondary copies can be retained on-site as a backup to the primary copy, or they can be sent off-site for safe storage for disaster recovery.

USTMIGRT

As illustrated in scenario #5 earlier (see Scenario #6: Deferred Merge Backups on Disk (In Conjunction with USTMIGRT)), USTMIGRT allows you to do backups from multiple Clients, temporarily storing them on disk, and then consolidating them later onto a single tape or tape set. This improves efficiency, since the number of concurrent backups is not restricted to the number of available tape drives.

USTREGEN

UPSTREAM control records in the repository normally point to the original “copy 1” backup data set, so that all restore requests use that primary copy. When a secondary copy is created by USTVAULT, it also creates a copy of the control records, updated to point to the secondary copy. These updated records are placed in a special “vault control data set” that is added to the end of the copied tape (they are not recorded in the UPSTREAM repository).

If you need to access the copied backup at any time (e.g., at a disaster recovery site), the USTREGEN utility is used to quickly update the UPSTREAM repository to point to the copied backup, enabling all restores to use that copy.

USTREGEN (see Updating the Repository for a full description), is also useful for:

• Re-creating the appropriate control file pointers for copies of UPSTREAM backups that have been created using a utility other than USTVAULT.
  
• Re-creating control file pointers for a backup that has been deleted from the repository, either in error, or legitimately by USTMAINT.
  

If you have the UPSTREAM encryption option licensed and enabled (see UPSTREAM Data Encryption), and you need to REGEN an encrypted “copy-2” backup, USTREGEN decrypts the necessary information to add the backup to the UPSTREAM. The actual backup data and control information remain encrypted.

UPSTREAM Reporting Tools

UPSTREAM includes several useful reporting tools.

General Reporting Tool (USTRPORT)

USTRPORT is a general, batch-driven reporting program, that can read UPSTREAM history records and report on them in a variety of ways.

Selection criteria can be specified to create custom reports detailing various aspects of UPSTREAM operation. As an example, you can select only records for a given backup profile (or group of profiles), or only those backups that fall in a certain date/time range. All reports can be customized to include only the data you want to see, and in a layout that you specify.

See Reporting with USTRPORT for documentation on USTRPORT.

Backup File Reporting Tool (USTBKPRT)

USTBKPRT is another batch-driven reporting utility. It can be used to display the contents of any UPSTREAM sequential backup, either on disk or tape. You can point it to a single backup data set, and it displays the name and statistics of every Client file that was included in the backup.

See Reporting with USTBKPRT for documentation on USTBKPRT.