The UPSTREAM Repository

This section describes the UPSTREAM repository, which stores information on your current backups, together with historical information about backups, restores and vaulting operations.

The repository may also contain information pertaining to some of the optional features available under UPSTREAM, such as registered names.

The repository consists of three files. The generic name to which they are referred is shown below, together with the DD statement name (in brackets) to which they are referenced in the UPSTREAM startup PROC. The individual repository files are described in more detail in the sections indicated.

• The CATALOG file (USTCATLG).
  See Define the “CATALOG” Repository Data Set
  
• The FILEINFO file (USTFILEI).
  See Define the “FILEINFO” Repository Data Set

The contents of the repository files can be reported in various ways. -

• USTRPORT (Reporting with USTRPORT)
  
• Via the Client GUI interface (For more information, see UPSTREAM Client Guide)
  
• Using the Director interface (For more information, see UPSTREAM Client Guide for more details.)
  

The remainder of this chapter describes in detail the format of the repository files and the contents of the individual files and recommendations for sizing them. It also describes how to maintain and re-organize the repository files, as well as how to re-load data into them in the event of a problem.

6.2 Choosing the Repository File Format

IAM vs. VSAM

The UPSTREAM repository files can be created in one of two formats.

• As a VSAM KSDS
  
• As an IAM file
  

The recommended format is IAM (Innovation Access Method), which is a proprietary file format that provides VSAM-compatible keyed access, but with a fraction of the usual overhead normally associated with VSAM.

The IAM format provides high-speed low-overhead access to the repository data. It uses an advanced internal file structure that is far superior to regular VSAM and requires 30 to 70 percent less DASD space than a similarly defined VSAM file.

These savings are achieved by IAM through the following design features:

• IAM loads the entire index structure in above-the-bar, 64 bit storage in the UPSTREAM address space to hold the index for the UPSTREAM repository files, eliminating the need for imbedded indices.
  
• IAM files have an internally stored, compressed index, which requires minimal DASD space.
  
• IAM's free space allocation techniques make much more efficient use of DASD space for the data records in the file.
  

UPSTREAM repository files in IAM format can be allocated on any DASD device and appear to the z/OS operating system as physical sequential files (DSORG=PS), even though the data is accessed internally by a key field, just like under VSAM.

IAM repository files appear in the z/OS catalog as non-VSAM type entries.

Allocating the Repository Files, IAM PRO

The repository files are initially created through a TSO/ISPF dialog during the installation of UPSTREAM. See Define the “CATALOG” Repository Data Set. See Define the “FILEINFO” Repository Data Set.

If they are in VSAM format, they can be created and allocated with standard IDCAMS. If they are in IAM format, a special UPSTREAM utility USTCAMS must be used. See USTREORG and USTCAMS.

Regardless of the format chosen, the repository files may be SMS-managed and they can be allocated as extended format (EF) files, which will support hardware compression, Alternatively, they may be allocated as Large format files, or as Basic format files.

IAM PRO and Reducing Virtual Storage Utilization

If you choose to allocate the UPSTREAM repository files in IAM format and are running UPSTREAM z/OS release 3.6.0 or later, you may wish to consider using two new IAM features, Prime Related Overflow (PRO) and 64BIT INDEXSPACE. Both features are recommended for the USTFILEI control file.

UPSTREAM z/OS 3.6.0 includes support for IAM 9.0 functionality and IAM PRO. The minimum requirements for IAM PRO are a z/OS operating system and a z/Architecture processor. IAM Version 9.0 utilizes various functions and features only available in z/OS operating systems and instructions available only on processors that support z/Architecture. PRO format repository files created with Version 9.0 are not downward compatible with prior UPSTREAM releases. Users wishing to revert back to UPSTREAM z/OS 3.5.0 need to reallocate their repository files.

IAM PRO is an enhancement that provides greater scalability for IAM format repository files, reduces virtual storage use, and requires fewer reorganizations during a given period. The PRO format has a different internal structure to the overflow area than previous IAM releases and is indexed at the block level rather than the record level. This generally results in a much smaller in-storage index and a decreased need to reorganize the file. PRO can be implemented by defining new repository files with the PRO=YES CREATE operand. This is specified with the USTOVRID DD statement used when defining the files.

The 64BIT INDEXSPACE feature of IAM 9.0 uses above-the-bar, 64 bit addressable z/OS storage in the UPSTREAM address space to hold the index for the UPSTREAM repository files. This eliminates the two gigabyte index limitation of a data space and frees up above-the-line storage that would otherwise be occupied in the UPSTREAM address space, thus increasing the amount of virtual storage for other UPSTREAM functions, such as deferred merge (USTMERGE), vaulting (USTVAULT) and migration (USTMIGRT). When migrating to UPSTREAM z/OS 3.6.0, this feature is enabled by default unless overridden with the INDEXSPACE ACCESS operand defined with the USTOVRID DD statement placed in the UPSTREAM startup JCL (Define the UPSTREAM Started Task PROC.)

Enabling PRO format repository files

Allocate the UPSTREAM repository files

1. 126_List1_Num_First_1133018 1. Either use the output file of a current REORG or perform a backup of your control files created by a DUMPX command for use in the REPRO referred to below, The DUMPX command can be submitted either from the UPSTREAM ISPF panel 6, Operator Commands or via the modify command below. For more information, see Making a Sequential Copy of a Repository File.

   F UPSTREAM,DUMPX DD=ddname
2. Define appropriate new FILEINFO and USTCATLG files using the UPSTREAM ISPF option 3 (DEFINE) option.
   • Use the allocation of your existing control files for sizing of your new control files.
     
   • “File Structure ===> PRO” is specified by default. When enabling PRO, do not change this value.
     
   • Specify the name of the data set created by a current REORG or DUMPX command in the “specify Repro from dsname ===>” field for the REPRO control card with REUSE option to be generated. If you wish, you may stop your z/OS Storage Server Started Task and use your existing control files as the input for the Repro.
     
   • Do a PF3/END from the DEFINE panel to be presented with Browse, Edit, or Submit options.
     

   • The JCL will contain the USTOVRID DD control card similar to the one below.

     //USTOVRID DD *
     CREATE DD=CATALOG,B=4,PE=20,VARO=YES,REL=NO,PRO=YES
     CREATE DD=FILEINFO,B=2,PE=100,VARO=YES,REL=NO,PRO=YES
3. Submit the JCL and define your new repository control files.
   
4. Update the z/OS Storage Server Started Task JCL. Either rename the new control file data set names to your old names or change the z/OS Storage Server Started Task JCL to point to the new names.
   
5. Restart the z/OS Storage Server Started Task
   

Fallback procedures, disabling PRO format

Should it be necessary to remove the PRO feature, contact BMC Support for problem diagnosis. If you are downgrading from UPSTREAM z/OS 3.6.0 back to UPSTREAM 3.5.0, the PRO and the 64BIT INDEXSPACE features are not supported in UPSTREAM z/OS 3.5.0 and prior releases and both features must be removed. Contact BMC Support to discuss this process.

Allocate the UPSTREAM repository files

1. Either use the output file of a current REORG or perform a backup of your control files created by a DUMPX command for use in the REPRO referred to below, The DUMPX command can be submitted either from the ISPF panel 6, Operator Commands or via the modify command below. Please refer to Making a Sequential Copy of a Repository Filefor further details.

   F UPSTREAM,DUMPX DD=ddname
2. Define appropriate new FILEINFO and USTCATLG files using the UPSTREAM ISPF option 3 (DEFINE) option.
   • Use the allocation of your existing control files for sizing of your new control files.
     
   • “File Structure ===> PRO” is specified by default. To disable PRO, specify ENHANCED, do not specify PRO.
     
   • Specify the name of the data set created by a current REORG or DUMPX command in the “specify Repro from dsname ===>” field for the REPRO control card with REUSE option to be generated. If you wish, you may stop your z/OS Storage Server Started Task and use your existing control files as the input for the REPRO.
     
   • Do a PF3/END from the DEFINE panel to be presented with Browse, Edit, or Submit options.
     

   • The JCL contains the USTOVRID DD control cards similar to the example below.

     //USTOVRID DD *
     CREATE DD=CATALOG,B=4,PE=20,VARO=YES,REL=NO,ENHANCED
     CREATE DD=FILEINFO,B=2,PE=100,VARO=YES,REL=NO,ENHANCED
3. Submit the JCL and define your new repository control files.
   
4. Update the z/OS Storage Server Started Task JCL. Either rename the new control file data set names to your old names or change the z/OS Storage Server Started Task JCL to point to the new names.
   
5. Restart the z/OS Storage Server Started Task
   

Reducing virtual storage utilization using a 64 BIT INDEXSPACE or Dataspace

For users of UPSTREAM z/OS 3.6 and later, the default method to hold the index for IAM format repository files and reduce virtual storage utilization is to use above-the-bar, 64 bit addressable z/OS storage in the UPSTREAM address space. For users not yet running UPSTREAM z/OS 3.6, you may wish to consider placing the indexes for these files in a separate, above-the-line, 2GB z/OS dataspace instead. Both of these features free up storage that would otherwise be occupied in the UPSTREAM address space, and increases the amount of virtual storage for other UPSTREAM functions. Only one method may be used at a time.

64 BIT INDEXSPACE (UPSTREAM z/OS 3.6)

• We recommend users specify a MEMLIMIT=8G parameter on their started task EXEC card. This limits the indexspace to 8 GB to prevent exhaustion of storage needed for the z/OS page data sets.
  
• Users currently specifying INDEXSPACE=YES via a USTOVRID DD statement in their started task JCL, need to either change INDEXSPACE=YES to INDEXSPACE=64BIT or remove the INDEXSPACE operand entirely (since INDEXSPACE=64BIT is the default, either action will have the same affect.)
  
• Restart the UPSTREAM z/OS Started Task to enable the change.
  

2GB z/OS dataspace (UPSTREAM z/OS 3.5 and earlier releases)

• Define a USTOVRID DD statement referencing a member in your UPSTREAM control library data set containing the IAM ACCESS command INDEXSPACE=YES and place the USTOVRID DD statement in the UPSTREAM startup JCL (Define the UPSTREAM Started Task PROC) Please refer to the example below.

  The “&ALLDD” in the example shown below creates a separate dataspace for the indexes of all three repository files.

• 
  
  • USTOVRID DD statement in the UPSTREAM startup JCL -
    //USTOVRID DD DISP=SHR,DSN=your.control.library(membername)
  • Contents of your.control.library(membername)
    ACCESS DD=&ALLDD,INDEXSPACE=YES
• Restart the UPSTREAM z/OS Started Task to enable the change.
  

6.3 The CATALOG (USTCATLG) File#

The CATALOG (USTCATLG) repository file contains the primary control records for the main functions of UPSTREAM. Those records include:

• “History” Records (for backups, restores, vaulting)
  
• “Backup Data Set” Records(for backups only)
  
• “File Specification” Records(for backups only)
  
• “Vault Copy” Records(for USTVAULT operations)
  
• “Registered Name” Records
  

The following sections describe these CATALOG file records in more detail.

“History” Records

Instead of writing z/OS SMF records for every operation, UPSTREAM writes a “History” record into the CATALOG repository. These history records are created on the completion of each UPSTREAM function (backup, restore, or vault) and they contain information such as the function requested, the elapsed time, execution time, and the number of files processed. This information allows you to report on overall UPSTREAM activity, or on the activity of specific Clients or functions.

The setting of the MAXHIST MAIN configuration parameter (see UPSTREAM MAIN Configuration Options) controls the retention of the history records by specifying the number of days that the records should be retained in the CATALOG file before being purged (See Repository Maintenance)

The default setting for the MAXHIST parameter is 365 days, but should be altered to equal the highest retention of your full backups, so that you have full history reporting for all backups that still exist.

“Backup Data Set” Records#

“Backup Data Set” records are created for each backup that you run. The record contains information such as the name of the backup profile used, the type of backup requested, the time and date of the backup, and the z/OS data set name created to hold the contents of the backup.

The retention of “backup data set” records is controlled indirectly by the retention and expiration of the actual backups they describe. UPSTREAM does not automatically uncatalog its backups from z/OS. Instead, it assumes that your Tape and/or DASD management systems will uncatalog the backups when they reach their expiration. This is controlled by the RETPD or EXPDT attributes of the associated backup profile (Backup Profiles), or by GDG roll-off processing, if the backups are allocated as GDGs.

Alternatively, you can manually delete specific backups through the UPSTREAM TSO/ISPF interface (see Managing Your Backups), or through the Client panels (See the UPSTREAM Client Guide).

Regardless of the retention technique used, once the backup is no longer cataloged, the corresponding “backup data set” record in the CATALOG file becomes eligible for purging by the MAINT command or USTMAINT utility (Repository Maintenance.)

“File Specification” Records#

A “file specification” record is created in the CATALOG repository file for each file specification (see Backup Architecture) used in a backup. For example, if you specified that you wanted to back up the /usr/* and /opt/* directories for a particular file server, this would result in the creation of two “file specification” records in the CATALOG file.

Old “file specification” records are purged when their associated “backup data set” record is purged from the CATALOG file.

“Vault Copy” Records#

If you are performing vaulting with USTVAULT, two “vault copy” catalog records will be created for each vault run. See Copying Backups with USTVAULT.

• The first record contains information relating to the tape retention data set that is created as the first file on the vaulted tape.
  
• The second record is for the vault control data set that is created as the last file on the vaulted tape.
  

These “vault copy” records are purged (by the same rules described previously for “backup data set” records) on the expiration of the vaulted backups that they describe.

When the first data set on the vault tape is no longer cataloged, it is eligible for purging. See Repository Maintenance.

“Registered Name” Records

If you are using the optional registered name facility (see Registered Name Service), a “registered name” record will be recorded in the CATALOG file for each Client that signs on with auto-registration enabled, or which is manually registered via the TSO/ISPF dialog.

“Registered name” records are utilized in order to establish an alias for a remote Client and to conceal its network address from the connection process. This allows the use of IP DHCP protocol when interfacing between the Client and the z/OS Storage Server software.

The “registered name” record also contains information regarding the version of the UPSTREAM software installed on the Client when that Client last connected to the z/OS Storage Server. This information can be used by the “automatic software distribution” feature to update the version of UPSTREAM software that is running on the Client.

If a “registered name” record is not accessed within 90 days, it becomes eligible for deletion. The records can also be manually deleted, if required, via the UPSTREAM TSO/ISPF dialog.

CATALOG File Sizing

The CATALOG repository file is generally less than 5 cylinders in size, even in the largest shops. The approximate sizing for the CATALOG file is fairly straightforward and can be expressed as follows:

where:

• H - # of history records per backup (always one).
  
• B - # of backup data set records per backup (always one).
  
• F - # of file specification records per backup (variable).
  
• V - # of vault requests (two per vault run).
  
• R - # of defined registered names (optional, variable).
  
• N - # of retained backups (full, incremental, and non-merge).
  
• T - The total number of CATALOG records to allocate.
  

The easiest method for performing this calculation is to use the “DEFINE” panel in the UPSTREAM TSO/ISPF dialog (see Section “Define the “CATALOG” Repository Data Set”), which calculates the size of the file, based on input provided by you, and then generates all the necessary z/OS JCL to define the file.

The contents of the CATALOG file changes on a regular basis, as each request issued to UPSTREAM is completed. The file should be adequately allocated to allow for uninterrupted service for at least one week.

It should also be regularly reorganized (see Repository Maintenance).

6.4 The FILEINFO (USTFILEI) File#

The FILEINFO (USTFILEI) file contains “File Information” Records that are created when a backup is run.

The following section describes these FILEINFO record in more detail.

“File Information” Records#

A “file information” record is created for each Client file that is backed up under a backup profile. If a file is backed up many times, multiple records will appear for that file in the FILEINFO repository file.

The “file information” records contain details of your backups, such as the date/time and location of the backup, as well as the name and size of the files that have been backed up.

“File information” records are used when constructing a list of files to be restored, and for general reporting on the contents of a backup.

“File Information” records are purged (see Repository Maintenance) when their associated “backup data set” record is purged from the CATALOG file.

FILEINFO File Sizing

The FILEINFO file is generally quite large in size, often extending across multiple volumes.

Since most Clients will contain a very large number of files that you will wish to backup and track, the FILEINFO repository file is the most important capacity planning issue when setting up and maintaining the UPSTREAM repository.

The approximate sizing for this file is fairly straightforward and can be expressed as follows:

N * F = T

where:

• F - Average # of files backed up (very variable per server)
  
• N - # of retained backups (full, incremental, and non-merge)
  
• T - The total # of records to allocate.
  

The easiest method for performing this calculation is to use the “DEFINE” panel in the UPSTREAM TSO/ISPF dialog (see Define the “FILEINFO” Repository Data Set), which will calculate the size of the file, based on input provided by you, and then generate all the necessary z/OS JCL to define the file.

The FILEINFO file will change as each backup is completed. The file should be adequately allocated to allow for uninterrupted service for a reasonable length of time.

Due to the size of this file and the large amount of activity that it receives, it should be regularly reorganized (see Repository Maintenance).

Repository Utilization Statistics

You can check the current utilization of the three repository files through the UPSTREAM TSO/ISPF dialog, using option 2 (“Status”).

Repository File Interrelationships

The FILEINFO repository will be the target of keyed access requests based on the version date, profile name, and other fields contained in the CATALOG file records.

For these reasons, the USTCATLG and USTIFLEI files should be regarded as a single entity for the purpose of backup and recovery and for maintaining those files at a single point-in-time.

Repository Maintenance

The CATALOG and FILEINFO repository files are generally accessed by a key in order to reduce file access times during record location and insertion operations. Under normal operating circumstances, the vast majority of activity against the repository will be for the insertion of records, usually relating to backup operations.

Due to the nature of the insertion process on any type of key sequenced file, VSAM or IAM, the optimization of the file will be slowly reduced as a result of that heavy insertion. This may affect the future ability to insert records, and could impact the performance of the repository.

Various utility processes are provided to address this problem, as described in the indicated sub-sections that follow:

• USTMAINT (MAINT)- This utility purges obsolete records from the CATALOG and FILEINFO repository files.
  
• USTMAINT (MAINTF)- This utility purges “orphan” records from the FILEINFO repository that may have been created due to system or application failures during backup processing.
  
• USTREORG and USTCAMS- These utilities are used to reorganize one or more of the repository files to reclaim space freed up by the preceding two utilities.
  

USTMAINT (MAINT)#

The USTMAINT utility processes the UPSTREAM Catalog repository file looking for backups that have been previously taken, but which are no longer recorded in the z/OS catalog. If it locates a backup that is no longer cataloged, it purges all “backup data set” records from the CATALOG file and any related “file information” records from the FILEINFO file. Any “history” records for the backup in the CATALOG file will be retained, until eventually removed based on the value specified for the MAXHIST MAIN configuration parameter (see UPSTREAM MAIN Configuration Options).

The space utilized by the deleted repository records is not reclaimed, however, until the CATALOG and FILEINFO files are reorganized (see USTREORG and USTCAMS).

USTMAINT can also be run manually at any time via the following methods:

• Via an z/OS modify operator command: F UPSTREAM,MAINT
  
• Through the TSO/ISPF Operator Command dialog
  
• With “COMMAND=MAINT” in a USTBATCH job, as shown here:
  

  • UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters
    

    //jobname JOB (accounting,information),'job id data',
    //        NOTIFY=userid
    //*
    //*
    //MAINT    EXEC PGM=USTBATCH
    //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
    //SYSUDUMP DD SYSOUT=*
    //USTLOG   DD SYSOUT=*
    //USTPARM  DD *
    HOSTDNS=YOUR.LPAR.DNS.NAME
    HOSTPORT=1972               *default
    *
    COMMAND=MAINT
    *
    ENDPARM
    /*

  • UPSTREAM 3.9.0 and prior using USTBATCH to Started Task VTAM SNA Parameters

    //jobname JOB (accounting,information),'job id data',
    //        NOTIFY=userid
    //*
    //*
    //MAINT    EXEC PGM=USTBATCH
    //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
    //SYSUDUMP DD SYSOUT=*
    //USTLOG   DD SYSOUT=*
    //USTPARM  DD *
    APPLPREF=UPSTR              * VTAM APPL Prefix                         
    USAPPL=UPSTREAM             * Name of UPSTREAM Started TASK VTAM APPL  
    LOGMODE=#INTER              * VTAM LOGMODE to use from USTBATCH to STC
    CONV=WAIT                   * WAIT for Backup to complete before ending
    *
    COMMAND=MAINT
    *
    ENDPARM
    /*

USTMAINT (MAINTF)

When an UPSTREAM backup task is in progress, control records are added to the FILEINFO repository file as each Client file is backed up. The corresponding (final) record in the CATALOG repository file is not written until the backup completes.

If a UPSTREAM backup is interrupted prior to completion, perhaps due to a system or application ABEND, a re-start of the z/OS system while the backup is in progress, or even a forced cancellation of the backup itself, this will create “orphan” entries in the FILEINFO file with no corresponding record in the CATALOG file.

The USTMAINT (MAINTF) utility function will correct this situation and should be run at least once a week.

However, unlike the MAINT function described previously, MAINTF is not run automatically when UPSTREAM is started up. It can only be run manually by one of the following methods:

• Via an z/OS modify operator command: F UPSTREAM,MAINTF
  
• Through the TSO/ISPF Operator Command dialog
  
• With “COMMAND=MAINTF” in a USTBATCH job, as shown here:
  

  • UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters
    

    //jobname JOB (accounting,information),'job id data',
    //        NOTIFY=userid
    //*
    //*
    //MAINTF   EXEC PGM=USTBATCH
    //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
    //SYSUDUMP DD SYSOUT=*
    //USTLOG   DD SYSOUT=*
    //USTPARM  DD *
    HOSTDNS=YOUR.LPAR.DNS.NAME                               
    HOSTPORT=1972               *default
    *
    COMMAND=MAINTF
    *
    ENDPARM
    /*

  • UPSTREAM 3.9.0 and prior using USTBATCH to Started Task VTAM SNA Parameters

    //jobname JOB (accounting,information),'job id data',
    //        NOTIFY=userid
    //*
    //*
    //MAINTF   EXEC PGM=USTBATCH
    //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
    //SYSUDUMP DD SYSOUT=*
    //USTLOG   DD SYSOUT=*
    //USTPARM  DD *
    APPLPREF=UPSTR              * VTAM APPL Prefix                         
    USAPPL=UPSTREAM             * Name of UPSTREAM Started TASK VTAM APPL  
    LOGMODE=#INTER              * VTAM LOGMODE to use from USTBATCH to STC
    CONV=WAIT                   * WAIT for Backup to complete before ending
    *
    COMMAND=MAINTF
    *
    ENDPARM
    /*

USTSEVER

This utility removes all FILEINFO records for the most recent backup taken for a specified group of profiles. Its purpose is to help manage the size of this UPSTREAM FILEINFO file in very large shops. In order to do a restore from one of these “SEVERED” profiles, you must first perform REGEN function on the backup dataset in question, in order to make the files on the backup addressable.

USTREORG and USTCAMS

In order to reclaim space for deleted records and put all inserted records in their proper locations, a re-organization of the repository files is required at periodic intervals, based upon the level of insertion activity and the sizing of the file in question.

The reorganization process can be performed as either:

• An “online” re-organization, with USTREORG. See “Online” Re-organization (USTREORG).
  
• An “offline” re-organization, with USTCAMS. See “Offline” Re-organization (USTCAMS).
  

“Online” Re-organization (USTREORG)#

An online re-organization can be run with the USTREORG utility while the UPSTREAM started task is still active. However, all other UPSTREAM activity must be quiesced. If USTREORG is initiated before all activity is quiesced it will wait. This is the recommended method for re-organizing the repository data sets.

USTREORG utilizes the special USTCATLG and USTFILEI reorganization profiles. See USTFILEI and USTCATLG Profile Parameters for more details.

The sample JCL below shows how to execute the USTREORG function via a USTBATCH batch job. In this job we are re-organizing the CATALOG file (USTCATLG). To re-organize the FILE INFORMATION FILE (USTFILEI), comment out the USTCATLG reorg command and remove the comment for the USTFILEI command.

• UPSTREAM 3.9.1 and later using USTBATCH to Started Task TCP/IP Parameters
  

  //jobname JOB (accounting,information),'job id data',
  //        NOTIFY=userid
  //*
  //*
  //REORG    EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  HOSTDNS=YOUR.LPAR.DNS.NAME                               
  HOSTPORT=1972               *default
  *
  COMMAND=REORG DD=USTCATLG
  * COMMAND=REORG DD=USTFILEI
  *
  ENDPARM
  /*

• UPSTREAM 3.9.0 and prior using USTBATCH to Started Task VTAM SNA Parameters
  

  //jobname JOB (accounting,information),'job id data',
  //        NOTIFY=userid
  //*
  //*
  //REORG    EXEC PGM=USTBATCH
  //STEPLIB  DD DISP=SHR,DSN=your.upstream.loadlib
  //SYSUDUMP DD SYSOUT=*
  //USTLOG   DD SYSOUT=*
  //USTPARM  DD *
  APPLPREF=UPSTR              * VTAM APPL Prefix                         
  USAPPL=UPSTREAM             * Name of UPSTREAM Started TASK VTAM APPL  
  LOGMODE=#INTER              * VTAM LOGMODE to use from USTBATCH to STC
  CONV=WAIT                   * WAIT for Backup to complete before ending
  *
  COMMAND=REORG DD=USTCATLG
  * COMMAND=REORG DD=USTFILEI
  *
  ENDPARM
  /*

“Offline” Re-organization (USTCAMS)#

As an alternative, you can utilize the USTCAMS utility program to perform an “offline” reorganization, but this requires that the z/OS Storage Server started task be down or quiesced for the duration of the reorganization.

Unlike with USTREORG, the special USTCATLG and USTFILEI profiles are not utilized when performing an “offline” reorganization with USTCAMS.

The sample JCL below illustrates how to perform the reorganization via a USTCAMS batch job, which is a simple REPRO out to a work data set, immediately followed by a REPRO back into the database file. This works in much the same way as a normal VSAM KSDS reorganization with IDCAMS. In the example, we are reorganizing the FILEINFO file.

Enhancing REORG Performance via use of FDRREORG

FDRREORG is a separately-priced component of the FDR DASD Management System. FDRREORG can enhance performance of a reorganization of a multi-volume UPSTREAM control file by using parallel devices for the dump and reload to reduce the reorganization elapsed time. When the UPSTREAM REORG is started, USTREORG looks for the USTREOC and USTREOF DD statements and, if present in the UPSTREAM PROC, calls the FDRREORG utility for that particular file reorganization.

FDRREORG provides an easy and automated method of reorganizing VSAM, IAM and PDS data sets. FDRREORG eliminates the manual effort of running reports to see how many CI/CA splits a cluster has or how much available space there is in a PDS and manually submitting batch jobs to REPRO or IEBCOPY the files.

For more information on FDRREORG or the rest of the FDR family, visit our web site at:

www.innovationdp.fdr.com

Enabling FDRREORG:

1. Add a DD statement to the UPSTREAM started task PROC specifying a 80-byte card-image parameter file or member such as a member of SYS1.PARMLIB.
   
2. Specify a DD statement name of USTREOC to use FDRREORG for the USTCATLG file, or USTREOF to use FDRREORG for the USTFILEI file.
   
3. In the card-image parameter member created in the previous step, include the appropriate statements to be passed to the FDRREORG utility.
   

This is a sample USTREOF input file to use FDRREORG to reorganize a control file:

Please refer to the FDRREORG manual for compete details on each of the keywords.

Retaining the Re-Organization “Backup Files”#

Both the “online” and “offline” methods described previously will create an z/OS sequential “backup file” on disk or tape containing the data records from the repository file being re-organized. These records are then re-loaded in key sequence.

However, if you have to restore from the “backup file”, you MUST restore all three repository files back to the same point-in-time, as outlined earlier under repository file interrelationships.

Making a Sequential Copy of a Repository File

Use the DUMPX command to cause the UPSTREAM started task to allocate and write a sequential copy of one of the repository control files (USTCATLG and USTFILEI). This command is equivalent to the dump phase of a REORG process without the RELOAD phase. The allocation will be based on the control file reserved profile names (USTCATLG and USTFILEI) just as is done in the REORG process.

Updating the Repository with USTREGEN

The USTREGEN utility (Updating the Repository) can be used to update the UPSTREAM repository with information obtained from an UPSTREAM backup or vault file.

USTREGEN can be used for the following situations:

• When repository information for a backup has been lost or is incorrect
  
• When a vaulted copy of a backup is to be used in place of the primary copy
  
• To add information to the repository for a “physical” backup taken with FDRSOS
  

See Updating the Repository for more details on USTREGEN.

Increasing the Size of the FILEINFO Repository File

Use the procedures below to increase the size of your UPSTREAM FILEINFO repository file. (The size of the FILEINFO file should be increased if the data set is at 16 extents. When the data set is out of space, a UST051E error will be posted to the Started Task USTLOG DD statement with a FDBK code ending in “1C”).

You need to create a backup of your FILEINFO file for use by the REPRO with reloading the FILEINFO file. You can either use the output file of a current REORG or perform a backup of your FILEINFO file created by a DUMPX command. (If using DUMPX, the allocation is based on the USTFILEI reserved profile, just as is done in the REORG process, and if the backup data set is defined as a GDG, this will roll-off the oldest reorg data set.)

1. The DUMPX command can be submitted either from the UPSTREAM ISPF panel 6, Operator Commands, or via the modify command below.
     F UPSTREAM,DUMPX DD=fileinfo
2. Define a new FILEINFO file using the UPSTREAM ISPF option 3, Define, option.
   
   • Select option 3, FILEINFO. Use the allocation of your existing FILEINFO data set for sizing of your new FILEINFO file and increase the number of cylinders to allocate.
     
   • Specify the name of the data set created by a current REORG or DUMPX command in the “Specify Repro from dsname ===>” field for the REPRO control card with REUSE option to be generated. If you wish, you may stop your z/OS Storage Server Started Task and use your existing FILEINFO repository file as the input for the Repro.
     
   • Do a PF3/END from the “Define the File-Information cluster” panel and a PF3/END from the “Define” panel to be presented with Browse, Edit, or Submit options. If not previously specified, add STEPLIB and JOB card specifications. If you wish to save the JCL, use the ISPF CREATE command and save the JCL to a library that you specify.
     
3. Submit the JCL and define the new FILEINFO file.
   
4. Update the z/OS Storage Server Started Task JCL. Either rename the new FILEINFO data set name to your old name or change the z/OS Storage Server Started Task JCL to point to the new data set name.
   
5. Restart the z/OS Storage Server Started Task to open the new FILEINFO file.
   

SEVER’ing an UPSTREAM Profile GROUP from the FILEINFO File

The SEVERxx command, removes the FILEINFO records from the most recent backup for each profile in group “xx”. This reduces the number of records resident in the FILEINFO file for these profile(s), thereby reducing the overall DASD space usage for this file. This command can be issued from the z/OS Console, from the UPSTREAM ISPF option #6 panel, via the USTBATCH utility program, or via the USTCMD utility. Each profile is identified via the command group specified in the command suffix: SEVERxx. Each profile must contain the GROUPID specified in its GROUPID field of its configuration record in the active USTCONFIG member.

The format of the command if issued from the z/OS console is: