UPSTREAM Operation

Overview

As described in System Overview, the central point of control for UPSTREAM is the z/OS started task. After the initial “manual” stopping/starting of the started task during the testing of UPSTREAM, the startup process is then usually initiated automatically during the z/OS IPL process. This is usually done via a PROC (usually called “UPSTREAM”), stored in a system PROCLIB. See Define the UPSTREAM Started Task PROC.

Once the started task is up and running, there may be numerous occasions when an operator or administrator needs to communicate with the UPSTREAM system. Examples of this interaction with the UPSTREAM started task include:

• Starting (Start the UPSTREAM z/OS Storage Server) and Stopping (Stop the UPSTREAM z/OS Storage Server) the z/OS started task, TCP/IP Re-Connect,Freeing (De-Allocating) a DDNAME Allocation
  
• Viewing status displays (see Status Display (MODIFY Command))
  
• Controlling UPSTREAM sub-tasks (Controlling UPSTREAM Sub-Tasks)
  
• Controlling multiple UPSTREAM started tasks (Controlling UPSTREAM Sub-Tasks)
  
• Requesting diagnostics (UPSTREAM Diagnostics)
  
• Message/Summary log handling (Log Handling)
  
• Configuration maintenance (UPSTREAM Configuration Maintenance)
  
• Adjusting operating controls (Adjusting UPSTREAM Main Configuration Options)
  
• Initiating commands with USTCMD (Initiating Commands with USTCMD).
  

The various sub-sections of this chapter describe the mechanisms and commands that can be used by an authorized operator or administrator of UPSTREAM to communicate with the z/OS started task.

Start the UPSTREAM z/OS Storage Server

The normal z/OS START command is used to start the UPSTREAM z/OS Storage Server started task:

If you like, you can assign a name that is different from the PROC name. In the example below, the PROC is called UPS311, but the started task has the name “UPSTREAM”:

When UPSTREAM has completed its initialization, it issues the following message, showing the member name of the configuration file (if it is a PDS), the VTAM application ID, and the SECLVL (security level) in use:

If any errors are encountered during the initialization phase, they are reported to the z/OS system console as well as the UPSTREAM log file (USTLOG). If you encounter any problems with the initialization, be sure to check both sources for diagnostic information.

Startup Options

When starting UPSTREAM, options can be specified via PARM= on the START command, for example:

The options specified here on the START command override any PARM= options specified in the UPSTREAM startup PROC (see Define the UPSTREAM Started Task PROC). The supported options on PARM= are as follows:

SCHEDULE

Start the UPSTREAM automatic scheduler USTSCHED (UPSTREAM Scheduler).

NOMAINT

Bypass the automatic execution of USTMAINT (UPSTREAM Operation) during startup. USTMAINT should be run periodically in order to cleanup any obsolete entries in the UPSTREAM repository. If you choose to run with NOMAINT specified, remember to execute USTMAINT at regular intervals.

RBSZ

Overrides the default setting of the TCP/IP receive buffer size (TCPRCVBUFRSIZE) that controls the number of bytes received by the z/OS TCP/IP stack before acknowledging this to the remote system.

Default: 131,072 (effective for most shops).

SBSZ

Overrides of the TCP/IP default settings of the TCP/IP send buffer size (TCPSENDBUFSIZE).

Default: 65,535 (effective for most shops).

URBSZ

Overrides the default setting of the TCP/IP receive buffer size (TCPRCVBUFRSIZE) for z/OS UNIX Systems Services (USS) based Client communication. This controls the number of bytes received by the z/OS TCP/IP stack before acknowledging this to the remote USS system.

Default: 32,768.

USBSZ

Overrides the TCP/IP default setting of the TCP/IP send buffer size (TCPSENDBUFSIZE) for z/OS UNIX Systems Services (USS) based Client communication.

Default: 131,072.

16.3 Stop the UPSTREAM z/OS Storage Server

If the UPSTREAM Storage Server started task has to be stopped at any time prior to its normal termination at z/OS shutdown, there are three methods available to request this termination:

• z/OS STOP
  
• z/OS MODIFY
  
• z/OS CANCEL
  

The differences between these methods are described below.

z/OS STOP

z/OS STOP causes UPSTREAM to perform an orderly “quiesce” stop. All sub-tasks in progress at the time the command is received continue until completion, but no new sub-tasks are allowed to start. When all sub-tasks have terminated, the main UPSTREAM task then ends.

The format of the STOP command is:

UPSTREAM issues the following acknowledgment message:

   UST008 UPSTREAM STOP ACCEPTED -- SHUTDOWN IN PROGRESS

If any BACKUP/RESTORE sub-tasks are active, UPSTREAM issues:

   UST240W* ACTIVE TASK FOUND--SHUTDOWN AWAITING TASK TERMINATION

z/OS MODIFY

In those instances when an immediate termination is required, an z/OS MODIFY command can be used. This command causes UPSTREAM to notify all currently processing sub-tasks to terminate. Any backups in progress are suspended. When all sub-tasks have performed an orderly termination, the main task is notified and it then ends.

The format of the MODIFY command is:

UPSTREAM issues the following acknowledgment message:

UST242* QUIT ACCEPTED -- SHUTDOWN IN PROGRESS

z/OS CANCEL

Although the UPSTREAM system can also be closed down by an z/OS CANCEL command, this should not ordinarily be used as it may cause a problem with data integrity across the CATALOG and FILEINFO repository data sets (The UPSTREAM Repository).

For this reason, the UPSTREAM started task protects itself against operator CANCEL commands, as follows:

• If a CANCEL is issued, UPSTREAM intercepts the CANCEL.
  
• If no backup, restore or utility tasks are active, the CATALOG and FILEINFO repository data sets, together with the USTLOG data set, are closed and the CANCEL command is allowed to proceed. This may take up to one minute.
  
• If a shutdown is already in progress (perhaps because of a previous STOP or CANCEL command), all sub-tasks are stopped. The repository and log data sets are closed and the CANCEL is allowed to proceed.
  
• However, if there are any backup, restore, or utility tasks active, a status display is issued (see Status Display (MODIFY Command)and Status Display (TSO/ISPF)) and message UST240W* is issued to give the operator several options:
  
  • Ignore the “CANCEL” and continue with normal operations
    
  • Wait for an orderly shutdown (an internal “STOP” command is issued).
    
  • Terminate a specific sub-task and ignore the “CANCEL”
    
  • Re-display the status
    
  • Accept the “CANCEL” (closing the repository and log data sets first).
    

If UPSTREAM does not come to a normal termination in a reasonable amount of time, another CANCEL can be issued to terminate it.

Prior to using the CANCEL command, it is recommended that you first use the z/OS console DUMP command to obtain an SVC dump of the UPSTREAM address space. This must be done before issuing the CANCEL. The resultant SVC dump provides invaluable information for any subsequent problem determination.

TCP/IP Re-Connect#

In the event UPSTREAM loses it's TCP/IP connection, you can attempt to reconnect via the TCPSTART command:

If this command fails to reconnect, you need to stop and start the region.

Freeing (De-Allocating) a DDNAME Allocation#

In the event UPSTREAM has not released a device allocation, you can use the FREE command to cause the UPSTREAM started task to issue a dynamic deallocation request based on a DDNAME you specify: You may determine the DDNAME from the started task JES and initiator logs.

16.4 Status Display (MODIFY Command)#

The UPSTREAM started task supports an z/OS STATUS inquiry command:

UPSTREAM responds with UST014* messages indicating the following details for each active sub-task.

• Profile name
  
• Network address
  
• Number of files processed (if applicable)
  
• CPU time
  
• Type of operation (e.g., “backup”, “restore”)
  
• Internal sub-task ID (see below)
  

Here is an example of a status display. If no sub-tasks are currently active, message UST016* would be issued in place of the UST015* indicating “No tasks currently active”. The final value on each of the UST014* messages (e.g., “0013”) is the unique internal ID allocated to each sub-task.

A similar display of active sub-tasks can also be obtained via Option 2 (“Status”) on the UPSTREAM TSO/ISPF dialog main menu. This feature requires the program USTATUS to be defined as an ISPF authorized program (see Define the UPSTREAM Started Task PROC).

The elapsed time and CPU time accumulated by each sub-task are also displayed. If appropriate, the number of files and kilobytes of data processed by the sub-task is shown, together with the bytes per second processed by the sub-task. For some types of operations the file count (# file) may display other quantities, such as “number of backup versions” in the case of a “version inquiry” operation.

In addition to the “sub-task-specific” information, the top half of the panel also provides information about the UPSTREAM system itself, including the configuration file name, VTAM application ID, TCP/IP connection information. It also shows utilization statistics for the repository files, which can be useful in determining when a reorganization of these files may be required, as described in The UPSTREAM Repository. The statistics detail the space remaining in the file. In the example, the USTFILEI is nearly empty at 95% space remaining.

The display is refreshed every time you press ENTER. To have it automatically refreshed at intervals, overtype the refresh rate (in seconds). Once in automatic mode, press PA1 or ATTN to return to manual mode.

Commands such as “TERM” or “SUSPEND” can be entered in the command line against an active sub-task, but see Controlling UPSTREAM Sub-Tasksfor more information on controlling the behavior of active UPSTREAM sub-tasks.

Controlling UPSTREAM Sub-Tasks

UPSTREAM starts an internal sub-task for every backup, restore, or file inquiry operation. It also uses a sub-task to process some of the utility functions, such as USTVAULT, USTMERGE, and USTMIGRT. On occasions, it may be necessary to manually control the behavior of one or more of these sub-tasks - e.g., to begin, terminate, suspend, or restart it. The command line in the ISPF Status display (Status Display (TSO/ISPF)) and the USTCMD program (Initiating Commands with USTCMD) are two ways of issuing those controlling commands. UPSTREAM sub-tasks can also be controlled through standard z/OS console commands.

Starting a Sub-Task

You can use an z/OS console command to initiate a sub-task for many of the UPSTREAM utility operations, including USTVAULT, USTMIGRT, USTMERGE, USTREGEN, USTMAINT, and USTREORG. Here are two examples:

F UPSTREAM,VAULT01F UPSTREAM,MIGRT05

Terminating a Sub-Task

To terminate a sub-task, enter one of the following commands. If the sub-task does not terminate within a few minutes, it may have hung during termination; if so, re-issue the command to force termination:

F UPSTREAM,TERM ID=task F UPSTREAM,TERM LU=luname

Suspending a Sub-Task

To suspend a sub-task you can enter one of the following commands. However, due to operating system considerations, this is not recommended:

F UPSTREAM,SUSPEND ID=task F UPSTREAM,SUSPEND LU=luname

Restarting a Sub-Task

To restart a suspended sub-task, enter one of the following:

F UPSTREAM,RESTART ID=task F UPSTREAM,RESTART LU=luname

Controlling Multiple UPSTREAM Started Tasks

Beginning with UPSTREAM 3.9.1, the UPSTREAM main ISPF panel lists details of the running UPSTREAM started tasks. Details include:

• the LPAR the started task is running on
  
• the number of active tasks running in that started task.
  
• the IP address and port of the running started task.
  
• if the started task Main APPLID option is set to NONE (started task configured for TCP/IP), this is noted under the VTAM APL column. Otherwise, the VTAM application major node of the started task is listed.
  

If you are using the 3.9.1 version panels with a back-level started task, that is noted and only the LPAR the started task is running on is noted.

Beginning with UPSTREAM 3.9.3, UPSTREAM started tasks that have run but are not currently running, are listed as “INACTIVE” in the TCP/IP Address column. Configuration and profile changes can be made to these started tasks.

To perform actions against a particular started task, place an s (or a non-blank character) to select the started task and the number of the option you would like to use and press ENTER. To use the STATUS display and the OPER commands, you need to be logged onto the same LPAR the started task is running on.

UPSTREAM Diagnostics

Various facilities are provided within UPSTREAM for obtaining internal diagnostics, often useful in problem determination:

• Internal Tracing
  
• Communications Tracing
  
• Abend on Message
  
• Initiating SVC Dumps
  

Internal Tracing

The internal trace facility should only be used when specifically requested by BMC Support for problem diagnosis. The trace records are written to the USTLOG data set, which should be retained for review.

Start Internal Tracing

To start the internal trace, enter one of the following commands:

If either LU= or ID= is specified, only that specific UPSTREAM sub-task is traced. If multiple TRACE commands are entered, only the last one entered is honored.

If necessary, and again only on the request of BMC Support, you can initiate internal tracing during the UPSTREAM startup procedure. This can be done by adding PARM='TRACE' to the EXEC statement in the UPSTREAM startup PROC. See Define the UPSTREAM Started Task PROC.

Stop Internal Tracing

To stop an internal trace at any time, enter:

Communications Tracing

UPSTREAM also includes a more generalized “communications” trace, which traces VTAM and TCP/IP activity within the UPSTREAM system. This trace has much less overhead than the full internal trace described previously, but should nonetheless still only be run at the request of BMC Support.

Start Communications Tracing

To start the communications trace, enter the following command, where “nnnnn” (1-5 digits) is the maximum bytes of data to print for each trace entry. The default, if omitted, is 20 bytes:

If necessary, you can initiate the communications trace during UPSTREAM startup. This can be done by adding PARM='COMTRACE' to the EXEC statement in the UPSTREAM startup PROC. See Define the UPSTREAM Started Task PROC.

Stop Communications Tracing

To subsequently stop the communications trace at any time, enter:

Abend on Message

In order to diagnose certain errors, UPSTREAM provides the option to take an abend when a specific message is generated. Alternatively, the abend can be taken on the nth occurrence of the message.

Since this causes the sub-task generating the message (or occasionally the entire UPSTREAM started task) to abnormally terminate, this facility should be used only on the specific request of BMC Support.

To invoke the “Abend on Message” feature, enter the following command, where “mmm” is the 3-digit UPSTREAM message number (e.g., ABENDM=012 for message UST012E) and where “ccc” is the 3-digit count of occurrences of the message required before the abend occurs.

F UPSTREAM,ABENDM=mmm, orF UPSTREAM,ABENDM=mmm,ccc

The following example causes an abend on the fifth occurrence of the message UST235.

Initiating SVC Dumps

UPSTREAM includes various methods of obtaining SVC dumps:

SVC Dump ON Message

Similar to the Abend on Message diagnostic, UPSTREAM provides the option to take a SVC dump when a specific message is generated. Alternatively, the SVC dump can be taken on the nth occurrence of the message.

To invoke the “SVC Dump on Message” feature, enter the following command, where “mmm” is the 3-digit UPSTREAM message number (e.g., ABENDS=012 for message UST012E) and “ccc” is the 3-digit count of occurrences of the message required before the SVC dump occurs.

F UPSTREAM,ABENDS=mmm, orF UPSTREAM,ABENDS=mmm,ccc

The following example causes an abend on the fifth occurrence of the message UST235.

SVC Dump ON Demand

An SVC dump of the UPSTREAM started task can be initiated via an UPSTREAM modify command, enter:

The TITLE text can be the operators choice but is limited to 60 characters. The default title is “FDR/UPSTREAM SVC DUMP.”

Initiating an SVC Dump via a Batch JOB

As an alternative to initiating an SVC dump via an UPSTREAM modify command, an SVC dump can also be initiated via the batch job below.

PARM= Specify the UPSTREAM started task name to capture in the dump followed by a comma and the title for the dump.

Log Handling

During its operation, UPSTREAM writes informational and error messages into a log file (USTLOG) and optionally out to a summary file (USTSUMM). These log files may be SYSOUT or DASD data sets.

All of the messages that can be written to these logs are documented in UPSTREAM Messages, and the specification of the log names within the UPSTREAM startup PROC is described in Define the UPSTREAM Started Task PROC.

You may wish to see the messages on the log while UPSTREAM is still active. Unfortunately, it is often impossible to see the most recent messages (which are usually those of most interest). This is because they are still in a buffer and have not yet been written out to the log.

UPSTREAM offers two methods of making these messages available:

• Flushing
  
• Switching
  

Flushing the Log

You use the following “FLUSHLOG” command to request that UPSTREAM “flush” the log and summary buffers by closing the files and reopening them (with EXTEND, if on DASD, so that new messages will still be added to the end).

Switching the Log

UPSTREAM also optionally supports a second log and summary file, specified by the DDNAMEs of “USTLOG2” and “USTSUMM2” in the UPSTREAM startup PROC.

You can use the following SWITCHLOG command to request that UPSTREAM close the current log and summary file and “switch” to the alternate ones:

DASD files also need a DISP=SHR for flushing and switching to operate correctly.

UPSTREAM Configuration Maintenance

As described in UPSTREAM Configurator, the UPSTREAM configuration data set contains the main operating settings for the UPSTREAM started task, as well as descriptions of all the backup and utility profiles. The configuration, which was initially created during the installation process, can then be subsequently altered through:

• The UPSTREAM TSO/ISPF dialog (option 4 - “Configure”)
  
• The USTCONFG program (UPSTREAM Configurator)
  
• The Client panels (see the UPSTREAM Client guide).
  

Any changes to the configuration can be brought online dynamically, without having to stop/start the UPSTREAM started task. Having made the required changes to the configuration, you must do one of the following:

• Reactivate the existing (but now altered) configuration file
  
• Activate a brand new configuration file.
  

Reload an Existing Configuration

To reload or “reactivate” the configuration file that was last activated successfully (but has subsequently been changed), enter the following REFRESH command. This format of the REFRESH command is usually only possible if the USTCONFG DD statement in the UPSTREAM startup PROC points to a sequential configuration file:

Activate a New Configuration

If your configuration file is stored in a PDS, you can choose to “activate” a new set of configuration parameters, which have been saved to a new member of the same PDS. If the activation is successful, the new membername will be saved in memory as the current configuration. That membername will also be used if a REFRESH is later entered without an accompanying membername (i.e., a “reactivation”, as previously described):

Activate a Changed Encryption Member

If you have the UPSTREAM encryption feature licensed and enabled (UPSTREAM Data Encryption), and if you have implemented the facility with a USTCRYPT DD statement in your startup PROC, you can instruct the UPSTREAM started task to reload your encryption definitions via the following command:

Adjusting UPSTREAM Main Configuration Options

As an alternative to permanently (or semi-permanently) changing the Main options (ISPF option 4) configuration, you can temporarily modify these options while the UPSTREAM started task is running. These temporary changes stay in effect until the started task is terminated and re-started, at which time the “permanent” settings in the configuration file once again take effect. You might wish to use this facility to temporarily adjust the operating behavior of UPSTREAM, perhaps to react to a particular condition (e.g., tape drive availability).

MAXTASKS

Control the maximum number of backup/restore/inquiry tasks that can run concurrently. See MAXTASKS for additional documentation.

MAXTAPEB

Control the maximum number of tape drives that UPSTREAM uses for concurrent backups. See MAXTAPEB for additional documentation.

MAXTAPER

Specify the maximum number of tape drives that UPSTREAM uses for concurrent restores. See MAXTAPER for additional documentation.

TIMEOUT

Specify the maximum number of minutes that UPSTREAM waits for most communication requests to complete. See TIMEOUT= for additional documentation.

LOGBLKN

Specify the number of 64K internal buffers used to save USTLOG messages for display at a Client.

Default: 2.

Initiating Commands with USTCMD

Many of the z/OS console commands described in this chapter can also be issued through the utility program USTCMD. This command allows you to control UPSTREAM operation without actually being at an z/OS console. USTCMD can be invoked in one of two ways, as described below:

• Through the UPSTREAM TSP/ISPF dialog (see USTCMD via the TSO/ISPF Interface)
  
• Via an z/OS batch job (see USTCMD in Batch).
  

USTCMD via the TSO/ISPF Interface

Option 6 (“OPER”) on the UPSTREAM TSO/ISPF dialog main menu invokes a panel that allows authorized users to issue z/OS “console” commands.

If this panel is too large to fit on your TSO screen, it can be scrolled up and down with PF7 and PF8. Place an “S” next to the command to be issued (only one at a time), and fill in any required parameters on that line. You receive a confirmation that the command was issued, but you need to check the z/OS SYSLOG and/or the UPSTREAM log (USTLOG) for results.

You can also use USTCMD in a batch job to issue one or more z/OS console commands to UPSTREAM.