Automated Backups and Restores

UPSTREAM Dispatcher

To allow you to be able to easily manage and control your repeated backups and other operations UPSTREAM  includes an advanced scheduler with the Client software named the UPSTREAM Dispatcher. It allows you to:

• Easily setup, add, delete and modify any number of backups and jobs to be run repeatedly.
  
• Knowing the issues in UPSTREAM, it allows easy setup of fulls and incremental backups and easily add, modify or delete target systems.
  
• Notification for success or failure of operations.
  
• Because it's controlled by the UPSTREAM Director, can be setup or managed from anywhere.
  
• Operates with UPSTREAM z⁄OS the UPSTREAM Reservoir. Since it runs as an administrator on the UPSTREAM Reservoir machine, you do not have to store security information. Because it can run jobs, it can be used for Reservoir vaulting, migration, or copying.
  
• Offers full control of scheduled operations including scheduled stop and start of the Dispatcher itself.
  
• and much more
  

This section begins with the initial setup to use the Dispatcher followed by an example of a schedule with three systems running daily incremental backups and weekly fulls on a Reservoir system. Subsequent sections detail complex operations that can be performed with the Dispatcher.

UPSTREAM Dispatcher Setup

Scheduling UPSTREAM functions should be performed after you are fully familiar with UPSTREAM. On each of the systems you are scheduling backups for you should install and test UPSTREAM to validate connectivity and initial setup.

The Dispatcher is automatically activated by the configurator when you set the local host IP address on the Reservoir machine.

It is important to remember that the Dispatcher is run in a Client.

Mail notification is an useful feature with scheduling. We strongly recommend that you setup an UPSTREAM mail notification file to point to any administrators who are to be monitoring scheduled operations on the Dispatcher machine. See UPSTREAM Notification.

You must install UPSTREAM on the systems you wish to setup scheduling for, and you must define profiles for these systems either as standalone profiles or profile sets. Profile sets are recommended in cases where you wish to put multiple machine's backups on a tape as it helps sequence the tape usage.

An UPSTREAM Dispatcher Example

To begin setting up a schedule, enter the Director and press the Dispatcher tab. This displays the Dispatcher Wizard as shown in the following figure.

Dispatcher Wizard

The Dispatcher wizard is a window designed to always be on top of other windows as it controls the overall functions of managing Scheduling.

Press the DispatcherStatus button to display the Set Dispatcher Target Wizard.

Set Dispatcher Status

Like all Dispatcher wizards, this window is on top of all other windows. The left side of the window simply lists the steps to perform - there are no buttons there. The right side shows the selections or other actions you have to take to complete the step. In this case, there are no actions other than to press the button (which connects to the Dispatcher on the Reservoir target system).

The Director lists the number of Schedule Sets found, verified, and loaded. A schedule set is a list of machines and actions to be performed at any number of repeated times. Verified schedule sets can be loaded. Schedules loaded at Dispatcher start are loaded and active. Schedules new or updated must be manually loaded and are put in suspended state by default and must be Resumed to activate.

11.2 Creating a Schedule Set

To begin creating a Schedule Set, from the Dispatcher Wizard press the New button. This displays the following figure:

New Schedule Set Wizard

The purpose of a schedule set is to group a list of targets that are performing a similar operation at regular repeated times. If you have a separate set of targets you wish to schedule because of tape or other limitations, then you will need a separate schedule set for them. Similarly, if you have a list of targets but need to perform different operations on them (such as backup databases), you would again need a separate schedule set.

Step 1: Name the Schedule Set

You can use any name you wish, but this name is a file name so it has the character limitations of file names on the Dispatcher target (asterisk or slashes are invalid for example). For this example, let's use PROD. You can enter a description that helps you remember the purpose of this schedule set. Press the Next> button to continue.

Step 2: Choose Targets and Profiles

Highlight the Target Server in the first column, the Profile in the second column. If you are using a GLOBAL or profile prefix you can enter the profile in the edit field below the Defined Profiles. With this correct relationship, press the Add>> button to add it to the list.

Choose Targets and Profiles

When you have completed your list of systems and profiles press the Next> button to continue.

Step 3: Add Job / Run-Time / Action Steps#

The purpose of this step is to define the overall action and the time of day of the action. In this example, let's perform fulls at 10:00 PM on Fridays and incremental backups at 10:00 PM on Mondays through Thursdays. To setup the full action, press the Add Job radio button, pull down the Start at HH: pull down and select 22 (that results in 22:00 or 10:00 PM). Press the Add New Job With First Action and Run-Time button.

Add Job / Run-Time / Action Steps

This results in an entry being added to the Schedule Set - Prod window below. Since we want to run the incremental backups at the same time of day, but on different days, we need to add a new job. If we wanted to run a function immediately before or after the full then we would add it as a new step. So select the Add Job radio button and press the Add New Job With First Action and Run-Time button. If we need to update a job's start time, highlight the job to be updated in the Schedule Set window and press the Update Job Run Time of Day radio button, change the time and then press the Update Job Run Time of Day button.

In the Set Action frame, you should set the function to be performed. The Director knows how to setup a Backup, Migration, Report, Run Job, or Vault/Migrate/Copy and shows the appropriate screens for each function in the next step. As part of this setup of the incremental job, pull down the Start at HH: pull down and select 22 (that results in 22:00 or 10:00 PM) and press the Add the First Action and Run-Time button that adds a second job to the Schedule Set- Prod window.

Press the Next> button to continue.

Step 4: Set / Edit Action Parameters

In this step you decide the UPSTREAM actions and all parameters to be performed for each of the scheduled jobs.

Job 1 is going to be the full and Job2 is going to be the incremental. To setup the full, highlight the Backup step in the Job1 column, leave the All Targets radio button pressed and press the Go Edit Parameters button to setup the parameters for this backup.

This displays the Backup windows with the Wizard on top. It also displays a message box notifying you of a change of translation tables and a message box that says When finished editing settings, click the Done Editing or Cancel button in the Wizard dialogue.

In the Backup window you can make whatever changes you wish to make that affects the backup. Note that the Profile and Target values are ignored but all other changes are stored.

It is highly recommended that you use an include File Spec of \\. that indicates all local drives on the system for Windows, Novell, and UNIX systems. This backup is also a Full Merge to Tape so you will want to set those values. If you need to make Reporting, Local Backup, Novell Profile or other non-repeating options, you can press the Options button to set these values. Any file spec specific options (archive bits, hard-links, etc.) can be made on Specifications window in the same way as for the Backup tab in the Director.

When you have completed your changes, press the Done Editing button in the Wizard to return to the Job and schedule set specification, Step 4: Set / Edit Action Parameters in the Wizard.

To setup the incremental job, highlight Step 1 in the Job 2 column and press the Go Edit Parameters button.

Again, the Backup window is displayed with the Wizard on top. We recommend a File Spec of \\. and press the Incremental Merge and Tape radio buttons. Make any other changes that are appropriate for your environment. Press the Done Editing button in the Wizard to return to the Job and schedule set specification, Step 4: Set / Edit Action Parameters in the wizard. Note that the icons for the actions in the Schedule Set window reflect the type and destination of your backups.

Press the Next> button to go to Step 5: Set/Edit Schedule Dates.

Step 5: Set/Edit Schedule Dates

In the Schedule Set window highlight Step 1, Job1 and press the Set/Edit Job Schedule/Blackout Dates and Settings button. Use the Schedule Dates window, Schedule tab.

Schedule Dates

Because we want to run this job once a week on Fridays, pull down the Schedule Job: list, select Weekly and check the Friday check-box.

In this example, there's no reason to set a start or end date so we won't enter the Start/End Dates tab. There's no reason to set the maximum number of drives, the minutes past start time to run the job if the start time is missed or the retry minutes so we will not enter the Job Settings tab. There is no need to enter any blackout information so we won't enter the Blackout Dates tab or the Backout Start/End Dates tab. Press the Done button to return to the Dispatch Step 5: Set/Edit Schedule Dates window.

Press the Set/Edit Schedule Set Settings button to set notification.

The parameters are:

Logging

If you are running the Dispatcher on the Reservoir machine, when the Dispatcher completes running a job, it writes information concerning the job to the UPSTREAM log and optionally to a number of systems via UPSTREAM mail notification. Here you can enter the level of logging. Logging options include NONE (no client log messages), FULL (all client log messages), ERROR (only errors from the client log), and WARNINGS (warnings and errors from the client log). For this example we recommend FULL.

In the Mail Notification frame:

Configuration File

The Director performs a directory listing of the UPSTREAM notification files on the targeted scheduling machine. We recommend that you use e-mail notification and select a preconfigured notification configuration file (see UPSTREAM Notification for details on setting up Notification).

Include UPSTREAM Log in Message

Check this box to have the Dispatcher put the UPSTREAM log messages into the message sent. We strongly recommend that you check this box.

Only Mail On Failure

Check this box to have the Dispatcher only send messages if the operation fails. We recommend that this be checked in production and unchecked in test, so that you can see all events.

Press the Done button when you are satisfied with your settings.

In the Schedule Set window highlight Step 1, Job2, press the Set/Edit Job Schedule/Blackout Dates and Settings button. Use the Schedule Dates window, Schedule tab., select Weekly and check Monday, Tuesday, Wednesday, and Thursday. Press the Set/Edit Schedule Set Settings button and set the fields similarly to how you set them above. Press the Done button when you are satisfied with your settings.

Press the Next> button to complete the setup and the Finish button to save your settings.

Managing Schedules#

Once you have a schedule defined, you manage it from the Dispatcher Scheduling window that is on the Dispatcher tab when you do not have a schedule open for edit.

In the Schedule Set(s) Summary, window you can highlight a schedule that makes it the active schedule for the other windows on the screen. There are three check-box columns in that window. These are “display only” check-boxes (they cannot be checked by clicking on them). The states are changed through other actions, usually by pressing the buttons below.

Verified

This schedule has been verified by UPSTREAM and can be loaded.

Loaded

The Client has loaded the schedule into memory. It can only be activated when it's loaded. Schedules are loaded automatically when the Dispatcher is started if they are verified. When you add new schedules they must be manually loaded until the next start of UPSTREAM.

Suspended

A loaded schedule can be suspended - not active but known about by the Dispatcher.

In our example, the schedule is not yet loaded. So press the Load button to load the schedule into memory. Note that the schedule is marked as both Loaded and Suspended. The Dispatcher automatically marks a loaded schedule suspended so it doesn't automatically start. You could immediately begin executing the schedule by pressing the Resume button. However, for this example, we'll test the schedule.

To test the full schedule by running the first full backup, highlight PROD in the Schedule Set(s) Summary frame, and highlight the Backup in Job1 Step 1. This highlights all 3 systems in the Schedule Set Target(s) frame. If you wish you could highlight one or any set of these targets, but by default they are all highlighted. You can then press the Run Job Now button to begin the backups. When the backups complete (successfully or unsuccessfully) you will get an e-mail.

Once the backups have completed, you may choose to press the Resume button to have the schedule set activated. Note that if you don't want to have it activated the next time that UPSTREAM is started, you must Reopen the schedule definitions, use the Wizard to go to Step 5: Set/Edit Schedule Dates, and check the box that says Suspended at Start-Up.

The functions of the buttons on the Scheduling Window are:

Refresh Status

Press this button to have the status list and check-boxes updated.

Suspend

Suspend the highlighted schedule. Greyed unless its currently active.

Resume

Resume the highlighted schedule. For new schedules, you must always press the Load button to have it loaded into the Dispatcher and Resume to have it activated. Greyed unless its currently suspended.

Re-Load/Load

For new schedules you must always press the Load button to have it loaded into the Dispatcher (and Resume it to activate it as manually loaded schedules are suspended).
If you change a schedule you may have activated it at save time, but if you did not, you can press the Re-Load button to have the change activated. Any changes to the schedule are reloaded automatically when UPSTREAM is restarted.

Remove

Removes (deletes) the highlighted schedule.

Scheduled Commands

Allows you to schedule all Dispatcher control commands. Described below.

Refresh List

Updates the list of schedule sets from UPSTREAM.

View Dates

Displays a calendar of when the events will run. In our example, we see:

You can press the Day, Week, or Month tab to see the times when particular jobs will run.

Wizard

Displays the Dispatcher Wizard if the window has been closed.

Start Dispatcher

Starts the Dispatcher on the UPSTREAM machine. Greyed if the Dispatcher is already started.

Stop Dispatcher

Stops the Dispatcher on the UPSTREAM machine. Greyed if the Dispatcher is already stopped.

Running Jobs

You can see the list of jobs that are running or about to run in the Dispatcher in the Run Dispatcher Job(s) window:

Run Dispatcher Job(s)

This list shows all running jobs and those queued internally within the Dispatcher that are pending (for tape drives or other resources). You can remove them before they run by highlighting them and pressing the Remove Job(s) button. You may need to press the Refresh Status button to have the window updated to the current list.

We recommend that you monitor and manage jobs that have already started in the Open Systems Console.

Global Options

Global Options are settings that apply across all scheduled jobs. You access the global options setting by pressing the Global Options button in the Dispatcher Scheduling Window.

Global Options

Maximum Concurrent Jobs

Allows you to throttle the number of jobs that the Dispatcher will keep running, including backups, vaults, run jobs - all Dispatcher started functions. All zeros disables throttling.

Available Tapes

If left blank, the Dispatcher auto-detects the number of tape drives in the system and limits the number of active job that use tapes to not exceed that number. However, you can specify a smaller number, if you wish to limit the number of tape drives used specifically by the Dispatcher. This is useful if you wish to allow jobs to run and perform ad-hoc restores for example.
If this number is larger than the number of drives, then more jobs are started than the number of drives available and Reservoir does tape drive sequencing. This may result in a deadlock so it is recommended only in situations where you know you can avoid those.

Scheduled Commands

The Scheduled Commands feature of the Dispatcher lets you schedule scheduling features including starting and stopping the Dispatcher, suspending and resuming schedules and more. This is particularly useful for manually managing scheduled blackouts and the like.

From the Scheduling window, press the Scheduled Commands window. Press the Add button to add a scheduled command.

Scheduled Commands

Date Time

Press the Set button to display a calendar that allows you to set a date and time when the scheduled command will be executed.

Command

Select a command:

Start

Lets you schedule the start of the entire Dispatcher. The date/time are the only parameters.

Stop

Lets you schedule the stop of the entire Dispatcher. The date/time are the only parameters.

Suspend

Lets you schedule the suspend of a Schedule Set. Besides the date/time, you can specify a specific schedule set or all schedule sets, a specific target or all targets a specific job or all jobs.

Resume

Lets you schedule the resumption of a suspended Schedule Set. Besides the date/time, you can specify a specific schedule set or all schedule sets, a specific target or all targets a specific job or all jobs.

Load

Lets you schedule the load of a given Schedule Set. Besides the date/time, you must specify the name of the schedule set to load.

Run job

Lets you schedule the one-time execution of schedule set, one or all targets, one or all jobs.

Once you have specified the command press the <<Add button to add the command to the list of Scheduled Commands. Pressing the <<Add button will cause the command to be sent to the Dispatcher immediately. If there are errors, you will need to examine the messages (log) from that given target.

The buttons at the bottom of the window allow you to act on highlighted scheduled commands:

Add

Described above.

Execute Now

Allows you to execute the highlighted scheduled command immediately.

Refresh

Updates the list from the Dispatcher.

Remove

Deletes the highlighted scheduled command.

After your Dispatcher is executed the entry remains with the status Executed. You must manually delete these.

Connecting Schedules

There often a need to connect scheduled functions so that one follows immediately after another. For example, if you have a limited number of tape drives, you may want to run database backups immediately after the regular file system backups have completed so that they are guaranteed access to the tape drives.

Similarly, you may want to run a function only if the prior function has succeeded. For example, you may choose to only run a vault if the regular file system backups succeed.

To connect separate Dispatcher schedules:

• Create both the preceding and succeeding schedules as if they were independent jobs. Note that the time of the succeeding schedule will be ignored.
  
• In the preceding schedule, go into Step 3: Add Job / Run-Time / Action Steps, and press the Connect Another Job As Last Step radio button, select the name of the schedule set and the job you wish to succeed this job. Save the schedule.
  
• If you do not want condition codes checked, you are finished with configuration. If you want condition codes checked, go into the succeeding job Dispatcher configuration, go into Step 3: Add Job / Run-Time / Action Steps and press the Set/View Condition to Run radio button. You can then specify the condition code of the preceding job that allows this job to run.
  

If you want to chain a number of jobs together, each one connects separately to the next following the procedures above. Note that a job can only be the successor of one job; it cannot succeed two separate jobs.

More Complex Scheduling

Besides the basic scheduling demonstrated in the example above, there are a number of more complex features in scheduling that allow you to do things like:

• Execute functions that the Director does not know how to do (detailed VSS backups or Exchange Mail backups for example). Setup and save a parameter file using the UPSTREAM Local program. In Step 3: Add Job / Run-Time / Action Steps, select the Run Job action. In Step 4: Set / Edit Action Parameters, when you press Edit Parameters, you can then specify the ustbatch program that allows you to submit a request to the Reservoir system to have a function run on another system (using TARGNAME and WSPARM) and have the results returned. See the UPSTREAM Reservoir User's Guide for a description on using ustbatch.
  
• This allows you to run such functions as Reservoir vaults (using uscopy), database quiesce or anything that can be done by executing a program.
  
• Add steps to a job. This allows you to run a scheduled event immediately before or after another one. This could be used for example, to run a script after a profile set backup completes.
  
• Unique definitions for specific targets. This allows you set different UPSTREAM parameters for one or more targets within a schedule such as the job command line, UPSTREAM reporting, local backup and more.
  
• Bound schedules around start and end dates. Useful for scheduled events that only need to be done for a limited time or to setup a schedule that will begin when the equipment is fully in place.
  
• Execute other functions: In Step 3: Add Job / Run-Time / Action Steps in the Dispatcher wizard, you can choose for First Step UPSTREAM Action the functions: Run Job, Vault/Migrate/Copy, or Migration, instead of backup. The rest of the wizard setup is the same, except for Step 4: Set / Edit Action Parameters when you press Go Edit Parameters, you see the window that matches your function rather than the Backup window.
  
• Add steps to a job: In Step 3: Add Job / Run-Time / Action Steps in the Dispatcher wizard, you can press the Add Step to Job action button, which adds a new job step to the end of the highlighted job. In all subsequent windows, highlight the given step in the Schedule Set window to modify settings for the given step.
  
• Unique definitions for specific targets: In Step 4: Set / Edit Action Parameters in the Dispatcher wizard, in the Apply Edits to These Targets frame, if you press the Selected Targets radio button, all the targets are not highlighted. You can then highlight one or more specific targets that are affected with the changed parameters that you set when you press the Go Edit Parameters button.
  
• Bound schedules around start and end dates: In Step 5: Set/Edit Schedule Dates in the Dispatcher wizard, when you press the Set/Edit Job Schedule/Blackout Dates button, in the Schedule Dates window, press the Start/End Dates tab. Here you can set a specific start and end date for the job step.
  
• Set blackout dates: In Step 5: Set/Edit Schedule Dates in the Dispatcher wizard, when you press the Set/Edit Job Schedule/Blackout Dates button, in the Blackout Dates tab you can specify specific blackout dates or a daily/weekly/monthly schedule of blackout dates. You can then optionally press the Blackout Start/End Dates tab and set boundary dates for the blackout.
  

Visual Editor

The Director Visual Editor displays the jobs visually, allows easy connecting and disconnecting of jobs and allows access to job settings without having to go through the Wizard. To display the Visual Editor, in the Director, press the Dispatcher tab, select the Dispatcher Scheduling window and press the Visual Editor button.

Because the following figure allows direct access to many of the functions available in the Wizard, we strongly recommend its use before using the Wizard to change existing jobs.

Visual Editor

To change settings for a given job, click on the job with the mouse and press the right mouse button. This brings up a sub-menu:

Disconnect

Allows you to remove the connection from a preceding job to this one. Grayed if there is no preceding connected job.

Job Settings

Allows you to change number of tape drives, minutes past start time to begin job, and minutes to keep retrying failed UPSTREAM contact.

 Run-Time

Allows you to change the start time and set or unset a repetition time.

Set Dates

Allows you to change Schedule Dates, Start/End Dates, Job Settings, Blackout Dates, and Blackout Start/End Dates.

Schedule Set (All Jobs) Remove

Allows you to remove the schedule set entirely.

Schedule Set Suspend/Resume

Allows you to suspend or resume a schedule set.

Schedule Set Settings

Allows you to change logging and mail notification for a schedule set.

To connect jobs, grab a yellow oval job with the mouse and drop it on an orange function. This causes the job to be moved after the given function. Note that the job start time is ignored as it starts when the preceding connected job completes.

UPSTREAM Dispatcher Commands from the Command Line

Almost all the Dispatcher commands available from the Director interface can be issued from a command line using dispcommand utility.

Here is a brief help displayed when you start dispcommand without parameters:

Second set of parameters specifies the Dispatcher command:
 “stop” - stop the Dispatcher, no parameters other than time;
 “start” - start the Dispatcher, no parameters other than time;
 “suspend” - suspend ScheduleSet(s), see parameters below;
 “resume” - resume ScheduleSet(s), see parameters below;
 “load” - load ScheduleSet(s), see parameters below;
 “run” - run a Job from a ScheduleSet, see parameters below.

Command parameters - time spec:
 <YY/MM/DD/HH:MM:SS>

Additional parameter modifiers:
 “remove” - remove this previously posted command.
 “noconnjobs” - don’t start connected jobs (for “run” command only.)

The first parameter of the dispcommand is the IP address and the listening port number of the Client that has the Dispatcher running. You can dispcommand from any machine on your network that can communicate with the Client running Dispatcher.

Commands start and stop can take only a time parameter that specifies the time the command executes. If no time is specified the command is executed immediately.

The rest of the commands must have at least a schedule specification. A schedule specification requires two fields: the name of the schedule set and the job name. For run command you can also specify step name and the target name.

If step name is not specified, Dispatcher runs the whole job. If the step name is specified, then only that step of the job gets executed, and none of the connected jobs (if any) get started.

If the target name is not specified the Dispatcher starts the job on all targets specified in the schedule set. You can run the job on a particular target by specifying in here.

Time specification allows you to specify the exact date and time when you want the command to be executed. Omitting the time specification causes the command to get executed immediately.

Additional parameter modifiers have the following meaning:

remove

Can be appended to any previously issued command that had a time specification, and it removes that command from the Dispatcher scheduled command list.

noconnjobs

Can accompany only run command to prevent Dispatcher from starting jobs connected to the specified job.